profile-diff-action

Profile a pytest suite, script, or callable — then comment a CPU + memory hotspot diff on every pull request.

Performance regressions rarely announce themselves. A helper grows a nested loop, an innocent-looking .copy() creeps into a hot path, and three releases later a CI job that took 40 seconds takes two minutes — with no single commit obviously to blame. profile-diff-action puts the number in front of you at review time: it profiles your target on the base branch and on the PR branch, then posts a single, self-updating comment diffing the two.

## Performance profile diff

**Runtime (total cumtime):** 59.5 ms → 118.9 ms (+59.4 ms ▲)

**Peak memory:** 4.2 MB → 9.8 MB (+5.6 MB ▲)

### Top regressions ▲ (slower)

| Function | Base | Head | Δ | Δ% |
|---|---:|---:|---:|---:|
| `app/report.py:42(build_rows)` | 12.1 ms | 71.5 ms | +59.4 ms | +490.9% |
| `app/report.py:88(_dedupe)`    |  3.0 ms |  9.2 ms |  +6.2 ms | +206.7% |

### Removed hotspots

| Function | cumtime |
|---|---:|
| `app/report.py:60(_fast_dedupe)` | 2.9 ms |

It is a single self-contained package (profile_diff) using only the Python standard library — cProfile, pstats, tracemalloc, urllib, argparse. No third-party dependencies, nothing to pip install from an index.

Why this exists

Two numbers from cProfile tell different stories, and mixing them up is the most common profiling mistake:

• tottime — time spent inside a function itself, excluding its callees. Great for finding the one function whose own body is expensive.
• cumtime — cumulative time in a function and everything it calls. This is the number that tracks the cost of an entire code path.

For a CI signal you almost always want cumtime. A regression usually isn't a single line getting slower; it's a call tree getting more expensive — an extra O(n) pass wrapped somewhere up the stack. cumtime rolls that whole subtree into one comparable number per function, which is exactly what a branch-to-branch diff needs. That is why this tool matches functions by identity across two runs and diffs their cumtime, while also surfacing tottime and ncalls in the raw report. (More on reading the two columns: interpreting cProfile cumulative vs total time.)

Memory gets the same treatment via tracemalloc: the report records peak traced memory for the run plus the top allocation sites (file:line), so the comment can flag "this PR now peaks 5 MB higher" alongside the runtime delta.

Running it

This tool is meant to be run from a clone of the source. It is intentionally not published to PyPI; the GitHub Action checks out this repository and runs the module directly, and you can do the same locally.

git clone https://github.com/python-testing-debugging/profile-diff-action.git
cd profile-diff-action
python -m profile_diff --help

There are three subcommands. A typical local run is profile base → profile head → compare:

# 1. Profile each side into its own JSON report (do this on each checkout).
#    Put --pytest LAST: everything after it is forwarded to pytest.
python -m profile_diff run --label base --out base.json --pytest tests/
python -m profile_diff run --label head --out head.json --pytest tests/

# 2. Diff the two reports into a Markdown comment body.
python -m profile_diff compare base.json head.json > diff.md

# 3. (In CI) post or update the PR comment.
GITHUB_TOKEN=*** GITHUB_REPOSITORY=owner/repo PR_NUMBER=123 \
    python -m profile_diff comment diff.md

run — profile a target

Pick exactly one target selector:

Flag	Purpose
--pytest [ARGS...]	Profile an in-process pytest run. Pass it last — every following token (e.g. -q tests/) is forwarded to pytest.
--script PATH	Profile a Python script executed as __main__.
--callable pkg.mod:func	Profile a zero-argument importable callable.
--out PATH	Where to write the JSON report (default: stdout).
--top N	Keep the top N functions by cumtime (default 30).
--label NAME	Human label stored in the report (e.g. base / head).
--script-arg ARG	Extra argv for --script (repeatable).

compare — diff two reports

Flag	Purpose
--threshold-pct PCT	Ignore cumtime moves below this percentage (default 5).
--min-abs-ms MS	Ignore cumtime moves below this many milliseconds (default 0).
--top N	Movers to show in each direction (default 10).
--format {markdown,text,json}	Output format (default markdown).
--fail-on-regression	Exit 2 if any hotspot regresses beyond the threshold.
--out PATH	Where to write the report (default: stdout).

comment — post/update the PR comment

Flag	Env fallback	Purpose
--repo	GITHUB_REPOSITORY	owner/name slug.
--pr	PR_NUMBER	Pull-request number.
--token	GITHUB_TOKEN	Token with pull-requests: write.

The body argument is a path, or - to read the Markdown from stdin.

How it works

1. Profile. run executes your target once, wrapped in both a cProfile.Profile and a live tracemalloc trace. For --pytest it runs pytest in-process via runpy so the whole test session is captured; for --script/--callable it imports/executes the target the same way.
2. Extract stable records. pstats yields (ncalls, tottime, cumtime) per function. Frames are keyed as file:line(func), and file paths are normalised relative to the working directory so the same source profiled from two different checkout directories produces the same key. The tool's own harness frames are filtered out so the report is about your code.
3. Record memory. Peak traced bytes and the top allocation sites come straight from the tracemalloc snapshot.
4. Write JSON. A small, documented, versioned schema (schema_version, metadata, functions[], memory{}) — stable enough to diff across machines.
5. Compare. compare matches functions by key, computes cumtime deltas, applies the percentage / absolute-millisecond thresholds, and splits the survivors into regressions, improvements, and new/removed hotspots — with fully deterministic ordering.
6. Render & comment. The Markdown carries a hidden marker (<!-- profile-diff-action -->). comment lists the PR's comments, and if it finds that marker it updates the existing comment instead of posting a new one — so CI never spams the thread.

Exit codes

Command	Code	Meaning
any	0	Success (and, for compare, no gated regression).
compare	2	--fail-on-regression set and a hotspot regressed beyond the threshold.
any	1	Usage / runtime error (bad target, missing token, unreadable report).

Demo

From the repository root — profiles the bundled slow vs fast workload and prints a real diff (no network required):

PROFILE_DIFF_SLOW=1 python -m profile_diff run \
    --callable examples.workload:run_workload --label base --out base.json
PROFILE_DIFF_SLOW=0 python -m profile_diff run \
    --callable examples.workload:run_workload --label head --out head.json
python -m profile_diff compare base.json head.json

Full walkthrough (including how to produce a failing regression gate) in examples/README.md.

Using it as a GitHub Action

action.yml is a composite action that sets up Python, profiles the current checkout and the base ref (via git worktree), compares them, and comments on the PR. A consumer repo adds a workflow like .github/workflows/example.yml:

permissions:
  contents: read
  pull-requests: write

jobs:
  profile-diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0            # base commit must be reachable
      - uses: python-testing-debugging/profile-diff-action@v1
        with:
          target: "--pytest tests/"
          fail-on-regression: "false"

Background reading

This tool grew out of the performance-profiling material on python-testing-debugging.com, a site about advanced Python testing and debugging. If you want the theory behind the numbers this action prints, these are the most relevant pages:

• CPU profiling with cProfile and py-spy
• Interpreting cProfile: cumulative vs total time
• Memory profiling with tracemalloc
• Comparing tracemalloc snapshots to locate growth

License

MIT — see LICENSE.