Skip to content

ci: gate PR benchmark job behind a benchmark-relevant paths filter - #30

Merged
IlyasRidhuan merged 1 commit into
mainfrom
ci/benchmark-paths-filter
Aug 19, 2026
Merged

ci: gate PR benchmark job behind a benchmark-relevant paths filter#30
IlyasRidhuan merged 1 commit into
mainfrom
ci/benchmark-paths-filter

Conversation

@alejoamiras

@alejoamiras alejoamiras commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

The benchmark / Benchmark job takes ~35 minutes and runs on every PR — docs-only changes included. It only produces meaningful output when something that affects compiled contracts or the benchmark harness changed.

Change

A changes job (dorny/paths-filter, SHA-pinned to the verified v3.0.2 commit) gates benchmark at the job level. It runs only when the PR touches:

Path Why
src/**/*.nr contract sources (gate counts)
**/Nargo.toml workspace + per-contract manifests, aztec-nr git refs
benchmarks/** the benchmark definitions
src/ts/test/utils.ts every benchmark imports its setup helpers from here (verified: 6/6, and this file has no further local imports)
tsconfig.json TS resolution config for the benchmark harness
package.json, yarn.lock @aztec/* SDK + config.aztecVersion toolchain pins
.github/workflows/pr-checks.yml the workflow itself

format and checks (JS/Noir tests) are deliberately untouched.

Pre-merge verification (all done — no reviewer legwork needed)

Case Method Result
Required-check trap (skip leaves a required context "Expected" forever) GraphQL isRequired on live PR check contexts no status check is required on this repo, so the trap cannot trigger
Positive: workflow-file change → benchmark runs this PR's own CI run Changes: success, benchmark / Benchmark ran
Positive: .nr contract change → benchmark runs probe PR #31 (closed, run cancelled after verdict) benchmark / Benchmark started
Negative: docs-only change → skips probe PR #29 (closed) benchmark: skipped, run concludes green
Negative: TS-test-body-only change → skips probe PR #32 (closed, run cancelled after verdict) benchmark: skipped

Naming asymmetry to be aware of (matters only if checks are ever made required): when the job runs, the check is benchmark / Benchmark; when skipped, only benchmark exists. If benchmark should ever become a required check, that needs an always-run aggregate job evaluating needs results — happy to do as a follow-up.

Accepted trade-offs

  • Stacked PRs: a skipped run uploads no benchmark-baseline-<branch> artifact, so a PR stacked on a skipped branch gets no baseline for its comparison (if_no_artifact_found: warn downstream — degrades gracefully, no stale-artifact fallback: the download action doesn't search older runs by default).
  • Fail-open: if the changes job itself fails (rare — API outage), benchmark skips; the failed Changes check is a loud red X on the PR.
  • Single-file helper entry: if benchmark setup helpers ever move out of src/ts/test/utils.ts, the filter must be updated in the same PR.

Review

Adversarially reviewed via a Codex loop (2 rounds): its confirmed finding (missing utils.ts/tsconfig.json) is incorporated; its stale-baseline concern was checked against the download action's source and retracted; final verdict was ship-as-draft. actionlint passes.

🤖 Generated with Claude Code

Benchmark takes ~35 minutes and ran on every PR, docs-only included. A
changes job (dorny/paths-filter, SHA-pinned v3.0.2) now gates it on paths
that can affect benchmark output: contract sources, Nargo manifests, the
benchmark definitions and their shared setup helpers, dependency
manifests, and the workflow itself.

Verified on a probe PR: with job-level gating a docs-only PR still gets a
concluded run and the gated caller reports as a skipped 'benchmark'
check. The nested 'benchmark / Benchmark' check is not created when
skipped, so that exact context must not be listed as a required status
check (see PR description for the required-checks implications).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alejoamiras
alejoamiras marked this pull request as ready for review August 18, 2026 16:30
@alejoamiras
alejoamiras marked this pull request as draft August 18, 2026 16:30
@alejoamiras
alejoamiras marked this pull request as ready for review August 18, 2026 16:32
@github-actions

Copy link
Copy Markdown

Benchmark Comparison

CPU Cores RAM Arch
AMD EPYC 9V74 80-Core Processor 4 16 GiB x64

Contract: escrow

🚦 Function Gates DA Gas L2 Gas Proving Time (ms)
Base PR Diff Base PR Diff Base PR Diff Base PR Diff
(partial) withdraw 485,780 485,780 3,744 3,744 587,800 587,800 13,838 15,010 +1,172 (+8.5%)
withdraw 330,054 330,054 832 832 499,700 499,700 11,178 12,156 +978 (+8.7%)
withdraw_nft 378,231 378,231 1,440 1,440 527,400 527,400 12,313 13,338 +1,025 (+8.3%)

Contract: logic

🚦 Function Gates DA Gas L2 Gas Proving Time (ms)
Base PR Diff Base PR Diff Base PR Diff Base PR Diff
get_escrow 390,336 390,336 192 192 456,000 456,000 12,439 13,479 +1,040 (+8.4%)
secret_key_to_public_keys 387,695 387,695 192 192 456,000 456,000 12,459 13,481 +1,022 (+8.2%)
share_escrow 304,211 304,211 1,952 1,952 520,700 520,700 10,952 11,957 +1,005 (+9.2%)

Contract: multitoken

🚦 Function Gates DA Gas L2 Gas Proving Time (ms)
Base PR Diff Base PR Diff Base PR Diff Base PR Diff
burn_private 260,232 260,232 832 832 499,700 499,700 10,162 10,945 +783 (+7.7%)
burn_public 223,800 223,800 416 416 638,292 638,292 9,758 10,516 +758 (+7.8%)
initialize_transfer_commitment 225,054 225,054 768 768 474,500 474,500 9,690 10,557 +867 (+8.9%)
mint_to_private 297,795 297,795 1,408 1,408 511,400 511,400 10,932 11,829 +897 (+8.2%)
mint_to_public 223,800 223,800 416 416 637,800 637,800 9,657 10,510 +853 (+8.8%)
transfer_private_to_commitment 263,571 263,571 1,024 1,024 511,400 511,400 10,233 10,949 +716 (+7.0%)
transfer_private_to_private 336,124 336,124 2,048 2,048 555,100 555,100 11,350 12,343 +993 (+8.7%)
transfer_private_to_public 306,785 306,785 1,056 1,056 714,677 714,677 10,953 11,796 +843 (+7.7%)
transfer_public_to_commitment 223,800 223,800 640 640 663,516 663,516 9,772 10,506 +734 (+7.5%)
transfer_public_to_private 298,556 298,556 1,024 1,024 683,958 683,958 10,807 11,722 +915 (+8.5%)
transfer_public_to_public 223,800 223,800 480 480 673,505 673,505 9,690 10,509 +819 (+8.5%)

Contract: nft

🚦 Function Gates DA Gas L2 Gas Proving Time (ms)
Base PR Diff Base PR Diff Base PR Diff Base PR Diff
burn_private 275,232 275,232 416 416 661,052 661,052 10,545 11,397 +852 (+8.1%)
burn_public 223,800 223,800 448 448 670,034 670,034 9,681 10,489 +808 (+8.3%)
mint_to_private 341,272 341,272 1,600 1,600 735,342 735,342 11,468 12,447 +979 (+8.5%)
mint_to_public 223,800 223,800 448 448 670,718 670,718 9,684 10,499 +815 (+8.4%)
transfer_private_to_private 249,433 249,433 832 832 499,700 499,700 9,934 10,806 +872 (+8.8%)
transfer_private_to_public 275,261 275,261 416 416 659,258 659,258 10,665 11,419 +754 (+7.1%)
transfer_public_to_private 292,557 292,557 992 992 683,268 683,268 10,774 11,676 +902 (+8.4%)
transfer_public_to_public 223,800 223,800 384 384 633,285 633,285 9,661 10,563 +902 (+9.3%)

Contract: token

🚦 Function Gates DA Gas L2 Gas Proving Time (ms)
Base PR Diff Base PR Diff Base PR Diff Base PR Diff
burn_private 303,054 303,054 1,024 1,024 713,957 713,957 10,915 11,803 +888 (+8.1%)
burn_public 223,800 223,800 448 448 672,632 672,632 9,641 10,483 +842 (+8.7%)
initialize_transfer_commitment 225,054 225,054 768 768 474,500 474,500 9,656 10,423 +767 (+7.9%)
mint_to_private 355,507 355,507 2,144 2,144 738,070 738,070 11,648 12,688 +1,040 (+8.9%)
mint_to_public 223,800 223,800 448 448 672,227 672,227 9,639 10,511 +872 (+9.0%)
transfer_private_to_commitment 259,896 259,896 992 992 511,400 511,400 10,090 10,976 +886 (+8.8%)
transfer_private_to_private 343,699 343,699 2,592 2,592 557,600 557,600 11,476 12,560 +1,084 (+9.4%)
transfer_private_to_public 303,110 303,110 1,024 1,024 714,023 714,023 10,803 11,829 +1,026 (+9.5%)
transfer_private_to_public_with_commitment 307,280 307,280 1,600 1,600 747,323 747,323 10,961 11,889 +928 (+8.5%)
transfer_public_to_commitment 223,800 223,800 576 576 662,529 662,529 9,696 10,497 +801 (+8.3%)
transfer_public_to_private 296,520 296,520 992 992 683,304 683,304 10,819 11,833 +1,014 (+9.4%)
transfer_public_to_public 223,800 223,800 448 448 672,548 672,548 9,621 10,499 +878 (+9.1%)

Contract: vault

🚦 Function Gates DA Gas L2 Gas Proving Time (ms)
Base PR Diff Base PR Diff Base PR Diff Base PR Diff
deposit_private_to_private 507,983 507,983 1,312 1,312 878,248 878,248 14,177 15,380 +1,203 (+8.5%)
deposit_private_to_private_exact 611,708 611,708 1,888 1,888 915,316 915,316 16,094 17,448 +1,354 (+8.4%)
deposit_private_to_public 435,487 435,487 768 768 862,273 862,273 13,152 14,157 +1,005 (+7.6%)
deposit_public_to_private 424,931 424,931 1,984 1,984 965,045 965,045 13,015 14,057 +1,042 (+8.0%)
deposit_public_to_private_exact 424,942 424,942 1,952 1,952 949,469 949,469 12,965 14,048 +1,083 (+8.4%)
deposit_public_to_public 223,800 223,800 832 832 897,563 897,563 9,783 10,528 +745 (+7.6%)
issue_private_to_private_exact 611,708 611,708 1,888 1,888 915,976 915,976 16,048 17,411 +1,363 (+8.5%)
issue_private_to_public_exact 484,222 484,222 1,344 1,344 899,998 899,998 13,959 15,056 +1,097 (+7.9%)
issue_public_to_private 346,383 346,383 1,376 1,376 921,661 921,661 11,569 12,542 +973 (+8.4%)
issue_public_to_public 223,800 223,800 832 832 898,253 898,253 9,819 10,509 +690 (+7.0%)
redeem_private_to_private_exact 614,712 614,712 1,888 1,888 915,823 915,823 16,111 17,444 +1,333 (+8.3%)
redeem_private_to_public 435,430 435,430 768 768 862,717 862,717 13,154 14,193 +1,039 (+7.9%)
redeem_public_to_private_exact 428,003 428,003 1,952 1,952 949,790 949,790 13,037 14,099 +1,062 (+8.1%)
redeem_public_to_public 223,800 223,800 832 832 898,085 898,085 9,707 10,473 +766 (+7.9%)
withdraw_private_to_private 510,987 510,987 1,312 1,312 878,458 878,458 14,232 15,373 +1,141 (+8.0%)
withdraw_private_to_private_exact 614,712 614,712 1,888 1,888 915,769 915,769 16,122 17,483 +1,361 (+8.4%)
withdraw_private_to_public_exact 484,165 484,165 1,344 1,344 900,232 900,232 13,983 15,036 +1,053 (+7.5%)
withdraw_public_to_private 441,260 441,260 2,528 2,528 967,356 967,356 13,216 14,312 +1,096 (+8.3%)
withdraw_public_to_public 223,800 223,800 832 832 898,388 898,388 9,743 10,491 +748 (+7.7%)

@IlyasRidhuan
IlyasRidhuan merged commit 564a48d into main Aug 19, 2026
8 checks passed
@IlyasRidhuan
IlyasRidhuan deleted the ci/benchmark-paths-filter branch August 19, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants