Skip to content
This repository was archived by the owner on Oct 17, 2026. It is now read-only.

test(perf): time the fastest batch so load can't fail the gate - #7

Merged
oddurs merged 1 commit into
mainfrom
test/robust-perf-gate
Oct 17, 2026
Merged

oddurs merged 1 commit into
mainfrom
test/robust-perf-gate

Conversation

@oddurs

@oddurs oddurs commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Problem

The debug ranking gate averaged twenty runs. On this Mac, routinely at load average 20–48, one busy moment pushed it over its 40 ms budget and the pre-push hook refused the push. Interleaved runs on unchanged main measured 37, 45, 14, 11, and 13 ms.

Approach

Time five batches of four runs and keep the fastest. Other work can only slow a batch down, so the fastest batch is the closest measurement of the code itself. The same twenty calls, the same 40 ms budget.

Look at sceptically

  • This makes the gate more lenient towards intermittent slowness in the code itself, such as a cache that's cold on one batch in five. Ranking has no such state, so I don't think it matters here.
  • A real 10× regression still fails: the fastest batch would then sit near 100 ms.

Checklist

  • scripts/task check passes locally (three runs at load ~19: 9.2, 9.4, 12.8 ms)

The debug ranking gate averaged twenty runs, so one busy moment on a loaded Mac pushed it over budget: main itself measured 37 and 45 ms against 40 while its fastest runs took 11. Load can only slow a batch down, so the fastest of five batches measures the code rather than the machine. The budget is unchanged.
@oddurs
oddurs merged commit 6e0bc6c into main Oct 17, 2026
2 checks passed
@oddurs
oddurs deleted the test/robust-perf-gate branch October 17, 2026 01:46
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant