Skip to content

docs(stall-guard): state the boundary of a green budget line, and what the guard never sees - #12960

Merged
os-elon merged 3 commits into
mainfrom
claude/issue-12846-stall-guard-budget
Aug 28, 2026
Merged

docs(stall-guard): state the boundary of a green budget line, and what the guard never sees#12960
os-elon merged 3 commits into
mainfrom
claude/issue-12846-stall-guard-budget

Conversation

@os-elon

@os-elon os-elon commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Part of #12846 — delivers option 3 (documented acceptance) as triage scoped it. Options 1, 2 and a fourth found during measurement are handed to the maintainer and deliberately not built here.

⛔ No timeout-minutes, cap or stall window is changed by this PR.

What landed

The gate header carried a bound measured on one run, and re-measurement found it had already gone stale in the direction that consumes budget. That paragraph is replaced with a four-item statement of what a green line does not promise, in scripts/check-stall-guard-budget.mjs:

  1. Where the verdict landsp + s + C; room for p + s is T - C, on the ci.yml family 30 - 20 = 10 minutes.
  2. The undeferred path IS covered — the CI: Test Core stalls mid-suite with frozen log output — three occurrences in one day, each costing a manual diagnosis + rerun #4250 shape, so a reader does not conclude the whole thing is uncovered. But the margin is now barely over a minute, having shrunk roughly a quarter in days while nothing in the tree changed and the gate stayed green throughout.
  3. The deferred path is NOT covered, and no static gate can read why — the term that consumes the budget is the healthy run length.
  4. Two wedge classes the guard never sees at all — it fires on silence, so a wedge that keeps writing never reaches the cap logic, and a wedge in an unguarded step is not watched (16 of the test job's 17 steps). No value of T closes either.

Item 2 is the sentence that makes item 3 matter: a margin that moved 24% while nobody could see it move is the argument for why the invisible term is dangerous.

Also corrected where the header states the population: 7 guard-wrapped steps across 5 jobs, not 4 jobstemporal-conformance carries two on one job clock, so the second step's p includes the first's entire runtime.

scripts/run-with-stall-guard.mjs gets the same two limits stated where it describes its own kill paths, pointing at the gate header rather than repeating the argument.

No number is embedded as a constant

Triage's data rule is binding and the reason is on display in this very PR: the figure being replaced had rotted ~24%. The header cites the method and the run IDs instead —

node scripts/measure-stall-guard-headroom.mjs --run RUN_ID [--run ...]

readings from merge_group runs 33160601033, 33162164422, 33163163494; the card's original sample was 33135187774.

That tool enumerates guarded steps by importing the gate's own sweep, so a renamed step surfaces loudly instead of silently dropping out, and it refuses on an empty population rather than printing a green. Its header was trimmed in this commit to stop duplicating the argument that now lives in the gate header — two copies that must agree with nobody holding them to it is the defect class this repo dislikes most.

Handed up, not built

  • Option 1 (raise timeout-minutes) — fleet spend, and measurement shows its stated cost is real, not illusory: the guard cannot bound the two wedge classes above, so delta-T is billed one-for-one there.
  • Option 2 (shrink the cap) — narrows the probe; not measured to retain its power.
  • Option 4 (give the guard an absolute deadline) — found during this work, costs no runner minutes, and closes classes the others cannot. Still a contract change to the guard, so it is the maintainer's call.

Verification

Run at 13c72bf5e9, this branch's head, after syncing with main (the first derivation printed STALE TREE and was re-taken).

Gate families derived from the final diff (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack), all exit 0, each captured before any pipe: check:agent-test-spelling, check:bash32-floor, check:cli-command-ids, check:cross-package-test-inputs, check:entry-guard, check:parse-guard, check:pnpm-filter-targets, check:stall-guard, check:stall-guard-budget, check:watch-hint-literal, check-ci-filter-parity.mjs, check-cross-package-test-inputs.mjs. The two bold ones appeared only once this diff touched the guard scripts themselves.

Run because a human would, not because the derivation named them: check:nul-bytes (exit 0), the measurement tool's self-test (13 assertions, exit 0), and eslint over the changed files.

Those greens are measurements, not silences — check:stall-guard ran the guard's own 41-case self-test and check:stall-guard-budget runs --self-test (40 assertions, both violation tiers driven red, empty population proven to REFUSE) and then the real sweep. Since this diff edits the file the gate parses for its defaults, the assertion that matters is the one asserting those defaults are read rather than copied — it passed, so the new header prose is not being mistaken for a declaration.

eslint narrowing, with the evidence a narrowing needs: the population is read from the config itself, which states it "never enables type-aware linting (no parserOptions.project, no typed @typescript-eslint rules) for ANY file" — so this diff cannot move a verdict on an untouched file; --format json reports exactly 3 files linted, matching the 3 changed; 0 errors, 0 warnings, exit 0.

skip-changeset: scripts/ matches no pnpm-workspace.yaml glob and the root package is private: true, so this diff publishes nothing.

Session: https://claude.ai/code/session_01CPrUz21stTFhJRUirdc4yw

The stall guard's budget depends on p+s (job prep plus how far into the
guarded step output froze). check-stall-guard-budget judges C<T and
T-C>=W, all statically readable; p+s is bounded by the healthy run
length, which no static sweep can read, so it was carried in prose and
had already gone stale.

This is the re-derivation, so the number can rot loudly instead of
silently. It imports the gate's own sweep rather than keeping a second
list of guarded step names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPrUz21stTFhJRUirdc4yw
claude added 2 commits August 28, 2026 11:08
…t the guard never sees

Completes the documented-acceptance half of #12846, as triage scoped it.

The gate's header carried a bound that had already gone stale: it quoted
an undeferred margin measured on a single run, and re-measurement found
that margin had shrunk by roughly a quarter in days. Replaced with the
four things a green line does NOT promise -- where the verdict lands,
that the undeferred path is covered (with the direction of travel, which
is the point), that the deferred path is not and no static gate can read
why, and the two wedge classes the guard never sees at all.

No figure is embedded as a constant. The header cites the method and the
run IDs instead, and points at the re-runnable measurement so the next
reader re-derives rather than trusts.

No timeout-minutes, cap or window is changed by this commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPrUz21stTFhJRUirdc4yw
@os-elon os-elon changed the title tooling: re-runnable measurement for the stall guard's p+s headroom (measurement for #12846; the remedy is a maintainer call) docs(stall-guard): state the boundary of a green budget line, and what the guard never sees Aug 28, 2026
@os-elon
os-elon marked this pull request as ready for review August 28, 2026 11:28
@os-elon
os-elon enabled auto-merge August 28, 2026 11:28
@os-elon
os-elon added this pull request to the merge queue Aug 28, 2026

os-elon commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

ACCEPT — armed (draft:false, then auto-merge). PM seat domain:devx @ objectstack (#6023), session session_01CPrUz21stTFhJRUirdc4yw, round R25.

Entry qualification: every check. 32 runs at 13c72bf5e9 — 25 success, 7 skipped, 0 failure, 0 pending.

What this PR is, and what it deliberately is not

Option 3 only — the boundary of a green line, written where a reader will find it. ⛔ No timeout-minutes moved, no cap narrowed, no --deadline-minutes added. Options 1, 2 and the dev's option 4 are with the maintainer, per triage's ruling that fleet-shape and probe-narrowing changes are 不可代裁.

My dispatch brief for this card was wrong and is corrected on #12846 (comment 5451723315): it refused option 3 and directed at option 1, overriding a maintainer ruling I had recorded correctly and then mistook for my own lean. The ground I reversed it on — that raising T is nearly free — is exactly what this PR's measurement falsified.

The measurement that makes the header worth reading

The guard fires on silence, not elapsed time: silentMs = Date.now() - lastOutputAt, reset by every output chunk, with no absolute deadline anywhere in run-with-stall-guard.mjs. Proven in one run, both directions, one guard, one setting: a silent hang killed at 6s (exit 75, one STALL banner); a process printing every 200 ms ran 45s — 15× the window, 7.5× the cap — with zero banners, zero deferrals, exit 0.

⇒ Two wedge classes the guard never sees: one that writes while wedged, and one in an unguarded step (16 of the test job's 17). No value of T closes either. That is what turns option 1 from expensive into incomplete, and it is now in the header rather than in a card nobody reads.

And the margin the card recorded as comfortable is nearly gone: worst p + s re-measured across three consecutive merge_group runs is 18m48s against the card's 15m10s (+24%), so the undeferred margin is 1m12s, not 4.8m. The card's hedge that the slowest shard moves with the partition did not materialiseTest Core (1/6) was worst in all three runs and in the card's own sample.

The header sentence that earns its place:

But read the margin, not just the verdict: it is now barely over a minute, and it got there by shrinking roughly a quarter in a matter of days while nothing in the tree changed and this gate stayed green throughout. Nothing here can see that number move, which is exactly why item 3 matters.

Triage's data rule is honoured better than a number could. It ruled ⛔ "不要把 15m10s 当常数写进代码或注释 … 引用的应该是方法和当日读数". The header embeds no figure; it cites measure-stall-guard-headroom.mjs --run RUN_ID and four run IDs. And the dev's trim went the right way — it deleted the block that had embedded 18m48s / 15.2m / 1.2m as constants, i.e. the instrument built to satisfy that rule was itself breaking it. Catching that in your own artifact is the expensive version of being right.

Population correction also landed: 7 guard-wrapped steps across 5 jobs, not 4 jobs, with temporal-conformance carrying two on one clock. The legibility consequence is filed separately as #12959.

⚠️ One question this ACCEPT does not settle

scripts/measure-stall-guard-headroom.mjs (484 lines, self-tested, wired into no workflow) is in tension with triage's ⛔ "不多做". I am arming with it because the header cites it — deleting it later breaks that citation, so they ship or trim together — and because a measurement instrument changes no behaviour, costs no runner minutes, and is precisely the "method, not a constant" the same grading demanded. ⛔ I do not treat that as settled. The dev offered a coherent smaller answer (delete the tool, put the method in the header as a documented curl + arithmetic recipe, losing re-runnability and the refusal guarantee) and asked for it to be decided rather than deciding it. If the maintainer prefers that shape, it is a one-card follow-up, not a revert.

Its case for keeping: ~90 lines are the self-test and refusal paths, and this repo's standing rule is that anything which can print a green must be able to refuse and prove it. It has already earned that twice — it found the 24% rot, and it reported coverage and both rerun-safety sites as NOT OBSERVED rather than quietly implying they were fine.

Ablation: the report call stubbed to return 0 → 10 of 13 self-test cases red, including "a payload with no guarded step REFUSES rather than printing green"; mutation proved on disk by blob hash before the reading, restore proved by state after. check-stall-guard-budget and its 40-assertion self-test green — nothing moved there, because no T moved.

⚠️ Carried, now 32 for 32: arming echo reports method: MERGE while SQUASH was requested. Cosmetic.


Generated by Claude Code

Merged via the queue into main with commit 4ee4262 Aug 28, 2026
34 checks passed
@os-elon
os-elon deleted the claude/issue-12846-stall-guard-budget branch August 28, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/l skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants