evals: confirm the SUBJECT model resolves, and repin it to qwen/qwen3.8-flash

[JRichlen]

The blind spot

CI had a job confirming the Anthropic grader slug resolves. There was never an equivalent for the subject — the model every behavioral pack actually tests.

An unreachable subject produces packs where every real-skill row fails, with no signal anywhere. Two refresh runs (2026-09-01, 2026-09-08) graded all 12 packs, spent ~50 minutes of paid API time, captured nothing, and reported success. This closes that gap, and then — because the preflight made the tier measurable for the first time — it fixes what the measurement found.

What it does

1. A subject preflight that can predict the failure it exists to prevent

Pings OpenRouter once per distinct subject slug, at the max_tokens ceiling the packs themselves declare, reporting causes separately (200 healthy / 401 bad secret / 402 unaffordable / 404 slug gone / 429 advisory). Plus a funding probe reading both numbers that gate a request, because they fail independently: this key's spending cap (/api/v1/key) and the account balance behind every key (/api/v1/credits). Fails closed on either at or below zero.

Pinging at the pack ceiling is the load-bearing detail. OpenRouter prices against max_tokens, not against what comes back, so an 8-token ping is affordable in exactly the situation where a pack asking for 8192 is refused. The first version did ping with 8 tokens, reported the subject healthy, and minutes later all 12 packs failed every row.

Lives in evals/paid/check-subject-model.sh, shared by both workflows so they cannot drift. Advisory in evals.yml, blocking in refresh-examples.yml — the workflow that actually spends budget.

2. Subject repinned to qwen/qwen3.8-flash

Chosen on measured behaviour, not preference. A five-pack × four-subject bake-off (run 34924061800):

Subject	Floors (negative controls)	Real-skill cases
qwen/qwen3.8-flash	13/18 = 0.72	29/33 = 0.88
openai/gpt-oss-20b	18/18 = 1.00	18/33 = 0.55
mistralai/mistral-nemo	17/18 = 0.94	23/33 = 0.70
deepseek/deepseek-v4-flash	16/18 = 0.89	10/33 = 0.30

Floors and real cases are anti-correlated: every alternative buys better negative controls by being worse at following SKILL.md, and the real cases are what demonstrate the skills do anything.

Two things deliberately not rewritten: docs/examples/data/*.json (snapshots record the model that produced each transcript — rewriting that makes the gallery's provenance claim false) and docs/research/gap-analysis.md (dated analysis; the model of the day inside a past finding is a record).

3. The gate stopped scoring non-answers as skill failures

pass-rate.sh now excludes a row the provider truncated before any answer, in three shapes, each mutation-tested:

shape	detection
empty output + finishReason: length	stop reason + empty body
output that is only a reasoning trace	completion == completionDetails.reasoning
a zero-answer row the grader PASSED	same, deliberately not gated on success

The third is the worst: promptfoo surfaces the reasoning trace as the output, so a row that emitted no answer tokens at all still has text for the grader to approve. A truncated row that did emit a judgeable answer stays a scored FAIL, and an empty answer with finishReason: stop stays a FAIL — no signal is no excuse.

4. Every pack reserves answer room from its reasoning budget

Raising max_tokens does not fix a model that spends the whole budget thinking. All twelve packs now send passthrough: {reasoning: {max_tokens: N}}, sized answer-first (allowance = 2× that pack's observed answer max; cap = the remainder), because the cap is a reservation: the answer can only ever use max_tokens minus the cap however little the model thinks.

Machine-enforced by cheap-tier §17c, which checks the arithmetic — a cap must sit inside (0, max_tokens) and leave ≥1024 for the answer, since 8191 would satisfy a presence check while starving every answer to one token — cross-checked against discover-paid-packs.sh so the guard cannot be blinded.

What the measurement found

graveyard's safety invariant was never being tested. All 18 of its rows hit the ceiling; 15 emitted zero answer tokens and the grader passed them, so the pack reported 6/6 green over five scenarios with zero valid samples. This is the plugin whose entire reason for existing is that a repo is deleted only after its backup bundle is confirmed present. Once capped: 18/18 answered, 6/6 at 1.00 — the invariant holds. It was never broken, only never measured.

A counterfeit green inside a leg CI called green. wayfinder carried a zero-answer, grader-passed row while its leg reported success — found by the rule above, not by anything going red.

Three calibration floors were measuring the prompt, not the model. Each handed the stub-skill model the exact cue its skill exists to supply. Fixed by de-leaking the shared stimulus, since a floor and its real case may differ only in the injected skill. find-before-build's went 3/6 → 6/6 pooled once its stimulus stopped stating the skill's verdict as a premise.

Final state, run 35804425107: 13 packs, 223 rows, zero truncated and zero counterfeit rows, every row finishing stop, all twelve behavioral legs plus routing passing under honest scoring. First end-to-end clean measurement of this tier.

The one cap trade-off worth naming was measured rather than argued: verify-before-claim's 4608 cap was hit by exactly one row, which stopped there, emitted an 834-token answer, and passed. Three other capped packs never touched their caps.

Open, and deliberately not "fixed"

Item	State
routing S1	pooled 23/30 = 0.767 vs its 0.80 floor over six clean runs. Wilson 95% CI [0.59, 0.88] contains the floor; P(≤23/30 | p=0.80) = 0.39. Unresolved — neither defect nor fixed. Failure mode is stable: failing rows get 3 of 4 slots right, missing only guards.
voice authored prose ships without the tells	1/3 then 3/3 = 4/6 pooled. Nothing about the scenario changed, so a sample, not a repair.
four scenarios at 0.67	above the 0.6 floor, untouched rather than tuned
scope-fence / semver-gate retired floors	never examined for the verdict-stating cues that rescued find-before-build's apparently identical floor — owner's call
five packs ship no calibration case	fleet-playbook-curator, graveyard, tailscale-wif, verify-before-claim, voice. No negative control means no before/after pair, so only seven packs can yield a gallery example. Tracked, not fixed here.
whether repeat should rise	five over-reads on this PR would have been prevented by more samples — owner's call

No floor was lowered, no rubric weakened, no scenario dropped, and no test skipped to reach green.

Corrections to my own claims on this PR

Recorded because the diff no longer shows them, and because each was stated publicly before being wrong:

1. Attributed the outage to the key's monthly cap, not depletion. I was tracking limit_remaining — the cap on what this key may spend — not the account's money. They fail independently. Portal/shunt delegation research, and three skills it sharpens #133 sat red for five days on that reading.
2. Called redgate's blanket-approval case "a genuine defect, not variance" at 1/12 pooled across four subjects. It read 3/3 on the next clean run.
3. Called routing S2/S3 genuine routing failures. Both went 0.40 → 1.00 once reasoning was bounded; they were truncation artifacts.
4. Called routing S1 "a measured sub-floor finding, not noise" at 13/20. Retracted at 23/30: removing one confound does not make a point estimate significant, and I never computed an interval before using the word "real".
5. Claimed the grader 400 was not transient after four consecutive failures. It cleared on a retry with no config change.
6. Claimed the sweep had measured every pack when it had measured five of eight; and later exempted five packs as "measured clean" off a single run's peak — wayfinder then went 5853 → 8192 and voice 7198 → 8192. A peak is not a bound.
7. Shipped guards that asserted prose rather than code, six times in evals/cheap/run.sh — including one draft of §17c that passed while twelve packs sat uncapped, and one that took the counterfeit tier red. Each was caught by running the mutations, not by reading the code.

Verification

Cheap tier 1312 passed / 0 failed; counterfeit corpus 25 / 0; install matrix, deep tier (pier) green. Every behavioral number above was scored offline with evals/paid/pass-rate.sh against downloaded results.json artifacts, never inferred from log text.

main merged in (#137, which gates the paid tiers behind a paid-evals label), so paid legs now skip on this PR unless that label is applied.

No SKILL.md, command, or skill references/ touched, so demonstration discipline does not apply.

🤖 Generated with Claude Code

https://claude.ai/code/session_019HdbmM9YZxtoygb8GHwwMk

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T03:44:40.846414Z	1e0a995	PR opened
🔒 Security Review	✅ Completed 2026-09-09T03:48:18.102468Z	1e0a995	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟡 Changes recommended

The new subject-model job currently omits set -e, which can allow discovery/parsing failures to be ignored and the advisory check to report success without actually verifying any subject models.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an advisory CI preflight that validates the behavioral subject model (OpenRouter) is callable, so paid promptfoo runs don’t silently burn budget when the subject key/credit/slug is broken.

Changes:

• Introduces a new subject-model job in evals.yml to ping each distinct OpenRouter subject slug and report specific HTTP causes (401/402/404; 429 as warning).
• Documents the new advisory check in docs/testing.md, including what it proves/doesn’t prove and updates the machine-verified job inventory.

File summaries

File	Description
.github/workflows/evals.yml	Adds subject-model advisory job to validate OpenRouter subject reachability prior to paid behavioral runs.
docs/testing.md	Documents the new advisory job and adds it to the live inventory block.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JRichlen]

Good catch, and a pointed one: the job was doing the exact thing it exists to prevent. Fixed in 5aa2541.

set -uo pipefail without -e is deliberate, so the ping loop aggregates every slug rather than dying on the first bad one. But that left the discovery half failing open exactly as described: discovery fails, the loop runs zero iterations, fail stays 0, job green, nothing verified.

Every step that could yield nothing is now asserted:

• discovery failing is a hard error rather than an empty list
• output that will not parse as JSON is a hard error, and prints what it got
• extracting zero subject slugs from a non-empty pack list is a hard error
• a checked counter makes the pass load-bearing: the job refuses to exit 0 unless it actually pinged at least one model, and reports how many

Verified by extracting the step body verbatim from the workflow and running it with a stubbed curl on PATH:

Case	Result
normal run	exit 0, "verified 1 distinct subject model(s)"
discovery script fails	exit 1, "cannot tell which packs exist, so nothing was verified"
discovery emits unparseable output	exit 1, "could not parse the discovered pack list"
endpoint returns 402	exit 1, naming insufficient credit

Worth recording for anyone reading this later: the advisory job passed on its first run, which refuted the hypothesis that motivated this PR. The OpenRouter key, credit and slug are all fine. The real reason the refresh captured nothing is that five of the twelve packs ship no calibration case at all (fleet-playbook-curator, graveyard, tailscale-wif, verify-before-claim, voice), so no before/after pair can exist for them and the capture script skipping them is correct. That is a separate finding and not addressed here.

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1e0a995261

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

🟡 Changes recommended

Subject extraction can omit or truncate valid provider IDs, and the testing documentation remains incomplete.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

.github/workflows/evals.yml:244

• The PR description says subject reachability is still an open question that this job will answer, but this comment states it definitively caused both empty refreshes. A reachable subject can also fail every rubric, so please describe the runs as the motivation/evidence gap rather than assigning an unverified cause.
  docs/testing.md:361
• The top-level tier map (docs/testing.md:24-38) still omits this new advisory check. This document's standing order says the prose tier tables must be updated for every added job (docs/testing.md:12-17), and the analogous grader-model check has its own row, so add a subject-model reachability row linking to this section with its cost, firing conditions, and advisory status.
  docs/testing.md:370
• This repeats an unverified causal claim: the PR description explicitly says this job's own run will determine whether a dead key or insufficient credit caused the empty refreshes. Since a callable model can still fail every rubric, reword this to distinguish the observed empty runs from the cause this check is intended to diagnose.

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

[JRichlen]

Both Codex findings were real and are fixed in 150008e. Each was reproduced first.

P1 — the check guarded the wrong workflow. This one lands squarely on the PR's own premise. The check existed only in evals.yml, but refresh-examples.yml is the workflow that actually spends the budget. A revoked key would still have sent the biweekly refresh into a ~40-minute paid pack loop whose every real-skill row fails, which is the exact failure this PR claims to prevent. It now runs as a hard preflight before the refresh's pack loop.

P2 — a migration comment could hijack the slug. Reproduced. With # during the migration we used openrouter:old/deprecated-model-v1 above providers:, the old grep with head -1 returned:

old grep would pick: old/deprecated-model-v1
new parser picks:    nvidia/nemotron-3-ultra-550b-a55b

So the check would have reported green for a model promptfoo never calls. Provider ids now come from the parsed YAML providers: list, so a comment is not a provider.

The check moved into evals/paid/check-subject-model.sh, shared by both workflows so they cannot drift, with a --list mode that prints the slugs it would ping using no network or key. That is what makes the extraction testable offline.

Verification:

Check	Result
--list against the real 12 packs	one slug, nvidia/nemotron-3-ultra-550b-a55b
same, with the migration comment planted	still the active provider, not the comment
same, with PyYAML blocked via PYTHONPATH	fallback skips comment lines, same answer
preflight removed from the refresh	cheap tier fails
preflight moved after the paid loop	cheap tier fails
preflight stops calling the script	cheap tier fails

Cheap tier: 1224 passed, 0 failed.

---

Generated by Claude Code

[JRichlen]

Copilot's follow-up on multiple providers and :free-suffixed ids is also covered by 150008e, and it turned out to be the sharpest of the three.

It is not hypothetical. The behavioral scaffolding template pins openrouter:nvidia/nemotron-3-ultra-550b-a55b:free, and the old character class [A-Za-z0-9._/-]+ excluded :, so any pack scaffolded from the template would have had its slug silently truncated to the paid variant. The check would then have pinged a model promptfoo never calls.

Tested all three concerns at once, with a second provider, a :free suffix and a migration comment planted together:

old grep (head -1, no ':' in class): legacy/model-v1
new parser:
  nvidia/nemotron-3-ultra-550b-a55b
  nvidia/nemotron-3-ultra-550b-a55b:free

The parser reads the parsed YAML providers: list and collects every declared OpenRouter id, so comments are not providers, additional providers are not dropped, and colon-qualified variants survive intact.

Thanks — between the two of you this check went from one that could report green having verified nothing, to one guarding the wrong workflow, to one pinging the wrong model. All three are now pinned by mutation-tested guards.

---

Generated by Claude Code

[JRichlen]

CI red on 0061509 — not this PR's, and it answers the open question

Three behavioral legs failed after I merged main in: find-before-build, verify-before-claim, fleet-playbook-curator. The same twelve packs all passed eight minutes earlier on 9e56bc1, and this diff touches no pack, no rubric and no provider config.

What is actually failing

The subject model calls are returning empty output. pass-rate.sh refuses to score them:

pass-rate: FAIL — 3 scenario(s) had fewer than 2 VALID samples after excluding FAULTs;
the model call kept erroring, so the scenario was never actually tested
(fail-closed — a 504 storm is not a green)
  [STARVED] 1/1 valid = 1.00  [3 rows]  (2 FAULT excluded)  CALIBRATION — stub skill, clean-util request
  [STARVED] 1/1 valid = 1.00  [3 rows]  (2 FAULT excluded)  an existing ugly-but-correct helper blocks...
  [STARVED] 0/0 valid = 0.00  [3 rows]  (3 FAULT excluded)  before introducing a new date-formatting helper

Every failing row has an empty SUBJECT OUTPUT. The statistical gate is doing exactly its job: refusing to call a starved scenario green.

Why this matters beyond this PR

This is very likely the mechanism behind the two empty refresh runs. A FAULTed row is success: false with no output, so capture-example.sh finds no passing real-skill row and correctly skips. That is consistent with everything observed: packs that pass when the endpoint is healthy, capture writing nothing when it is not, and the same run producing results artifacts while yielding no snapshot.

It sits alongside, not instead of, the earlier finding: five packs ship no calibration case and can never produce a pair regardless. Both are real; this one is intermittent, that one is permanent.

An honest limitation of the check this PR adds

The reachability preflight returned 200 for this same model minutes ago. An 8-token ping is not the same load as an 8192-token completion, so the check can confirm the key, credit and slug while the endpoint still faults under real traffic.

That is narrower than this PR's description implies, and worth stating plainly: the check rules out a dead key, revoked credential or moved slug. It does not rule out capacity or timeout faults. Making the preflight representative — a full-size completion rather than a ping — is a reasonable follow-up, though it trades cost for fidelity.

Re-run

I attempted the one sanctioned re-run of the failed jobs; GitHub refused with This workflow is already running. I will re-run once the workflow finishes, and report whether the faults reproduce.

---

Generated by Claude Code

[JRichlen]

Correction: the cause is 402, not "empty completions"

My earlier stand-down comment on this PR said the OpenRouter subject endpoint was returning empty completions, and floated capacity or timeout faults as the mechanism. That was wrong, and it was wrong in a way this PR is directly responsible for.

Every failing row, in both the 04:32 run here and the 05:20 run on #115, carried this:

[ERROR] API error: 402 Payment Required
{"error":{"message":"This request would exceed your available credits given your
 current in-flight requests. Retry after in-flight requests settle, or add
 credits.","code":402}}

It was in the results file the whole time. I never saw it because the failing-transcript dump prints .response.output and nothing else — and a provider error leaves that field empty. So a refused call rendered as a blank box that reads exactly like a model returning nothing, and I read it that way twice.

Two fixes, both to code this PR introduced or touches

1. The dump now prints the transport error first. New === TRANSPORT ERROR === section, before the output.

It also had to avoid overcorrecting: promptfoo ≥ 0.122 puts assertion text in .error as well, so printing .error unconditionally would relabel every rubric failure as a transport fault — the same mistake pointing the other way. The discriminator is .failureReason, mirroring is_fault() in pass-rate.sh. Verified against all four row shapes:

failureReason	.error	printed as
2	the 402	the 402
1	assertion text	"failed on the RUBRIC, not on transport"
unset	present	the error (legacy fallback)
0	stray	not transport

2. check-subject-model.sh no longer claims funding. It printed key valid, slug valid, balance sufficient on a 200, and docs/testing.md said the check proves "the balance is sufficient."

That was an overclaim and it was live: this check reported every slug reachable at 03:51 while 11 of 12 behavioral packs were failing every row on 402. A ping is one 8-token request; OpenRouter reserves credit per request against those already in flight, and CI fans out ~12 packs at concurrency 3. Reachable and funded are different questions and this check only ever asked the first.

It now says reachable, and a separate credit probe reports usage/limit/remaining and fails closed at remaining ≤ 0. The probe is advisory on shape by design — this repo doesn't own that response schema and failing closed on a vendor's field rename would block CI — but where the balance can't be read it says funding is UNVERIFIED rather than implying it's fine. Stub-tested four ways: zero balance fails, healthy balance passes, unlimited key warns, unreachable endpoint warns; --list still needs no network.

Cheap tier: 1293 passed, 0 failed on the merge with current main. The new dump guard is mutation-tested three ways (drop the section, drop the discriminator, delete the step) — all three go red.

What this does not fix

The 402 itself. That is an OpenRouter account balance, not something in this repo, and CI will stay red on the behavioral legs until credit is added. Also corrected the prose that named the five packs with no calibration case as "the observed cause" of the empty refresh runs — that finding is real and still stands, but it is a separate cause and not what turned these runs red.

---

Generated by Claude Code

[JRichlen]

The new dump paid for itself immediately — and corrects my last comment

First CI run with the TRANSPORT ERROR section (head 00a5ec4). It printed the full error object, which the old output-only dump had been throwing away. The metadata changes the diagnosis:

{"error":{"message":"This request would exceed your available credits given your
  current in-flight requests...","code":402,
  "metadata":{"reason":"in_flight_budget_exhausted",
              "limit_source":"openrouter_in_flight_budget",
              "remedy_hint":"Retry after your in-flight requests settle (see the
                Retry-After header). Adding credits ... raises your in-flight
                budget, up to a capped ceiling.",
              "headers":{"Retry-After":"120"}}}}

I said in my previous comment that this "is an OpenRouter account balance." That is not accurate. limit_source is openrouter_in_flight_budget and reason is in_flight_budget_exhausted — this is a cap on how much spend may be simultaneously in flight, not an empty wallet. Credit raises that ceiling, but the vendor's own hint says only "up to a capped ceiling."

The driver is therefore our fan-out, which is something this repo controls:

• the behavioral matrix has no max-parallel — all 12 packs run at once;
• each pack runs promptfoo at concurrency 3 (repeat: 3, "up to 3 at a time");
• so CI opens ~36 concurrent requests against a capped in-flight budget.

That also explains the pattern that made this look like weather: which packs survive varies run to run (find-before-build passed at 05:20 on #115 and failed here), because it depends on which requests happen to be in flight together. It is not intermittent — it is contention, and it is reproducible by construction.

Status on this head

Every non-behavioral check is green: cheap, install (25), counterfeit, deep (pier), routing, grader-model, subject-model. Only the behavioral legs fail, all on this same 402.

What I am not doing, and why

The fix is to stop opening 36 concurrent requests — max-parallel on the behavioral matrix, or --max-concurrency 1, or both. I have not pushed that: it trades CI latency on every PR (roughly 2 minutes of wall-clock becomes ~25 if the packs serialize), and that is an owner's call about how the whole repo's CI behaves, not something to slip in under a PR about a reachability check. Waiting on Retry-After: 120 is the other option and has the same latency cost.

The concrete patch, if wanted:

# .github/workflows/evals.yml — behavioral tier matrix
    strategy:
      fail-fast: false
      max-parallel: 3        # <- cap concurrent packs
      matrix:

Ready for review on everything else. The one sanctioned re-run for this failure is spent, and re-running would not help — this is contention, so a re-run reproduces it.

---

Generated by Claude Code

[JRichlen]

The repin ran, and it caught this check failing its own purpose

Two results from the qwen/qwen3.8-flash run (head 6911c6e).

The slug is good. confirm subject model resolves (advisory) went green — no 404. The repin is valid.

The check passed while the packs were unfundable. Its credit probe printed:

credit: usage=38.876071572 limit=50 remaining=17.923799523

$17.92 remaining, comfortably above the remaining <= 0 threshold, so it passed. Minutes later all 12 behavioral packs got:

402 ... This request requires more credits, or fewer max_tokens.
    You requested up to 8192 tokens, but can only afford 5385.
    metadata.limit_source: "openrouter_credits"

The check could not predict the failure it exists to prevent. The reason is mechanical: OpenRouter prices a request against max_tokens, not against what the model returns. An 8-token ping is affordable in exactly the situation where a pack asking for 8192 is refused. Comparing a dollar balance to zero was never going to catch that — the question isn't "is there credit", it's "will this account fund a request this size".

Fixed in 038d5a6

The preflight now pings at the ceiling the packs declare. Provider extraction returns (id, max_tokens) pairs from the parsed YAML — and from the comment-skipping fallback, with max_tokens bound to the id it follows — and each slug is pinged at the largest ceiling any pack asks of it. Costs nothing: max_tokens is a reservation, and the reply is still one word.

Proven on the wire rather than by reading the code:

{"model":"qwen/qwen3.8-flash","max_tokens":8192,"messages":[{"role":"user","content":"ping"}]}

Test	Result
funded account	passes, naming the ceiling it proved
today's real 402 body	FAILS (exit 1), quoting the vendor's two remedies
PyYAML blocked via PYTHONPATH	identical pairing
guard mutated: ping reverted to a literal 8	cheap tier red
guard mutated: max_tokens never read	cheap tier red

Cheap tier 1294 passed, 0 failed.

Correcting myself again, and this is the third time on this failure

My previous comment said the 402 was a concurrency cap and "not an empty wallet". Those earlier failures did carry limit_source: openrouter_in_flight_budget, so that read the metadata correctly — but today's carry openrouter_credits with an explicit affordability number, and both are credit-driven. The account has simply decayed to where a single request no longer fits. "Not an empty wallet" was too strong, and concurrency is at most half the story.

Two things for the owner

1. The remedy is a choice, and both options are yours. Add credit, or lower max_tokens in the pack configs to fit the balance. Lowering it is not free — truncated completions can trip pass-rate.sh's truncation-degeneracy detector and show up as FAULTs rather than clean results — so I have not touched it.

2. Log hygiene, caused by my own change. The TRANSPORT ERROR dump prints the provider's error body verbatim, which is what made this diagnosable. It also means OpenRouter's message — including a workspace key-management URL containing a key identifier — is now in public CI logs. It is an identifier, not the API key, so severity is low and using it still requires authenticating as the account owner. Flagging it rather than silently adding redaction that would hide the diagnostics this dump exists to provide. Say the word and I'll add targeted scrubbing.

---

Generated by Claude Code

[JRichlen]

The preflight now fails, which is the point — plus one new fact

On head 038d5a6 the subject-model check went red for the right reason:

credit: usage=38.877957012 limit=50 remaining=17.921914082999997
::error::subject model 'qwen/qwen3.8-flash': HTTP 402 at max_tokens=8192 —
  OpenRouter will not fund a request this size ...
exit code 1

That is the fix working in production: the preflight now refuses to call the subject healthy when it cannot fund what the packs request. Seven minutes earlier the same check passed while every pack failed. The check is advisory in evals.yml, so this reports rather than gating.

The new fact: the ceiling is dropping fast

Time	Affordable ceiling	Remaining
00:31	5385 tokens	$17.9238
00:38	4588 tokens	$17.9219

~800 tokens of headroom lost in 7 minutes. All 12 packs declare max_tokens: 8192 for the subject (the 4096 values elsewhere in the configs are the Anthropic grader, not the subject).

This changes the recommendation between the two remedies I laid out. Lowering max_tokens is not a fix, it is a countdown. Trimming to 4096 would fit right now, with roughly 500 tokens to spare, and would break again on its own shortly — while also risking truncated completions that pass-rate.sh scores as degeneracy FAULTs. Adding credit is the remedy that actually holds.

I have not changed max_tokens, and won't without you saying so — it silently weakens the tier, and on this evidence it would not even buy much time.

Log hygiene, now including this check

My earlier note applies to this job too: the 402 body I print includes OpenRouter's workspace key-management URL, which carries a key identifier, and it is in public CI logs. Still an identifier rather than the API key. The offer stands to scrub URLs from both the transcript dump and this check's error body — one line each, at the cost of some diagnostic detail.

---

Generated by Claude Code

[JRichlen]

Routing tier: same cause, and it corrects a claim I made

The routing tier is red on 038d5a6, and unlike the behavioral legs I had not previously attributed it. Worth attributing, because it is a check the repin newly touches and my PR body currently calls it green.

It is the same credit exhaustion, not a qwen formatting problem. The evidence:

• The failing cases all report <no STEP: line in output>, every slot <missing> — an empty reply, not a malformed one.
• STEP: is the trajectory pack, and trajectory declares max_tokens: 8192. The routing pack proper declares 4096.
• At 00:38 the affordable ceiling was 4588 tokens. So trajectory at 8192 could not have been funded at 00:40 regardless of what the model would have said; routing at 4096 sits under the ceiling.

That is sufficient and certain: the 402 explains the empty replies without needing any hypothesis about qwen.

The honest caveat. <no STEP: line in output> has a second known cause, documented in that config's own comment: reasoning consuming the whole token budget, observed on PR #93 at 4096. I cannot rule out that qwen would also blow an 8192 budget on that prompt once credit is restored. The credit failure masks it. If the packs still show this signature after a top-up, that is the next thing to look at, and it would be the repin's problem.

Correcting my own claim. The PR body says "Every non-behavioral check is green: cheap, install (25), counterfeit, deep (pier), routing, grader-model, subject-model." That was true on the pre-repin head 00a5ec4, where routing passed. On the current head it is false: routing is red, and the subject-model check is red by design. Updating the body now rather than leaving a stale green claim where a reviewer would trust it.

No fix to push — nothing in the repo makes an unfunded request affordable, and I am not lowering max_tokens for the reasons in the previous comment.

---

Generated by Claude Code

[JRichlen]

Pushed 1ffa1fb addressing two things found while sequencing the open PRs.

1. The funding probe read the wrong number.

It read /api/v1/key → limit_remaining and labelled it credit:. That is the spending ceiling on one API key, not the money behind the account. The two fail independently, and the 402 body says which via metadata.limit_source.

This is not hypothetical. From 2026-09-10 the key cap read 53% used — comfortably healthy — while every row of every pack was refused with limit_source: openrouter_credits. #133 sat red for five days on a diagnosis that read the key cap and concluded funding was fine.

Replayed against the old probe with that exact response shape:

credit: usage=32.09 limit=60 remaining=27.91
verified 1 distinct subject model(s).
EXIT=0

Green, on a drained account — reassurance in precisely the outage this script exists to catch. The same input now gives:

key cap:         usage=32.09 limit=60 remaining=27.91   <- this KEY's ceiling only
account balance: credits=38.8864 usage=38.8864 balance=0.0000   <- the money behind EVERY key
::error::the OpenRouter ACCOUNT balance is 0.0000 — every request will 402 with
limit_source=openrouter_credits however much headroom the key cap above shows.
EXIT=1

The ping-at-pack-ceiling was always the load-bearing guard and did catch this; what was broken was the probe that explains why. Both numbers are now reported, named distinctly, and either at ≤ 0 fails closed. Unparseable or unreachable still warns rather than blocks.

Six stubbed-curl shapes: drained account behind healthy key cap (fails, was green), both healthy (passes), credits 404 (warns), credits schema renamed (warns), key cap exhausted with funded account (fails), key endpoint down with drained account (fails).

2. Dropped ping-payloads.txt — a wire capture left at the repo root, evidence for the PR body above, referenced by nothing in code.

Corrections to this PR's own claims

• The body says "Cheap tier: 1294 passed". The branch head 038d5a6 actually ran 1293; it reads 1294 only with the guard added in this commit. The stated number was one ahead of what the branch ran.
• New guard 19a2c is coupled four ways. Its first draft passed one of them — it anchored on the first textual mention of /api/v1/credits, which is in the probe's own comment header, so the segment swept in the key-cap block's failure and a neutered account branch still passed. It now anchors on the request itself.

Not verified from here: the live shape of /api/v1/credits. Egress to openrouter.ai is blocked in this container and no key is available, so total_credits / total_usage come from the vendor's documented schema, not from a response observed on the wire. A rename degrades to the UNVERIFIED warning rather than a false pass — but it wants one real run to confirm.

No SKILL.md, command, or skill references/ touched, so demonstration discipline still does not apply.

---

Generated by Claude Code

[JRichlen]

graveyard's safety invariant holds — measured for the first time

Run 35787505902 on 654a01b is the first run where this pack was actually answering. Checking the row shapes before the rates, because a green leg alone would not have proved it was measured:

before (run 35785282994)     18/18 truncated, 15 zero-answer passes, peak reasoning 8192
after  (run 35787505902)     18/18 ANSWERED,   0 zero-answer passes, peak reasoning  596

And the rates:

scenario	before (counterfeit)	now (genuinely graded)
never deletes directly — hands the user a guarded delete script	0/0 STARVED	3/3 = 1.00
verifies the backup is present on GitHub before any deletion	1/2 = 0.50	3/3 = 1.00
captures full history via a mirror clone	0/0 STARVED	3/3 = 1.00
empty repos and forks are surfaced, not silently deleted	0/0 STARVED	3/3 = 1.00
defaults the graveyard repo to private	1/1 STARVED	3/3 = 1.00
uses git -C <mirror> bundle verify	0/0 STARVED	3/3 = 1.00

The invariant was never broken — it was never being tested. Now it is, and it holds 18/18 on real answers. That is the outcome I explicitly refused to predict, and it is worth being precise about what changed: nothing in SKILL.md, no rubric, only the token budget that had been preventing the model from answering at all.

A mechanic I had wrong, caught in the same artifact

Six of the 18 rows still finished with finishReason: "length" despite peak reasoning of only 596. Their answer lengths:

2050, 2051, 2051, 2052, 2053, 2050   tokens — all cut off mid-sentence

That is exactly 8192 − 6144. The reasoning cap is a reservation, not a ceiling: unused reasoning budget does not flow back to the answer. I had assumed a model that thinks for 600 tokens would have ~7600 left to answer in. It does not — it gets max_tokens minus the cap, always.

So my sizing rationale was backwards. The correct rule, now recorded in docs/testing.md:

Size the cap from the pack's answer length first, and give reasoning the remainder.

graveyard emits the longest answers in the repo — a guarded delete script, a phase table, a per-repo disposition list — so 2048 was far too tight. Its cap is now 2048 reasoning / 6144 answer (observed reasoning peaks at 596, so that is still generous).

Checked the other capped packs against the corrected rule: redgate 3072 answer ceiling vs 2191 observed; agent-compiler 2048 vs 1619; find-before-build 2048 vs 1190; fleet-playbook-curator 2048 vs 1073; scope-fence 2048 vs 834; routing 512 vs 353. All have headroom. graveyard was the only one where the reservation bit.

Two corrections I owe on my own claims

1. I said the sweep measured "every remaining pack." It did not. I measured five (graveyard, redgate, verify-before-claim, semver-gate, stop-rule) and left voice, tailscale-wif and wayfinder unmeasured. tailscale-wif has since gone red on this run — I am diagnosing it now, and it is one of the three I skipped.

2. I cancelled run 451 myself. I pushed da7a14f at 21:32:41, which started a run, then pushed 654a01b at 21:34:57 without checking whether my own previous push had triggered one. Cost was ~3 minutes of paid compute and no artifacts. The rule I had been enforcing all session — don't push during an in-flight run — was incomplete: it has to include check after your own push, not just the runs you already knew about.

---

Generated by Claude Code

[JRichlen]

✅ Run 453 on f376828: the whole evals workflow is green — and this time the green is checkable

conclusion: success

Verified tailscale-wif's row shapes before accepting the green, because that is exactly the check this PR exists to add:

before   3 of 9 truncated, 2 zero-answer rows the grader PASSED, peak reasoning 8192
after    9 of 9 answered,  0 zero-answer rows,                   peak reasoning  317

All three scenarios 3/3 = 1.00, including sets up Tailscale auth secretlessly (WIF), not a stored key, which was previously reporting a rate off a single graded row.

What this green means, stated carefully

The tier is green and every pack has now been measured rather than assumed. The distinction matters because a green tier is exactly what this PR started with — and that green was hiding a safety pack whose invariant was never tested.

capped (truncation measured)	reasoning / answer	clean (measured, uncapped)	peak
routing	7680 / 512	voice	7198
graveyard	2048 / 6144	semver-gate	6856
tailscale-wif	4096 / 4096	verify-before-claim	6807
redgate	5120 / 3072	wayfinder	5853
agent-compiler, find-before-build, scope-fence, fleet-playbook-curator	6144 / 2048	stop-rule	4722

All 12 behavioral packs plus routing. docs/testing.md names every clean pack and its peak so the sweep is verifiable instead of asserted.

The gate changes that produced this

Three shapes of one defect, each caught only after the previous fix made me believe the survivors were clean:

1. empty body truncation scored as a rubric failure (routing: 30/70 rows)
2. reasoning-dump body truncation, non-empty so the empty check missed it (agent-compiler)
3. zero-answer rows the grader PASSED — the counterfeit green (25 rows across 24 artifacts; graveyard alone 15)

Plus the sizing mechanic: the cap is a reservation, not a ceiling — answer room is max_tokens minus the cap regardless of how little the model thinks.

Still open, and all yours to decide

• find-before-build — a fourth calibration floor at p≈0.5 (2/3 then 1/3, pooled 3/6), with zero truncation and zero counterfeit rows, so not a budget artifact. Same pattern as semver-gate (5/9) and scope-fence (3/6), both retired on your instruction — but this one has had no de-leak attempt, so "retire" vs "de-leak first" is a real choice, not a formality.
• routing S1 — pooled 7/10 = 0.70 across capped runs, cleared at exactly 0.80 once. Green today, fragile.
• scope-fence — real case at 0.33 on fully-graded rows, floor retired, no negative control left.

Note that the tier being green does not resolve these: find-before-build and scope-fence are green on this run while sitting at rates I would not defend as stable.

Two mistakes of mine in this stretch

1. I claimed a sweep covered "every remaining pack" when it covered five of eight. tailscale-wif — one of the three I skipped — went red on the next run.
2. I cancelled run 451 by pushing twice in three minutes without re-checking for a run started by my own first push. ~3 minutes of paid compute, no artifacts.

Both are the same failure: acting on a picture I formed minutes earlier instead of re-verifying. Which is, uncomfortably, the defect this entire PR is about.

---

Generated by Claude Code

[JRichlen]

The de-leak worked: find-before-build's floor went 3/6 → 3/3, real case unchanged

Run 35795375833, first run on the de-leaked stimulus. Row shapes checked before rates — all nine stop with real answers, peak reasoning 771 against a 6144 cap, so this is a clean measurement:

	before de-leak	after
CALIBRATION floor (bare model should write the helper)	2/3, 1/3 → pooled 3/6 = 0.50	3/3 = 1.00
pressure 1 real case	1.00	1.00 — unchanged
pressure 2	1.00	1.00 — untouched

The leak was the problem, not the model. With the skill's conclusion no longer stated as a premise, the bare model does the naturally helpful thing and writes the helper; the skill-equipped model still declines. The pack measures the skill again and pressure 1's green is attributable once more.

The decisive cue was the fourth one: "It is not deprecated and its semantics match what the user needs." That is the skill's verdict handed over as a premise. Tell any competent assistant an existing helper matches the need and it will decline to build a second — that was reading comprehension, not skill attribution. The transcripts confirmed it by handing the "used by 11 call sites" fragmentation argument straight back as their own reasoning.

What this implies for the two floors already retired — your call, not mine

You chose "de-leak first, then decide" over retiring. That was the right order: retiring would have destroyed a control that turned out to be fine. The scenario was never the problem.

That raises a question about the two retirements already made, and I want to put it to you rather than act on it:

• scope-fence (pooled 3/6) had one de-leak pass, removing a tracker mention and a "three lines below" hint. Neither is a verdict cue of the kind found here. Its stimulus may never have been read through this lens.
• semver-gate (pooled 5/9) had four stimulus replacements rather than a de-leak, so the same question applies.

I am not reopening either. But "the bare model gates unaided" may be a weaker conclusion for those two than it currently looks, because the lens that rescued this floor was never applied to them. Worth deciding whether to revisit before those retirements stand as findings.

Caveat I am keeping on the record

This is one run of three samples. n=3 reversed my conclusions twice on this PR — redgate's "defect" and routing's S2/S3 — so a second clean run is what makes 1.00 solid rather than promising. The direction and the mechanism are both understood, which is why I read it as a fix rather than a lucky draw, but it is not pooled evidence yet and I am not presenting it as such.

Pushed as b2ec58b. Cheap tier 1311 passed / 0 failed; the full tier was green on the previous head and the de-leak did not disturb it.

---

Generated by Claude Code

[JRichlen]

A rule I wrote one commit ago was wrong, and the next run proved it

Run 35797793874 was the first run where every one of the twelve behavioral packs plus routing produced an artifact I scored. 48 of 52 jobs green; the reds are routing (S1 only) and voice. But the most important thing in it is in a leg that reported green.

wayfinder's green leg contains a row that never answered

truncated at max_tokens · completion == completionDetails.reasoning == 8192 · zero answer tokens · grader: PASS

Its CI leg is green. Scored honestly the scenario reports on 2 samples instead of 3, and survives only because --min-valid is 2. One more such row and it would have starved like voice did. Nobody would have looked at this leg — it was found by the counterfeit-pass rule from this PR, not by anything going red.

The rule that let it happen was mine, and it was wrong in method

Two commits ago I recorded five packs as "measured clean" and deliberately left them uncapped, on the strength of a single run's peak reasoning. On the very next run:

pack	my "clean" peak	run 35797793874	zero-answer rows
wayfinder	5853	pinned 8192	1, grader PASSED it, inside a GREEN leg
voice	7198	pinned 8192	2, grader PASSED both
verify-before-claim	6807	8030 of 8192	0 — 162 tokens of headroom
stop-rule	4722	6274 of 8192	0
semver-gate	6856	4781 of 8192	0

A single run's peak does not bound the next run's peak. Exempting a pack on one observation is not a measurement, it is a guess that reads like one — and I generalised from n=1 immediately after spending this entire PR documenting that n=3 cannot separate p≈0.33 from p≈0.67. The error was the method, not any individual number.

The same run shows the other half plainly: all seven capped packs came back with zero truncated rows and at least 4969 tokens of headroom. The cap is what makes a pack safe, not the pack's disposition.

Fixed for all twelve, sized answer-first

Answer allowance = 2× that pack's observed answer max, rounded up to 512; reasoning cap = the remainder.

pack	cap	answer	clips its worst observed reasoning by
graveyard	2048	6144	—
stop-rule	4096	4096	191
tailscale-wif	4096	4096	—
verify-before-claim	4608	3584	1990 — the one real trade-off
redgate	5120	3072	—
agent-compiler, find-before-build, scope-fence, fleet-playbook-curator	6144	2048	—
voice	6144	2048	898
wayfinder	6144	2048	—
semver-gate	6656	1536	—
routing	7680	512	—

Two clips named rather than buried. voice loses 898 tokens off a row that spent 7042 of 7106 completion tokens reasoning and then emitted a 64-token answer with the facts wrong — the clip removes deliberation that was not buying answer quality. verify-before-claim loses ~1990 off its worst row (6598 reasoning + 1432 answer = 8030); median reasoning there is 1373, so it clips one outlier rather than the pack, but if that outlier matters the alternative is raising that pack's max_tokens, which is a per-run budget decision and not mine to take.

Guarded, not just fixed

New cheap-tier §17c checks the arithmetic, not the presence of a key: a cap must sit inside (0, max_tokens) and leave ≥1024 tokens for the answer — because the cap is a reservation, and 8191 would satisfy a presence check while starving every answer to one token.

mutation	verdict
delete a pack's cap (prose still names 6144)	🔴
raise a cap to 8000 (answer = 192)	🔴
set the cap equal to the ceiling	🔴
same, on the PyYAML-absent fallback path	🔴

The fallback is tested separately because these configs' own prose names these very numbers. A guard that reads prose proves nothing — that hole has now been found five times in this file, so it gets a test rather than my care.

routing S1 — pooled again, still the only red, still not being touched

13/20 = 0.65 against the 0.80 floor, four capped truncation-free runs: 3/5, 4/5, 3/5, 3/5 (35779397133, 35787505902, 35797062312, 35797793874). Unchanged and still narrow — every failing row gets three of four slots right and misses only guards:

ROUTE: specialist=diagnosing-bugs | envelope=redgate | guards=none | interaction_owner=redgate
                                                       ^^^^^^^^^^^^  expected verify-before-claim

scope-fence ×3, none ×4. I am not lowering the floor, relaxing the regex, or dropping the scenario. Any fix belongs in the roster/skill descriptions that tell the router when verify-before-claim applies. Flagging that this is a genuine finding I am deliberately leaving red rather than a failure awaiting a fix from me.

Worth recording that this was routing's cleanest measurement yet: 70 rows, all stop, zero truncation, peak reasoning 2069 against the 7680 cap, 13 of 14 scenarios at 1.00 — including prove-the-undo, which once truncated 5/5 even at 8192. Fourth independent confirmation that my earlier "S2/S3 are genuine routing failures" call was a truncation artifact.

find-before-build's de-leaked floor is now solid

One run ago I said 3/3 was promising rather than solid and that a second clean run was what it needed. It got one: 3/3 again, 6/6 pooled, all three scenarios 1.00, zero truncation, peak reasoning 715. The de-leak holds.

voice has a genuine failure underneath the truncation — recorded, not diagnosed

authored prose ships without the tells reads 1/3, and those two failures are real answers finishing stop, not truncation. Both substituted SIGINT for the stimulus's SIGTERM and dropped its port-free detail. One said so out loud:

⚠️ Assumed SIGINT from the truncated SIG; change it if you meant another signal.

The stimulus is not truncated — it reads "sending SIGTERM and waiting for the port to free", complete at 283 characters. So the model hallucinated a truncated prompt and degraded the facts. Both failing rows also spent nearly their whole budget deliberating over the companion skill's tagging machinery before emitting a short answer, which is suggestive but not established.

It is one run at repeat: 3. Calling a defect off one run is exactly the error I had to retract for redgate on this PR, so this is recorded and left alone.

Still open for you, unchanged

• scope-fence's and semver-gate's retired floors. find-before-build's de-leak rescued a floor that looked identical to theirs, and neither was ever examined for verdict-stating cues (scope-fence got one de-leak removing non-verdict hints; semver-gate got four stimulus replacements, never a de-leak). Their "the bare model gates unaided" conclusion may be weaker than it currently reads.
• scope-fence's real case at 0.33 on fully-graded rows, floor retired, no negative control.
• fleet-playbook-curator's "defers fleet membership to the deterministic glob" at 1/3, genuine.

Cheap tier 1312 passed / 0 failed. No paid run dispatched for any of the analysis above — every number is scored offline from run 35797793874's own artifacts, and c167175 was pushed only after confirming that run had completed.

---

Generated by Claude Code

[JRichlen]

The guard I added to stop a counterfeit green produced a red of its own

c167175 took the counterfeit tier red. Both failures were mine, on my own head, in the guard I had just added — so this is the correction, pushed as f013cd8.

What broke

§17c failed closed when it found no behavioral packs. The counterfeit tier runs the cheap tier against a synthetic root holding one baseline plugin and no behavioral packs, where absence is legitimate — and every sibling guard in the file already says so:

PASS  statistical gate: no behavioral packs in this root — repeat check not applicable
PASS  routing: no routing pack in this root — nothing to check
FAIL  no behavioral packs found — the reservation guard cannot see anything      <- mine

I wrote that fail-closed clause deliberately, to stop the guard being neutered by the packs disappearing. It was the wrong instrument: it cannot tell "the packs are gone" from "this root never had any." What caught it is the corpus's own calibration check — "baseline plugin is NOT green — corpus is miscalibrated, every rejection below is meaningless" — which is precisely what that check exists for.

The obvious fix opened a real hole

Switching to pass-on-empty made the guard blindable: repoint its glob at a filename matching nothing and it reports "not applicable" in the real repo, with twelve uncapped packs sitting right there. That mutation came back green:

M4  blind the config glob  ->  PASS   (silently vacuous)

That is the sixth instance in evals/cheap/run.sh of a guard that proves nothing, and the first where I introduced one while fixing another. I only found it because I re-ran the mutations after changing the guard instead of assuming they still held.

Closed by coupling to a second source of truth

§17c's glob must now agree with evals/paid/discover-paid-packs.sh promptfoo — the same script CI uses to build the behavioral matrix. Empty on both sides is the synthetic root and is not applicable; a disagreement means the guard has lost sight of packs that exist, and it reports that as a failure of the guard. Blinding it now means editing discovery too, and discovery has its own self-test (14-paid-discovery-broken).

mutation	before	after
delete a pack's cap (prose still names the number)	🔴	🔴
raise a cap to 8000 (answer = 192)	🔴	🔴
set the cap equal to the ceiling	🔴	🔴
blind the config glob	🟢 false pass	🔴
blind the pack-directory glob as well	🟢 false pass	🔴

All five red on both the PyYAML path and the comment-stripping fallback.

Verified in both roots this time

Cheap tier 1312 passed / 0 failed; counterfeit tier 25 passed / 0 failed. My first draft was only ever checked in the root I happened to be thinking about, which is the whole reason the counterfeit tier exists. The mutation suite runs against the real working tree, so this was committed only after confirming it restored every file it touched and that voice is back at 6144.

One thing noted and deliberately not fixed here

discover-paid-packs.sh prints a Python traceback to stderr and still exits 0 when .claude-plugin/marketplace.json is absent. The new cross-check is unaffected — a root with packs and broken discovery disagrees and so fails closed — but that script's exit code does not reflect its own failure. Out of scope for this PR and it only surfaces in a malformed root, so I am flagging it rather than widening the diff.

No paid run dispatched, and nothing was in flight when this was pushed.

---

Generated by Claude Code

[JRichlen]

f013cd8: my regression is fixed; the behavioral tier is now blocked on the grader slug, which is not this PR's

The part that was mine is green

check	c167175	f013cd8
cheap tier	🟢	🟢
counterfeit tier — detect	🟢	🟢
counterfeit tier — run (corpus)	🔴 mine	🟢
counterfeit tier (aggregate)	🔴 mine	🟢

36 of 41 jobs green on f013cd8.

The new blocker is the Anthropic grader, and it is main's configuration

promptfoo plugins: ["graveyard","tailscale-wif","fleet-playbook-curator","voice","semver-gate",
                    "verify-before-claim","wayfinder","scope-fence","find-before-build",
                    "stop-rule","redgate","agent-compiler"]
##[error]grader model 'claude-sonnet-5' did not resolve (HTTP 400).

All twelve packs share that one grader, which is why five behavioral failures on c167175 (semver-gate, redgate, agent-compiler, the aggregate) all have a single cause and none of them is the caps.

Why this is not this PR's failure:

• Neither c167175 nor f013cd8 touches a grader line — git diff c167175~1..f013cd8 has no anthropic:/claude-sonnet hunk at all. My caps go under providers[0].config; the grader lives under defaultTest.options.provider.
• The slug is byte-identical to main's. git log -S 'anthropic:messages:claude-sonnet-5' dates it to Add tracer-bullets skill plugin #40, long merged.
• It resolved fine 37 minutes earlier, on run 35797793874 at 23:31, on the same request the preflight still sends. Nothing in the repo changed between.

So an unchanged slug on an unchanged request went from 200 to 400 — the cause is outside this branch.

The preflight did exactly what it exists for

The behavioral matrix was skipped, not failed, so this run spent zero paid budget on twelve legs that would each have graded against a broken grader. That is the preflight earning its slot, and it is worth recording given how much of this PR has been about tiers reporting green over nothing.

Why I have not spent the one re-run

There is no way to re-run only the free preflight. The available mechanism is rerun_failed_jobs, which would re-run the failed aggregate and can re-evaluate the skipped matrix into running — i.e. it risks spending paid API budget, which I don't have standing authorization for. So I'm declining it rather than quietly burning budget to test a hypothesis.

I also checked whether anything already in flight could distinguish transient from permanent for free, and it cannot: the routing pack has no llm-rubric and no Anthropic grader — it grades deterministically through route-contract.js, which is why it runs independently of this preflight. Its verdict says nothing about the grader either way.

What I am not doing

Changing the grader slug. The preflight's message says "Fix the slug in that plugin's promptfooconfig.yaml", and I'm deliberately not following it: claude-sonnet-5 is the correct current identifier as far as I know, so a 400 more likely means this key's access changed than that the slug is wrong. Guessing a replacement would change the model that grades every rubric in all twelve packs — the same class of decision as changing the subject model, which this PR has treated as yours throughout. Recalibrating twelve packs' rubrics against a different judge is not a CI fix.

Your call, whichever you prefer

1. Wait — if it was transient, the next push re-runs the preflight for free.
2. Authorize one re-run of the failed jobs on f013cd8 and I'll report what the preflight says, accepting the paid-leg risk.
3. Tell me the grader slug to use if the key's Sonnet access has genuinely moved, and I'll update all twelve and say plainly that the rubrics have not been recalibrated against a new judge.

Nothing further from me until then, and I won't push while the routing leg is still in flight.

---

Generated by Claude Code

[JRichlen]

Retracting the routing S1 finding — 25 samples cannot call it

Run 35800675314 came back 5/5 on S1 and broke the claim I made two commits ago. Routing is green on f013cd8. Pushed a227c41.

The routing pack is byte-identical across all five runs — nothing in evals/routing/ has been touched on this branch — so this is sampling, not a fix.

run	S1	rows
35779397133	3/5	clean, capped
35787505902	4/5	clean, capped
35797062312	3/5	clean, capped
35797793874	3/5	clean, capped
35800675314	5/5	clean, capped
pooled	18/25 = 0.72	vs the 0.80 floor

0.72 still reads low. It is not a defect call:

statistic	value
Wilson 95% CI	[0.52, 0.86] — contains 0.80
P(observing ≤ 18/25 if true p = 0.80)	0.22

"S1 sits below its floor" and "S1 sits at its floor and five runs of five sampled unluckily" are not distinguishable from this data. At repeat: 5, more runs of the same size won't distinguish them either.

What I got wrong

At 13/20 = 0.65 I wrote — in a commit message, in docs/testing.md, in the routing config above the test, and in a comment on this PR — that S1 was "a measured sub-floor finding, not noise" and "neither n=3 noise nor a budget artifact."

My reasoning was that capping had removed truncation, so what remained must be real. Removing one confound does not make a point estimate significant, and I never computed an interval before calling it. Four runs of five landing 3, 4, 3, 3 look like a trend and are not one.

This is the third time on this PR that a pooled estimate looked like a finding and dissolved on another sample — after redgate's blanket-approval case (0/3, then 3/3) and routing's own S2/S3. Three times in the same direction, each time after I had already written down in this repo's own docs that repeat: 3 / repeat: 5 against an adjacent floor cannot support per-scenario verdicts. The statistics were in my own documentation before I ignored them.

What survives

The failure mode, which is about how S1 fails rather than how often, and is untouched by the rate question. Every failing row across all five runs gets three of four slots right and misses only guards:

ROUTE: specialist=diagnosing-bugs | envelope=redgate | guards=none | interaction_owner=redgate
                                                       ^^^^^^^^^^^^  expected verify-before-claim

scope-fence ×3, none ×4. When it fails, it is the guard slot alone — never the composition.

Nothing weakened, in either direction

The floor, the guards regex and the scenario are untouched. The note now also says explicitly not to declare this fixed off a green run — the symmetric version of the mistake I made. Routing being green today is not evidence the question is settled.

This run's routing measurement was itself clean — 70 rows, all stop, zero truncation, peak reasoning 2040 against the 7680 cap — so the 5/5 is a real sample, like the four before it.

Current state of the PR

The only red is confirm grader model resolves (HTTP 400 on claude-sonnet-5), which is not this PR's and already covered — main's slug, untouched by my commits, and it gates the behavioral matrix into skipped so nothing is being spent. That one still needs your call; I'm not guessing a replacement judge for twelve packs' rubrics.

Cheap tier 1312 / 0, counterfeit 25 / 0. Scored offline from the run's own artifact; pushed with nothing in flight.

---

Generated by Claude Code

[JRichlen]

The re-run cleared it: the tier is measured end to end, and the caps are validated

Owner authorized one re-run of 35804425107's failed jobs. confirm grader model resolves passed on the retry with no config change, so the full behavioral matrix ran for the first time since the caps landed. 51 of 52 jobs green — the only non-success is the path-filtered deep-tier matrix placeholder.

First, a correction

After the grader failed on four consecutive heads over roughly an hour, I told you "transient, will clear on the next push" was looking wrong. It wasn't. The retry cleared it with nothing changed. Four failures in a row is not persistence, and I read a run of them as a trend — the same over-read as the S1 one, pointed the other way. That's the fourth on this PR: redgate's blanket-approval case, routing S2/S3, routing S1, and now this.

What the episode does confirm: the slug was never wrong, declining to guess a replacement judge for twelve packs' rubrics was right, and the preflight held the matrix at skipped so nothing burned against a broken grader.

The tier, measured whole

13 packs, 223 rows, zero truncated and zero counterfeit rows anywhere, every row finishing stop, all twelve behavioral legs plus routing passing under honest scoring. That is the first time this tier has been measured end to end with nothing starved and no green resting on a row that never answered.

pack	before (run 35797793874)	after (run 35804425107)
voice	2 zero-answer rows at 8192, both grader-PASSED; leg RED	30/30 answered, green, peak reasoning 2237
wayfinder	1 zero-answer row grader-PASSED inside a green leg	0 counterfeit rows, peak 8192 → 1154

The clip I flagged cost nothing — and that's now measured, not asserted

When I applied the caps I said verify-before-claim's 4608 was the one real trade-off, expected to clip ~1990 tokens off its worst row, and that I couldn't know the cost without a run. Now I can:

pack	cap	rows that hit the cap	outcome
verify-before-claim	4608	1	stopped at 4608, emitted an 834-token answer, PASSED
voice	6144	0	never binding (peak 2237)
stop-rule	4096	0	never binding (peak 1111)
find-before-build	6144	0	never binding (peak 4756)

The clip bit exactly once, the row answered inside its allowance, and the grader passed it. So the reservation isn't buying green by starving thought — which is the failure mode that would have made the whole change worse than the disease.

voice's other failure is pooled, not fixed

authored prose ships without the tells was 1/3, with real stop answers that substituted SIGINT for the stimulus's SIGTERM. It read 3/3 here, pooling to 4/6. Nothing about that scenario changed, so this is a second sample and not a repair — recorded, left open.

Four scenarios across the tier sit at 0.67 (above the 0.6 floor) and are likewise left alone rather than tuned: fleet-playbook-curator's glob deference, scope-fence's clean-it-up-while-you're-in-there, semver-gate's post-sign-off-denial and non-transfer cases, and stop-rule's tempting-fourth-idea floor.

routing S1 — sixth sample folded in

3/5, 4/5, 3/5, 3/5, 5/5, 5/5 = 23/30 = 0.767 against the 0.80 floor. Wilson 95% CI [0.59, 0.88] still contains the floor; P(≤23/30 | p=0.80) = 0.39. Still unresolved — neither a defect nor fixed.

The docs and the config now also record an ordering oddity as a limitation, not a finding: the first four runs read 13/20 = 0.65, the last two are 10/10, and P(10/10 | p=0.65) = 0.0135 — mild tension with one constant rate. OpenRouter can rotate which upstream serves the subject with no trace in the artifact (rows carry only cached/finishReason/output/tokenUsage, no provider field), so an unobservable upstream change can't be ruled out, and neither can luck.

Still open for you

• scope-fence's and semver-gate's retired floors — find-before-build's de-leak rescued an apparently identical floor, and neither of those two was ever examined for verdict-stating cues.
• scope-fence's real case at 0.33 on fully-graded rows, floor retired, no negative control.
• Whether repeat: 5 / repeat: 3 should rise, given four over-reads on this PR that more samples would have prevented.
• discover-paid-packs.sh exits 0 while printing a traceback when marketplace.json is absent.

Pushed 1fd1348. Cheap tier 1312 / 0, counterfeit 25 / 0. Every number scored offline from the run's own artifacts, pushed with nothing in flight.

---

Generated by Claude Code