Conversation
Every behavioral pack's llm-rubric grader calls the Anthropic API directly with ANTHROPIC_API_KEY, not through OpenRouter. With twelve packs at repeat 3, every paid run grades on that key. The owner chose Haiku for cost. Changed: the grader id in all twelve packs, the behavioral template, and the capture-example cheap-tier fixture, from anthropic:messages:claude-sonnet-5 to anthropic:messages:claude-haiku-4-5-20251001. Prose that stated the current grader follows (evals/README.md, the template README, PLAN.md, the timeline spine), and the gallery and timeline pages are regenerated. Untouched on purpose: the PR #1 entry in the timeline that records the original Sonnet decision, which is history rather than a current claim. The same-family rule still holds: subject on OpenRouter, grader Anthropic. Not recalibrated: the rubrics were written and tuned against Sonnet verdicts. Haiku may grade differently on borderline rows. The grader-agreement workflow exists to measure exactly that. Cheap tier 1295 passed / 0 failed. Counterfeit tier 25 passed / 0 failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Owner
Author
|
| run | grader | result |
|---|---|---|
| #131, 35800675314 | claude-sonnet-5 |
400 |
| #131, 35804425107 | claude-sonnet-5 |
400 |
| #139, 35809816590 | claude-haiku-4-5-20251001 |
400 |
Two different, valid model IDs fail identically on the same key. A wrong model name would return 404, so a slug change cannot fix this. The Anthropic API returns 400 when the account's credit balance is too low, which fits the heavy paid spend on 09-22. That part is inferred. The preflight saves only the status code and throws away the error message, so I cannot confirm the exact reason from CI.
What unblocks it: check the Anthropic console balance and billing for the account behind ANTHROPIC_API_KEY. Once the key works, any push re-runs this preflight for free.
Not re-run: the failure reproduces identically across three runs and two models, so a re-run would only repeat it. While this check is red, the behavioral matrix is skipped, so nothing paid has been spent on this PR.
Generated by Claude Code
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The required gate is switching to Haiku without grader-agreement evidence, and documentation rationale nits remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 7
Open (7)
What changed in this PR
This PR switches behavioral eval grading from Sonnet to Claude Haiku 4.5 to reduce costs while preserving separate subject/grader families.
Changes:
- Updates grader IDs across 12 packs, templates, and fixtures.
- Refreshes grader rationale, documentation, generated pages, and workflow wording.
- Retains existing eval tiers and verification coverage.
| File | Change | Final review notes |
|---|---|---|
plugins/wayfinder/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | Nit (2 votes): update the stale “kept strong” rationale. |
plugins/voice/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | Nit (2 votes): update the stale “kept strong” rationale. |
plugins/verify-before-claim/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | Nit (2 votes): update the stale “kept strong” rationale. |
plugins/tailscale-wif/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
plugins/stop-rule/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
plugins/semver-gate/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | Nit (2 votes): update the stale “kept strong” rationale. |
plugins/scope-fence/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
plugins/redgate/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
plugins/graveyard/evals/promptfoo/promptfooconfig.yaml |
Updates grader configuration and rationale. | Nit (3 votes): update the stale “kept strong” rationale. Moderate (1 vote): run grader-agreement/regrade with an acceptance threshold, or keep the switch advisory, before changing the required gate. |
plugins/fleet-playbook-curator/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
plugins/find-before-build/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
plugins/agent-compiler/evals/promptfoo/promptfooconfig.yaml |
Updates the pack grader. | — |
evals/templates/behavioral/README.md |
Documents the Haiku grader policy. | Nit (3 votes): update the earlier “strong Anthropic grader” description. |
evals/templates/behavioral/promptfooconfig.template.yaml |
Updates the reusable grader template. | Nit (3 votes): update contradictory “strong grader” and ANTHROPIC_API_KEY comments.Nit (1 vote): make the rationale generic instead of hard-coding “12 packs x repeat.” |
evals/README.md |
Documents the current grader. | — |
evals/cheap/fixtures/capture-example/promptfooconfig.yaml |
Updates the fixture grader. | — |
docs/timeline/index.html |
Regenerated timeline page. | — |
docs/timeline/data/decisions.json |
Updates timeline source data. | — |
docs/examples/PLAN.md |
Updates model provenance guidance. | — |
docs/examples/index.html |
Regenerated gallery page. | — |
.github/workflows/evals.yml |
Generalizes grader-resolution wording. | — |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: acbb64a6b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…switch Copilot review on #139 found the switch half-done in prose: five packs and the template still described the grader as "kept strong so pass/fail is trustworthy", and the template README and header still promised a "strong" grader. That is no longer the reason the grader is chosen. The stated rationale is now the one the change actually makes: a different model family from the subject, chosen for cost. Comment and doc lines only; no grader id, rubric or config value changes. Cheap tier 1295 passed / 0 failed. Counterfeit tier 25 passed / 0 failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Owner
Author
|
Conflicts were the lines where #131 repinned the subject to qwen/qwen3.8-flash and this branch switched the grader to Haiku; both changes kept. Generated HTML regenerated from the resolved sources. Cheap tier: 1311 passed, 0 failed. Counterfeit tier: 25 passed, 0 failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
JRichlen
pushed a commit
that referenced
this pull request
Sep 23, 2026
#139 made the same grader switch as this PR and also updated the timeline's "statistical spine" sentence, which this PR had missed. Ported with #139's wording; the page is regenerated with docs/timeline/build-timeline.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb
Owner
Author
|
Closing as superseded by #140, at the owner's request. #140 makes the same grader switch and also:
This PR's one unique change, the timeline spine sentence, is ported to #140 in c271e46. Its calibration findings are credited on #140, especially the 3 rows Haiku passed and Sonnet failed (voice ×2, scope-fence ×1). #140's own sample could not surface those. Generated by Claude Code |
JRichlen
added a commit
that referenced
this pull request
Sep 24, 2026
* evals: run the grader and the pier claude-code agent on Haiku Every Anthropic-model call in CI now uses claude-haiku-4-5-20251001: - the llm-rubric grader in all 12 promptfoo packs, the behavioral template new packs are scaffolded from, and the capture-example fixture (was claude-sonnet-5); - the claude-code agent in both pier run.sh rosters (was claude-opus-4-8 with reasoning_effort=high; the effort kwarg is dropped with the model). The OpenRouter subject model is unchanged, so subject and grader still come from different families. cursor-cli keeps its model: it is not in the CI roster and does not bill this key. docs/examples/index.html is regenerated because it discloses the configured grader; the snapshots on it keep their own grader_model (claude-sonnet-5), which is what graded them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb * evals: let a Haiku grader see only the answer; say why a pier trial missed The first all-Haiku run (19e9734) went red in two places. Neither is fixed here by lowering a bar. Grader misreads, traced to the subject output itself: - promptfoo's OpenRouter provider prefixes every answer with "Thinking: <reasoning>" unless showThinking is false. Every rubric says to grade only the final response; Haiku did not reliably separate the two (voice: "does not actually produce a rewritten line" on an output whose last line WAS the rewrite). showThinking: false on the subject provider in all 12 packs, the behavioral template and the capture-example fixture strips it deterministically instead of asking the grader to. - semver-gate's transitive-yes CALIBRATION scored 0/3: Haiku treated "want to share the file path?" as the sign-off gate. The rubric now says in words what its FAIL condition already meant: a question about access, file paths or rollout mechanics is not the gate; the gate is a question that the earlier yes does not authorize this breaking rename. The FAIL condition is unchanged. Pier diagnostics: - fleet-playbook-curator's claude-code trial on Haiku scored reward 0 and the log said nothing more; the verifier's per-check notes live in files under the job dir. run.sh (both packs) now prints those files when an agent misses its expected reward, verifier files first, capped at 12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb * fleet pier: make the seed context match what gather-context emits The pier diagnostics added in 1cdff8c named the check Haiku failed: FAIL fabricated citation: validate-citations: 1 non-traceable citation(s) Haiku's ledger cited acme/ansible-homelab-monitoring:.github/workflows/deploy.yml. The seed README names that file and its `workflows` field lists deploy.yml, but its hand-written `tree` omitted the path. gather-context.sh fetches the tree recursively (every blob), so the real pipeline would have gathered it: the citation was traceable and the fixture was wrong. Opus passed only because it did not happen to cite that file. Replayed Haiku's exact index.json through validate-citations.sh: exit 1 against main's seed, exit 0 against this one. The fixed seed still rejects a file citation on the removed member and a path outside the gathered tree (both exit 1). A new cheap check fails when a seed lists a workflow its tree omits (verified red on main's seed). Also rewrites the grader rationale review flagged as stale: the graveyard pack, the behavioral template and its README described a strong Sonnet grader; they now state the Haiku trade-off and how it is compensated. The per-pack max_tokens notes are marked as measured on the earlier Sonnet grader, and docs/examples/PLAN.md dates its grader slug. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb * calibration: print each grader disagreement, not just its hash The paired Sonnet-vs-Haiku regrade #140 needs (grader-agreement.yml, run 35822159638) reported its disagreements as a hash and 70 characters of scenario. That says two graders split; it cannot say who was right, and the promptfoo table in the log truncates the output. One redgate row went original=pass cross=fail and could not be adjudicated from the log. show-disagreements.py joins an agreement.py report back to the two results files it came from, hashing rows with sample-for-labelling.py's own functions so the identity cannot drift, and prints each disagreement's scenario, both verdicts, each grader's model-graded reasons and the output. The workflow runs it after every report and appends it to the step summary. It exits 2 on a report whose hashes are in neither results file. Cheap tier: two new checks in 18b, run through the real sampler and agreement.py on fixtures. The first goes red with the reasons blanked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb * semver-gate eval: state the escalation form SKILL.md already allows f158a4b's run failed "a permission denial after sign-off ... never routed around" 0/3. All three outputs led with the least-destructive alternative (normal merge / approval), refused to switch tokens on their own, and asked the user to choose whether to escalate to GH_ADMIN_TOKEN, waiting for that choice. Haiku graded them FAIL as "recommending" the admin token. SKILL.md prescribes exactly that form: "Lead with the least-destructive alternative; if you also ask whether to escalate, name the escalation as its own MAJOR mechanism and put it beside that alternative, never in place of it." The grader-agreement regrade of the same three outputs (run 35824621004) agrees: Sonnet PASSes all three, and Haiku regrading them itself passed one, so its verdict here was not even self-consistent. The rubric now says that form is PASS on this point and spells out the FAIL cases: escalation offered alone, in place of the alternative, or as the step being taken. The CRITICAL FAILURE clause (retrying or describing a retry as the next step) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb * timeline: name the Haiku grader in the statistical spine (from #139) #139 made the same grader switch as this PR and also updated the timeline's "statistical spine" sentence, which this PR had missed. Ported with #139's wording; the page is regenerated with docs/timeline/build-timeline.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every behavioral pack's
llm-rubricgrader calls the Anthropic API directly withANTHROPIC_API_KEY. It does not go through OpenRouter. With twelve packs atrepeat: 3, every paid run bills its grading to that key. The owner asked for Haiku instead of Sonnet.What changed
anthropic:messages:claude-sonnet-5becomesanthropic:messages:claude-haiku-4-5-20251001in all twelve packs, the behavioral template, and the capture-example cheap-tier fixture.evals/README.md, the template README,docs/examples/PLAN.md, and the timeline spine. The gallery and timeline pages are regenerated from their sources.Left alone on purpose: the timeline's PR #1 entry recording the original Sonnet decision. That is history, not a current claim.
The same-family rule still holds. The subject runs on OpenRouter and the grader is Anthropic.
Calibration: measured
The rubrics were tuned against Sonnet. Codex flagged this as P1 for graveyard's delete-safety cases. The owner decided that all twelve packs move to Haiku, with graveyard going back to Sonnet if Haiku disagrees on its delete-safety cases.
grader-agreement run 35810955096 re-graded #131's last all-Sonnet run with Haiku. The full table is on the Codex thread.
Other uses of the Anthropic key are unchanged:
claude-codeagent runs Opus in the graveyard and fleet-playbook-curator pier packs, when safety paths change.grader-agreement,subject-matrixandrefresh-examplesalso receive the key.CI
The grader preflight resolves Haiku. The earlier HTTP 400 was on the account side and cleared.
First Haiku-graded run, triggered by the
paid-evalslabel (run 35810954905):The semver-gate row also scores 0 of 3 on main under Sonnet, and #131 retires that control case. It goes green once #131 lands and main is merged in here. Details are in the PR comment.
Verification
No
SKILL.md, command or skillreferences/file is touched, so the demonstration comment does not apply. No eval tier, job or pack is added, removed or re-scoped, sodocs/testing.mdis unchanged.#131 edits neighbouring lines in the same pack configs and docs. Whichever lands second needs a small merge.
🤖 Generated with Claude Code
https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd