fix(regression): judge integrity — close false-pass vectors (#41/#44) - #82
Merged
Conversation
…grading Addresses the release-readiness blockers on the auto-judge (#41) and model-churn routine (#44) — the judge could pass a scenario it never graded, undermining every judged result. - parseVerdict rejects an empty criteria array (B5: [].every() === true would pass) and, given an expected count, rejects a short array (H1: a judge that drops the criteria it would fail). A missing/malformed per-criterion verdict is no longer a pass. - judge.sh threads the scenario's criteria count (new `judge.mjs --count`) into `--parse`, adds `set -o pipefail`, and the CLI now exits 2 on a judge/setup fault, distinct from 1 (regression) — so run-all.sh stops mislabeling a broken judge as a skill regression (M4). - run-all.sh captures the run's OWN transcript path (run-test.sh now emits TRANSCRIPT_PATH/TRANSCRIPT_STATUS) and marks ERROR when no fresh transcript exists, instead of judge.sh auto-selecting a prior model's stale file (H2). - Split the vague-groupbuy criteria cell so its six gaps grade independently (M5, now 7 criteria); fixed extract-result.mjs's CLI guard (L1). - 5 new unit tests (empty, subset, count, malformed verdict, criteriaCount). The '9/9 Fable 5 PASS' should be re-run under this hardened judge before it is trusted. Relates to #41 #44. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwyFyRC6pepaK6DuMK7YeT
huiyu
added a commit
that referenced
this pull request
Aug 24, 2026
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwyFyRC6pepaK6DuMK7YeT
5 tasks
huiyu
added a commit
that referenced
this pull request
Aug 24, 2026
PRs #82–#89 merged to 3.0; blockers B1–B5 cleared and per-issue H/S/M findings closed or reclassified as tracked residual risks R1–R4 (post-3.0 live-integration work, not release blockers). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwyFyRC6pepaK6DuMK7YeT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First fix off the 3.0 release-readiness audit. Closes the judge false-pass vectors that undermined every judged result (blockers B5, H1; highs H2, M4; M5, L1).
parseVerdictrejects an empty criteria array ([].every()===truewould pass).judge.mjs --count), rejects a short array (a judge dropping the criteria it would fail). Malformed per-criterion verdicts no longer pass.judge.shgainsset -o pipefail.run-all.shjudges the run's own transcript (run-test.shemitsTRANSCRIPT_PATH/TRANSCRIPT_STATUS) and marks ERROR when none is fresh — no more grading a prior model's stale file.vague-groupbuysplit into 7 independently-graded criteria; L1 —extract-result.mjsCLI guard fixed.Verified: 14 unit tests pass; CLI exit codes confirmed (empty→2, subset→2, pass→0);
--count=7. Base is3.0(not main). Resolves PR #81 threads B5/H1/H2 on merge.The prior "9/9 Fable 5 PASS" should be re-run under this hardened judge before it's trusted.
Relates to #41 #44