Repository navigation
feat(verifiers): every built-in verifier writes its named grade - #790
Merged
earakely-scale merged 16 commits intoNov 19, 2026
Merged
earakely-scale merged 16 commits into
earakely-scale merged 16 commits into
Conversation
… records whatever it can't read Verification gains error: the grade is failed with it while the legacy entry the verifier recorded is still written, for a test command that timed out. Anything after verify that raises (scoring, the grade, the output) is a failed grade too, settings that can't be read are recorded as settings_error rather than losing the verifier's own error, a failed grade's reason is cut at REASONING_CHARS, and the settings a subclass adds are compacted as the base's own are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…box, env_outcome and aggregate write today Each path's verifications entry, byte for byte with its key order, and what it raised, recorded on the unchanged verifiers with offline fakes, so their move onto VerifierTaskStep can be checked against them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… it moves onto the verifier base rubrics_verifier (20 paths: each judge format, batches and retries, the skips, the gate, judge failures, unjudged rows, an agent judge, a run's overrides), user_sim_judge (15) and always_on_agent_verifier (4), each run offline with the judge faked below the verifier's own code. The entries, the models record and the error each run leaves are recorded under tst/data/verifier_golden and must stay equal, number types included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gate write their named grades The four move onto VerifierTaskStep and write the legacy entry they wrote before, which the recorded tests check byte for byte, beside a grade: - run_container_unit_tests_verifier: its reward or exit-code row with the score it read, its command, logs and result files as output; a timeout is a failed grade and a non-zero exit without reward_path a graded 0, both raising as before once recorded. Its settings hold a run's step_params command and the names of its env vars, never their values. - verify_sandbox: its rows, scored over those it didn't skip. - env_outcome_verifier: the script's rows as returned; one that returns None still records a 0 and fails its step, now saying why. - aggregate_verifiers: WEIGHTED_AVERAGE by default, its pooled score with source_verifier_ids as output; its entry gains an empty results list, which agent-env task run no longer prints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sim_judge write their named grades The rubrics family moves onto VerifierTaskStep, and its legacy entries stay as main wrote them (the golden tests recorded before the move pass unchanged, now with grade checks). - rubrics_verifier: `_judged` returns the verdict and writes nothing (it was `_grade`, whose name the base's grade builder takes); `_entries` and `_skipped` return what `_write_entries` and `_write_skipped_entries` wrote. A prompt error or a run with no screenshots stays graded 0 with its one row and the int score. `verify` gates the verdict and writes the `-trajectory-mistakes` entry itself, last. fail_task_when_not_passing fails a score under the step's min_passing_score (1.0 unless set), with today's message. Its settings record what the run graded with: each override it applies in place of the stored value, the judge model the run recorded (else the one it resolves), the agent it deploys as its judge, and `overridden`; never a key, header or base URL. - always_on_agent_verifier: its refusals and its additions to the entry (judge_reads, context_loss, assistant_harness) move into its `_judged`, on the verdict's output. - user_sim_judge: `verify` replaces `execute`; its three skips are real skips (they already recorded no score), its score stays its own 0/1 rule, and its gate fails the step once both are written. It takes score_aggregator and min_passing_score from the base; PINNED refuses a min_passing_score. Its settings leave out the overrides it ignores. The branch for a plan with no criteria, which plan_run never makes, is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g2-built-in-verifiers-move-onto-the
…TS.md says every built-in is on the base Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2-g2-built-in-verifiers-move-onto-the
…ailed grade and its env var names, end to end run_container_unit_tests_verifier's settings always carry overridden, as the rubrics family's do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e's base URL stays out of them, and the rubrics golden check says it leaves key order aside Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p made it before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…values from one resolver _run_values resolves every setting a run's step_params or user_overrides can change; the judge reads it and verifier_settings records it, so the two can't drift. The settings now record a run's effort and thinking budget only for an agent judge, and the agent sandbox only for a judge the step deploys, as the judge uses them, and the user sim judge records the batch concurrency its judge runs with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rds its files on AGENTENV_TEST_RECORD_GOLDEN=1 test_legacy_entries.py checks through tst/util/verifier_golden.py as the rubrics tests do, and every golden check compares key order too. Re-recorded once: the four simpler verifiers' file takes the base's two deliberate changes (an aggregate's empty results, the message for a check script that returns None), and the rubrics family's two files the order the base writes, every value the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2-g2-built-in-verifiers-move-onto-the
earakely-scale
deleted the
edgararakelyan/fd-5922-g2-built-in-verifiers-move-onto-the
branch
November 19, 2026 00:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The seven built-in verifiers still writing their own
verificationsentries move onto theVerifierTaskStepbase from #761:run_container_unit_tests_verifierverify_sandboxenv_outcome_verifieraggregate_verifiersrubrics_verifieralways_on_agent_verifieruser_sim_judgeEach one now implements
verifyand returns aVerification. The base writes the step's named grade tocontext.metadata["grades"][<step id>], beside the legacy entry every reader still takes. The legacy entries keep every key and value: golden tests recorded on main before the move check each one (number types included), along with the models record and the exception raised. The one change is key order in the rubrics family's entries: main wroteformatbeforeresultsandscore, and the base writes those two first. No reader depends on key order; the four simpler verifiers' entries keep theirs exactly.Base changes
Verification.error. A verifier sets it when its verdict can't be trusted. The grade isfailedwith that reason, cut at 2000 characters. The legacy entry is still written with the verifier's score, thenraise_afterraises.verify(). Scoring, building the grade and the output are inside thetry, so an exception there is a failed grade, raised again.verifier_settingsthat raises is recorded assettings_errorand never fails the step. The base now compacts every settings value over 1 KB to{sha256, bytes}, including a subclass's additions, and the defaultverifier_settingsreturns raw fields.criteria=None. Accepted: the legacy entry keepsresults: None, with score 0.0.Decisions
The trajectory-mistakes entry.
rubric_evidence_with_mistakesstill writes its<verifier id>-trajectory-mistakesentry.rubrics_verifier.verify()writes it itself, on the skip paths too. There is no base hook and no second grade.A timed-out
run_container_unit_tests_verifieris a failed grade that names the timeout. It raises the sameRuntimeErroras before. A non-zero exit with noreward_pathstays a graded 0, since that is a real verdict, and still raises.aggregate_verifiers' legacy entry gains"results": [].agent-env task runprints the results block only when the list isn't empty.rubrics_verifier'sfail_task_when_not_passinggate fails a score below the step'smin_passing_score. The default is 1.0, so nothing changes today.user_sim_judge's gate (user_sim_error.fails_task) is unchanged.verifier_settingsrecords the values the run graded with. A run's override replaces the stored value under the same key, andoverriddennames the keys it changed:step_params:criteria,grade_up_to_turn,command,env_vars;user_overrides:use_agent_judge,apply_trajectory_filter,judge_effort,judge_max_thinking_tokens, the judge batch options and the agent sandbox.judge_modelis the model the run recorded, else the one it resolves. Settings never hold an API key,[model.headers], a base URL or an env var's value:run_container_unit_tests_verifierrecords env var names only.In the rubrics family, the judge and its settings read a run's values from one resolver,
_run_values, so what's recorded can't drift from what the judge used. A run's effort and thinking budget are recorded only for an agent judge, and its agent sandbox only for a judge the step deploys, since nothing else uses them.user_sim_judgeleaves out only what its judge never reads: the criteria, which its plan replaces, and the trajectory filter.Grade rows stay full copies of the verifier's rows.
Defaults I chose
_gradebecomes_judged, which returns aRubricsVerdictand writes nothing._write_entriesand_write_skipped_entriesbecome_entriesand_skipped, which return. The subclasses'executeoverrides becomeverify/_judgedoverrides. Every method a test patches keeps its name.run_container_unit_tests_verifier;verify_sandbox(non-skipped rows only);aggregate_verifiers(its default aggregator is nowweighted_average);user_sim_judge(0 when a row fails, else 1);rubrics_verifier.user_sim_judge's three skips become real skips with a null score, as before.user_sim_judge's settings. It acceptsscore_aggregatorandmin_passing_score, but its pinned settings refuse amin_passing_score.user_sim_judge's no-criteria branch is gone:plan_runalways asks all seven checks, so it was unreachable.always_on_agent_verifierrefuses agrade_up_to_turnoverride oruse_agent_judge=Falsewithuse_trajectory. That refusal now leaves a failed grade before it raises, as any exception fromverifydoes.Tests
Golden tests, recorded on main before the move:
tst/unit/task_step/verifiers/test_legacy_entries.py: 48 scenarios for the four simpler verifiers.test_rubrics_verifier_golden.pyandtest_rubrics_subclasses_golden.py: 39 scenarios for the rubrics family.Each also checks that the grade agrees with the entry. The data is in
tst/data/verifier_golden/. All three files go through one harness,tst/util/verifier_golden.py, which compares values, number types and key order.After a deliberate change,
AGENTENV_TEST_RECORD_GOLDEN=1re-records the files; review their diff. That mode has been used once, on purpose, and changed:results, and the message for a check script that returns None);New unit tests: the base (the
errorfield, failures afterverify, unreadable settings, long errors, compacted additions) and the rubrics family's settings and gate.Integration, in
tst/integration/task_step/test_verify_sandbox_local.py:verify_sandbox, oneaggregate_verifiers);run_container_unit_tests_verifierin a local container, which leaves a failed grade beside its entry;task_steps_test.py, which needs a model endpoint, now also checks theenv_outcome_verifierandrubrics_verifiergrades.vm (agent-environments-admin):
agent-env rune2e and chaos (below).e2e and chaos
All on an agent-environments-admin vm, at this PR's head.
agent-env run, then each stored run read back withagent-env task get-instance:hellowith a control and an aggregate added. The twoverify_sandboxsteps are graded 1.0 and 0.0, andaggregate_verifiersis graded 0.667 (2 of 3 rows). Each legacy entry has the same score as its grade.echo-mcp.env_outcome_verifieris graded 1.0.desktop --task files.run_container_unit_tests_verifierruns inside the computer and is graded 1.0. Its settings name its env vars, never their values.rubrics_verifierjudges. The solver is Claude Sonnet 4.6 and the judge Claude Haiku 4.5. The grade rows match the legacy rows, andjudge_modelis the model the run recorded.fail_task_when_not_passingwithmin_passing_score0.5 andweighted_averagescores 0.5 and passes.all_passit scores 0.0 and the step fails withVerifierNotPassingError, after both the grade and the entry are written.--judge-modelrecordsoverridden: ["judge_model"]and the new model.Model spend was under $2.
Chaos, timeboxed under 30 minutes, with real local containers and stores; all 7 cases pass:
verifier_output_refwhose object's sha256 and size match; the legacy entry keeps the output inline.verify: a failed grade, the step raises, no legacy entry.docker kill)timed_out: true.step_paramscommandoverridden: ["command"]and its own score.{sha256, bytes}; the grade's rows are complete.One existing behavior surfaced:
run_container_unit_tests_verifiertreats exit 137 as a timeout, as main does. A killed or OOM-killed command therefore reads "timed out after Ns" in its failed grade. That's unchanged here, and could be a follow-up.Not run with a real judge:
user_sim_judgeandalways_on_agent_verifier. Each needs a user-sim task built for the purpose. Both makerubrics_verifier's judge call, which the run above exercised, and their 19 golden scenarios recorded on main cover the move.task_steps_test.py's judge tests skip without a default agent registered in the store, which the vm doesn't have.CI time: the integration jobs are no slower than main's latest green run. The fast job took 188 s against 192 s, and the slow and gateway shards were within run-to-run variance. One unit shard grew about 50 s; most of that is noise, and the golden tests take about 10 s.
Also noticed: the CLI's summary says "scored below 1" for a run whose grades all passed at their
min_passing_score. That summary is the CLI's own rule, untouched here; G7's result rules replace it.Docs to cover
errorfield onVerification.verifier_settingsholds: the run's values,overridden, and never a key or an env var's value.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new blocking issues were found.Summary
Moves the seven remaining built-in verifiers onto the shared grade writer while keeping their legacy results.
No new actionable issues found. The earlier endpoint and settings-error findings are addressed. earakely-scale accepted the rubrics key-order change because no reader depends on it; the current golden comparisons also check key order.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Built-in verifier] --> B[Return Verification] B --> C[Build named grade] C --> D[Save grade by step ID] D --> E[Save legacy verification entry] E --> F{Deferred exception?} F -->|Yes| G[Raise after saving] F -->|No| H[Return context] A -->|Exception| I[Save failed grade and re-raise]Reviews (6) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile