Skip to content

feat(verifiers): every built-in verifier writes its named grade - #790

Merged
earakely-scale merged 16 commits into
mainfrom
edgararakelyan/fd-5922-g2-built-in-verifiers-move-onto-the
Nov 19, 2026
Merged

earakely-scale merged 16 commits into
mainfrom
edgararakelyan/fd-5922-g2-built-in-verifiers-move-onto-the

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

The seven built-in verifiers still writing their own verifications entries move onto the VerifierTaskStep base from #761:

  • run_container_unit_tests_verifier
  • verify_sandbox
  • env_outcome_verifier
  • aggregate_verifiers
  • rubrics_verifier
  • always_on_agent_verifier
  • user_sim_judge

Each one now implements verify and returns a Verification. The base writes the step's named grade to context.metadata["grades"][<step id>], beside the legacy entry every reader still takes. The legacy entries keep every key and value: golden tests recorded on main before the move check each one (number types included), along with the models record and the exception raised. The one change is key order in the rubrics family's entries: main wrote format before results and score, and the base writes those two first. No reader depends on key order; the four simpler verifiers' entries keep theirs exactly.

Base changes

  • Verification.error. A verifier sets it when its verdict can't be trusted. The grade is failed with that reason, cut at 2000 characters. The legacy entry is still written with the verifier's score, then raise_after raises.
  • Failures after verify(). Scoring, building the grade and the output are inside the try, so an exception there is a failed grade, raised again.
  • Unreadable settings. A verifier_settings that raises is recorded as settings_error and never fails the step. The base now compacts every settings value over 1 KB to {sha256, bytes}, including a subclass's additions, and the default verifier_settings returns raw fields.
  • criteria=None. Accepted: the legacy entry keeps results: None, with score 0.0.

Decisions

  1. The trajectory-mistakes entry. rubric_evidence_with_mistakes still writes its <verifier id>-trajectory-mistakes entry. rubrics_verifier.verify() writes it itself, on the skip paths too. There is no base hook and no second grade.

  2. A timed-out run_container_unit_tests_verifier is a failed grade that names the timeout. It raises the same RuntimeError as before. A non-zero exit with no reward_path stays a graded 0, since that is a real verdict, and still raises.

  3. aggregate_verifiers' legacy entry gains "results": []. agent-env task run prints the results block only when the list isn't empty.

  4. rubrics_verifier's fail_task_when_not_passing gate fails a score below the step's min_passing_score. The default is 1.0, so nothing changes today. user_sim_judge's gate (user_sim_error.fails_task) is unchanged.

  5. verifier_settings records the values the run graded with. A run's override replaces the stored value under the same key, and overridden names the keys it changed:

    • from step_params: criteria, grade_up_to_turn, command, env_vars;
    • from user_overrides: use_agent_judge, apply_trajectory_filter, judge_effort, judge_max_thinking_tokens, the judge batch options and the agent sandbox.

    judge_model is the model the run recorded, else the one it resolves. Settings never hold an API key, [model.headers], a base URL or an env var's value: run_container_unit_tests_verifier records env var names only.

    In the rubrics family, the judge and its settings read a run's values from one resolver, _run_values, so what's recorded can't drift from what the judge used. A run's effort and thinking budget are recorded only for an agent judge, and its agent sandbox only for a judge the step deploys, since nothing else uses them. user_sim_judge leaves out only what its judge never reads: the criteria, which its plan replaces, and the trajectory filter.

  6. Grade rows stay full copies of the verifier's rows.

Defaults I chose

  • Rubrics restructure. _grade becomes _judged, which returns a RubricsVerdict and writes nothing. _write_entries and _write_skipped_entries become _entries and _skipped, which return. The subclasses' execute overrides become verify / _judged overrides. Every method a test patches keeps its name.
  • Direct scores wherever aggregating the rows would differ from what the verifier wrote:
    • run_container_unit_tests_verifier;
    • verify_sandbox (non-skipped rows only);
    • aggregate_verifiers (its default aggregator is now weighted_average);
    • user_sim_judge (0 when a row fails, else 1);
    • rubrics_verifier.
  • Rubrics' own skips stay graded 0. The prompt error and no-screenshots cases keep a score of 0. user_sim_judge's three skips become real skips with a null score, as before.
  • user_sim_judge's settings. It accepts score_aggregator and min_passing_score, but its pinned settings refuse a min_passing_score.
  • Dead code dropped. user_sim_judge's no-criteria branch is gone: plan_run always asks all seven checks, so it was unreachable.
  • Earlier refusals now leave a failed grade. always_on_agent_verifier refuses a grade_up_to_turn override or use_agent_judge=False with use_trajectory. That refusal now leaves a failed grade before it raises, as any exception from verify does.
  • No defensive code was added for cases no caller produces.

Tests

  • Golden tests, recorded on main before the move:

    • tst/unit/task_step/verifiers/test_legacy_entries.py: 48 scenarios for the four simpler verifiers.
    • test_rubrics_verifier_golden.py and test_rubrics_subclasses_golden.py: 39 scenarios for the rubrics family.

    Each also checks that the grade agrees with the entry. The data is in tst/data/verifier_golden/. All three files go through one harness, tst/util/verifier_golden.py, which compares values, number types and key order.

    After a deliberate change, AGENTENV_TEST_RECORD_GOLDEN=1 re-records the files; review their diff. That mode has been used once, on purpose, and changed:

    • in the four simpler verifiers' file, only the base's two deliberate changes (an aggregate's empty results, and the message for a check script that returns None);
    • in the rubrics family's two files, only key order.
  • New unit tests: the base (the error field, failures after verify, unreadable settings, long errors, compacted additions) and the rubrics family's settings and gate.

  • Integration, in tst/integration/task_step/test_verify_sandbox_local.py:

    • a task run whose stored instance carries each verifier's grade (two verify_sandbox, one aggregate_verifiers);
    • a timed-out run_container_unit_tests_verifier in a local container, which leaves a failed grade beside its entry;
    • a reward run whose grade names its env vars without their values.

    task_steps_test.py, which needs a model endpoint, now also checks the env_outcome_verifier and rubrics_verifier grades.

  • vm (agent-environments-admin):

    • the full unit tier: only the three tests that always fail on that host failed;
    • the fast and slow integration files that run these verifiers;
    • agent-env run e2e and chaos (below).

e2e and chaos

All on an agent-environments-admin vm, at this PR's head.

agent-env run, then each stored run read back with agent-env task get-instance:

  • hello with a control and an aggregate added. The two verify_sandbox steps are graded 1.0 and 0.0, and aggregate_verifiers is graded 0.667 (2 of 3 rows). Each legacy entry has the same score as its grade.
  • echo-mcp. env_outcome_verifier is graded 1.0.
  • desktop --task files. run_container_unit_tests_verifier runs inside the computer and is graded 1.0. Its settings name its env vars, never their values.
  • A Claude Code run on the echo env with two real rubrics_verifier judges. The solver is Claude Sonnet 4.6 and the judge Claude Haiku 4.5. The grade rows match the legacy rows, and judge_model is the model the run recorded.
    • The gate passes on a partial score. fail_task_when_not_passing with min_passing_score 0.5 and weighted_average scores 0.5 and passes.
    • The gate fails after recording. With all_pass it scores 0.0 and the step fails with VerifierNotPassingError, after both the grade and the entry are written.
    • A run override is recorded. The same run with --judge-model records overridden: ["judge_model"] and the new model.

Model spend was under $2.

Chaos, timeboxed under 30 minutes, with real local containers and stores; all 7 cases pass:

Fault What the grade does
A 42 KB result file The grade holds a verifier_output_ref whose object's sha256 and size match; the legacy entry keeps the output inline.
The object store's folder made read-only The stdout upload fails inside verify: a failed grade, the step raises, no legacy entry.
The container killed mid-command (docker kill) A failed grade, raised; the legacy entry has timed_out: true.
One step instance graded by 16 runs at once, each with its own step_params command Each grade has its own command, overridden: ["command"] and its own score.
An aggregate over 400 long missing ids The failed grade's reason is cut to about 2,000 characters; the exception keeps all 30,000.
300 criteria (44 KB of settings) The setting is kept as {sha256, bytes}; the grade's rows are complete.
A run killed (SIGKILL) while its verifier works, then resumed at that step The killed run's record has no grade for it; the resumed run's is graded.

One existing behavior surfaced: run_container_unit_tests_verifier treats exit 137 as a timeout, as main does. A killed or OOM-killed command therefore reads "timed out after Ns" in its failed grade. That's unchanged here, and could be a follow-up.

Not run with a real judge: user_sim_judge and always_on_agent_verifier. Each needs a user-sim task built for the purpose. Both make rubrics_verifier's judge call, which the run above exercised, and their 19 golden scenarios recorded on main cover the move. task_steps_test.py's judge tests skip without a default agent registered in the store, which the vm doesn't have.

CI time: the integration jobs are no slower than main's latest green run. The fast job took 188 s against 192 s, and the slow and gateway shards were within run-to-run variance. One unit shard grew about 50 s; most of that is noise, and the golden tests take about 10 s.

Also noticed: the CLI's summary says "scored below 1" for a run whose grades all passed at their min_passing_score. That summary is the CLI's own rule, untouched here; G7's result rules replace it.

Docs to cover

  • Every built-in verifier writes a named grade.
  • The error field on Verification.
  • What verifier_settings holds: the run's values, overridden, and never a key or an env var's value.
  • The trajectory-mistakes entry stays a legacy entry only.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

Moves the seven remaining built-in verifiers onto the shared grade writer while keeping their legacy results.

  • Built-in checks now leave a named grade with each legacy result.
  • Rubric judges save their verdicts and run settings as named grades.
  • Always-on and user-sim judges now leave named grades for their results.

No new actionable issues found. The earlier endpoint and settings-error findings are addressed. earakely-scale accepted the rubrics key-order change because no reader depends on it; the current golden comparisons also check key order.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Built-in verifier] --> B[Return Verification]
    B --> C[Build named grade]
    C --> D[Save grade by step ID]
    D --> E[Save legacy verification entry]
    E --> F{Deferred exception?}
    F -->|Yes| G[Raise after saving]
    F -->|No| H[Return context]
    A -->|Exception| I[Save failed grade and re-raise]
Loading

Reviews (6) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

earakely-scale and others added 11 commits November 18, 2026 14:14
… records whatever it can't read

Verification gains error: the grade is failed with it while the legacy entry the verifier recorded is still written,
for a test command that timed out. Anything after verify that raises (scoring, the grade, the output) is a failed grade
too, settings that can't be read are recorded as settings_error rather than losing the verifier's own error, a failed
grade's reason is cut at REASONING_CHARS, and the settings a subclass adds are compacted as the base's own are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…box, env_outcome and aggregate write today

Each path's verifications entry, byte for byte with its key order, and what it raised, recorded on the unchanged
verifiers with offline fakes, so their move onto VerifierTaskStep can be checked against them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… it moves onto the verifier base

rubrics_verifier (20 paths: each judge format, batches and retries, the skips, the gate, judge failures, unjudged
rows, an agent judge, a run's overrides), user_sim_judge (15) and always_on_agent_verifier (4), each run offline with
the judge faked below the verifier's own code. The entries, the models record and the error each run leaves are
recorded under tst/data/verifier_golden and must stay equal, number types included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gate write their named grades

The four move onto VerifierTaskStep and write the legacy entry they wrote before, which the recorded tests check byte
for byte, beside a grade:

- run_container_unit_tests_verifier: its reward or exit-code row with the score it read, its command, logs and result
  files as output; a timeout is a failed grade and a non-zero exit without reward_path a graded 0, both raising as
  before once recorded. Its settings hold a run's step_params command and the names of its env vars, never their
  values.
- verify_sandbox: its rows, scored over those it didn't skip.
- env_outcome_verifier: the script's rows as returned; one that returns None still records a 0 and fails its step,
  now saying why.
- aggregate_verifiers: WEIGHTED_AVERAGE by default, its pooled score with source_verifier_ids as output; its entry
  gains an empty results list, which agent-env task run no longer prints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sim_judge write their named grades

The rubrics family moves onto VerifierTaskStep, and its legacy entries stay as main wrote them (the golden tests
recorded before the move pass unchanged, now with grade checks).

- rubrics_verifier: `_judged` returns the verdict and writes nothing (it was `_grade`, whose name the base's grade
  builder takes); `_entries` and `_skipped` return what `_write_entries` and `_write_skipped_entries` wrote. A
  prompt error or a run with no screenshots stays graded 0 with its one row and the int score. `verify` gates the
  verdict and writes the `-trajectory-mistakes` entry itself, last. fail_task_when_not_passing fails a score under
  the step's min_passing_score (1.0 unless set), with today's message. Its settings record what the run graded with:
  each override it applies in place of the stored value, the judge model the run recorded (else the one it
  resolves), the agent it deploys as its judge, and `overridden`; never a key, header or base URL.
- always_on_agent_verifier: its refusals and its additions to the entry (judge_reads, context_loss,
  assistant_harness) move into its `_judged`, on the verdict's output.
- user_sim_judge: `verify` replaces `execute`; its three skips are real skips (they already recorded no score), its
  score stays its own 0/1 rule, and its gate fails the step once both are written. It takes score_aggregator and
  min_passing_score from the base; PINNED refuses a min_passing_score. Its settings leave out the overrides it
  ignores. The branch for a plan with no criteria, which plan_run never makes, is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TS.md says every built-in is on the base

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailed grade and its env var names, end to end

run_container_unit_tests_verifier's settings always carry overridden, as the rubrics family's do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 22:59
Comment thread tst/util/verifier_golden.py Resolved
earakely-scale and others added 5 commits November 18, 2026 15:06
…e's base URL stays out of them, and the rubrics golden check says it leaves key order aside

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p made it before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…values from one resolver

_run_values resolves every setting a run's step_params or user_overrides can change; the judge reads it and
verifier_settings records it, so the two can't drift. The settings now record a run's effort and thinking budget only
for an agent judge, and the agent sandbox only for a judge the step deploys, as the judge uses them, and the user sim
judge records the batch concurrency its judge runs with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rds its files on AGENTENV_TEST_RECORD_GOLDEN=1

test_legacy_entries.py checks through tst/util/verifier_golden.py as the rubrics tests do, and every golden check
compares key order too. Re-recorded once: the four simpler verifiers' file takes the base's two deliberate changes
(an aggregate's empty results, the message for a check script that returns None), and the rubrics family's two files
the order the base writes, every value the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 5b5fcc1 into main Nov 19, 2026
36 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5922-g2-built-in-verifiers-move-onto-the branch November 19, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant