Repository navigation
feat(verifiers): a VerifierTaskStep base writes each verifier's named grade - #761
Merged
earakely-scale merged 12 commits intoNov 18, 2026
Merged
earakely-scale merged 12 commits into
earakely-scale merged 12 commits into
Conversation
… grade A verifier on the new base implements verify() and returns a Verification; the base scores it and writes context.metadata["grades"][<step id>] beside the legacy verifications entries, which stay byte for byte as before. An exception from verify() is a failed grade, raised again; raise_after keeps a recorded verdict graded and still fails the step; a skip carries its reason. agent_prompt_response_verifier moves onto the base as the pilot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hared helpers The base works out a verification's status and score once, for both its grade and its legacy entry, and builds the grade as a Grade TypedDict, which readers import. The package-version lookup and canonical JSON it needs move to plugins/_requirements.py and utils/canonical.py, which bundle/requires.py and the ledger now share. The base's tests go through execute and the step journal's replay rather than its private helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lace of legacy_entries A verifier whose legacy verifications entry holds more than its results and score (a test's command and exit code, a judge's format fields) returns those as its output, which the grade keeps too, rather than spelling out whole entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not result The grade's copy of each criterion row carries passed, whether the row passes under scoring rules 1 (an explicit True result, as ALL_PASS and ANY_PASS count it), as the grade itself does one level up. The legacy verifications rows keep the verifier's result unchanged for the readers that take it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0-g1-a-verifiertaskstep-base-writes # Conflicts: # src/agent_env/bundle/ledger.py
…nfig it's checked against The test wrote the follower's earlier status and then had sync_time write the new config, which a status must postdate (by st_mtime_ns) to count. On a kernel whose file timestamps advance in coarse ticks, the two writes can share an mtime, and the earlier status of the same gateway and name then passed for the new config: once in CodeBuild's last 200 runs. The earlier status is now a second older, as a follower's earlier status is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot how long the put took The test also bounded ComputerEnv.put at 60 s, which reuses the image but still boots and probes it to mark it ready; on a busy runner that took 85 s (one slow shard on #761). Reuse itself is what its other checks assert: the same image, the same installed packages, reused and not emulated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings name the aggregator A skipped or failed grade's score is None rather than 0, as its legacy entry's already is, so a reader averaging scores can't count a skip or a broken verifier as a zero; how a skip counts in a run's result is the result rules' call. verifier_settings always names the score_aggregator the step scores with, default or not, so a changed default shows as drift, and leaves out min_passing_score, which the grade holds itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0-g1-a-verifiertaskstep-base-writes # Conflicts: # src/agent_env/task_step/task_steps/verifiers/agent_prompt_response_verifier.py
…r.io's busybox The fast integration tier pulls its bases straight from their registries, with no mirror step, and ECR Public answered 429 for busybox on #761's run, failing the three disk tests and the stopped-container test. mirror.gcr.io serves Docker Hub's busybox under the same digest for amd64 and arm64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andrewpark-scaleai
approved these changes
Nov 18, 2026
andrewpark-scaleai
left a comment
Contributor
There was a problem hiding this comment.
lgtm. read verifier.py in full and the pilot diff, the legacy entry stays the same and the grade is additive. small things for g2, none blocking:
_outputrunsjson.dumps(output)after verify returns and outside the try, so a plugin output with a datetime/set/Path fails the step with TypeError and writes neither the grade nor the legacy entry.default=stror catching it into a failed grade would cover it. the pilot returns no output so nothing hits it today.- the pilot's prompt_error path still returns score=0, so its grade says graded and passed: false for a run where nothing was judged. fine for keeping the legacy entry byte-identical, but anything reading
gradesbefore g2 will count agent errors as failed verdicts. - min_passing_score defaults to 1.0 for every aggregator, so a weighted 0.857 is not passed. worth a line in the docs for anyone moving a weighted verifier.
- a row's default score treats any truthy
resultas 1.0 butpassedneedsresult is True, soresult: 1scores 1.0 and isn't passed. matches how ALL_PASS already counts, just flagging for g10.
earakely-scale
deleted the
edgararakelyan/fd-5920-g1-a-verifiertaskstep-base-writes
branch
November 18, 2026 20:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
G1 of the Grades subgraph (FD-5920, umbrella FD-5919): a
VerifierTaskStepbase that writes each verifier's named grade tocontext.metadata["grades"][<step id>], beside the legacyverificationsentries every reader still takes. Moves no score.task_steps/verifiers/verifier.py(re-exported fromagent_env.task_step.task_steps.verifiers):VerifierTaskStep,Verification,VerifierStatusandGrade, theTypedDictreaders import. A verifier implementsverify(context) -> Verification; the base'sexecuteworks out its status and score once and writes the grade:name,score,min_passing_score,passed,verifier_status(graded, skipped or failed),verifier_error_reasoning,verifier_skip_reasoning,verifier_output(orverifier_output_refover 16 KB),verifier_criteria,verifier_type,verifier_versionandverifier_settingswithscoring_rules: 1.verify()writes a failed grade and no legacy entry (as today), then is raised unchanged, sorun_outcome,fail_task_on_errorandretry_configbehave as now.VerifierNotPassingError,run_container's timeout, when they move in G2) comes back asVerification.raise_after: the grade staysgradedwith its score, both entries are written, then the step raises.Verification(skipped="…")givesverifier_status: skippedand the reason. A grade that isn'tgraded(skipped or failed) hasscore: None, as its legacy entry's already is for a skip, so a reader averaging scores can't count it as 0 by accident; how a skip counts in a run's result is the result rules' call (G7, G8).verifications[verifier_id]is{results, score}, then a dictoutput's keys, so a verifier whose entry carries more (a test's command and exit code, a judge's format fields) keeps it; one that wrote fields beforeresults, as rubrics does, keeps the same contents in another key order. Rubrics' second (-trajectory-mistakes) entry is G2's call.verifier_id(defaults to the step id for a verifier new on the base; moved ones keep theirs),min_passing_score(default 1.0,config_problemsrefuses anything outside (0, 1]) andscore_aggregator(a per-class default), each serialized only when not default, so no stored step document changes.verifier_settingsdefaults to the step's own fields, one whose JSON is over 1 KB kept as{sha256, bytes}, plus the resolvedscore_aggregator, default or not, so a changed default shows as drift;min_passing_scorestays on the grade itself. A subclass adds what the run changes (thestep_paramsit applies, a judge's model) in G2.verifier_output_ref: {object_url, bytes, sha256}; if the store refuses it, the output stays inline withverifier_output_offload_errorand the step still passes.id,score,passed,weightandskipped.passedis whether the row passes under scoring rules 1, an explicitTrueresultas ALL_PASS and ANY_PASS count it; the other defaults are the onesaggregate_scorealready reads a row with, so the rows pass and score as the verifier's did. The legacy rows keep the verifier'sresult; G10 derives both from scores and thresholds.agent_prompt_response_verifiermoves onto the base. The other built-ins move in G2 (FD-5922).BASES;Verification,Grade,VerifierStatus,ScoreAggregatorandaggregate_scoreintoUSED(agentenvhub-sdk'scua_evaluatealready imports the last two).plugins/_requirements.py(distribution_holding), whichbundle/requires.pynow calls too, and canonical JSON intoutils/canonical.py, which the bundle ledger now uses (its digests are unchanged).Not in this PR, decided 2026-11-17: what a verifier reads (
reads) lands with its first consumer, the freeze (G11) or regrade (E10), as a per-instance method over aVerifierReadsenum (stored_run,live_env,live_agent_sandbox,collected_artifacts,env_snapshot); the paths a verifier uses are I5'sused_paths(FD-6051).skip_on_agent_failure,keep_processesandhidden_testscome with G8, G11 and G12. Adding any of them later is not a plugin-API break.Verified
test_agent_prompt_response_verifier.pywere recorded by running the same inputs on main (810e277) before the move, for its ALL_PASS, weighted-average andprompt_errorpaths; its storedto_dict()is equal too, andrecorded_scoresreads the same score.test_verifier_base.py: every grade holds exactly the fieldsGradenames; the full grade for graded, skipped, failed andraise_afteroutcomes; a direct score winning; one synthesized row when a verifier reports none; a dict output's keys joining the legacy entry, and a non-dict output's not; the pass rule at amin_passing_score;config_problems; round-trips with base fields omitted at their defaults; settings compaction; output inline, offloaded from a worker thread, and kept inline when the store refuses it; grade rows passing and scoring as the verifier's rows did under every aggregator; the grade and legacy entry surviving the step journal's replay; a subclass withoutverifyrefused by the plugin validator.tst/unit/task/episode/run_test.py: a realTask.runthrough the stores keeps both grades on the stored run, including a failed one from a tolerated verifier.version_lookup_benchmarkneeds a git checkout; twomcp_cli_buildertests run a generated CLI on a system Python without httpx).ruff checkclean.check_plugin_api.py --base origin/main: no break to the plugin surface.Flaky tests fixed on the way
Tests this PR's CI failed on, none from its change:
test_a2a_agent_clock.py::test_what_the_follower_showed_of_its_config_before_never_passes_for_the_new_one[before1](CodeBuild,assert 200 == 404).sync_timetakes a follower status as this config's only when the status file'sst_mtime_nspostdates the config it wrote; the test wrote the earlier status just before, and on a kernel that stamps files in coarse ticks the two shared an mtime. The earlier status is now a second older, as a follower's is. Made the status no older than the config and the unchanged test fails exactly as CodeBuild did; 1 of CodeBuild's last 200 builds hit it. Product code unchanged:<=would refuse a status the follower writes in the same tick after reading the new config, which is worse.computer_env_test.py::test_an_env_that_changes_only_its_display_and_name_reuses_the_image(integration-local-slow (8/12),assert 85.58 < 60). It bounded the wholeComputerEnv.putat 60 s, but a reused put still boots and probes the image to mark it ready. The timing bound is dropped; the checks that it reused (the same image and installed packages,reused, not emulated) stay.disk_test.py(3 tests) andcontainer_stopped_test.py(integration-local,429 Too Many Requestsfrompublic.ecr.awsforbusybox:1.37). The fast tier builds the disk tests' root filesystem with no mirror step, so an ECR Public rate-limit window fails it. The rootfs now buildsFROM mirror.gcr.io/library/busybox:1.37@<the same digest>, which mirror.gcr.io serves for amd64 and arm64; all four pass on a Linux test host from it.For the docs
The user guide's verifier page should cover:
VerifierTaskStepandVerificationfor plugin authors, the named grade's fields,min_passing_score, and thatverificationsstays as it is.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new blocking issues were found.Summary
Adds named verifier grades beside the existing verification results, with
agent_prompt_response_verifieras the first built-in user.No new actionable issues found in the latest change.
Reviews (18) · Last reviewed commit: "test(computer): the disk tests' root fil..." · Reviewed by Greptile