Skip to content

feat(verifiers): a VerifierTaskStep base writes each verifier's named grade - #761

Merged
earakely-scale merged 12 commits into
mainfrom
edgararakelyan/fd-5920-g1-a-verifiertaskstep-base-writes
Nov 18, 2026
Merged

earakely-scale merged 12 commits into
mainfrom
edgararakelyan/fd-5920-g1-a-verifiertaskstep-base-writes

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

G1 of the Grades subgraph (FD-5920, umbrella FD-5919): a VerifierTaskStep base that writes each verifier's named grade to context.metadata["grades"][<step id>], beside the legacy verifications entries every reader still takes. Moves no score.

  • task_steps/verifiers/verifier.py (re-exported from agent_env.task_step.task_steps.verifiers): VerifierTaskStep, Verification, VerifierStatus and Grade, the TypedDict readers import. A verifier implements verify(context) -> Verification; the base's execute works out its status and score once and writes the grade: name, score, min_passing_score, passed, verifier_status (graded, skipped or failed), verifier_error_reasoning, verifier_skip_reasoning, verifier_output (or verifier_output_ref over 16 KB), verifier_criteria, verifier_type, verifier_version and verifier_settings with scoring_rules: 1.
  • Failure: an exception from verify() writes a failed grade and no legacy entry (as today), then is raised unchanged, so run_outcome, fail_task_on_error and retry_config behave as now.
  • A verdict that also fails its step (VerifierNotPassingError, run_container's timeout, when they move in G2) comes back as Verification.raise_after: the grade stays graded with its score, both entries are written, then the step raises.
  • Skips carry their reason: Verification(skipped="…") gives verifier_status: skipped and the reason. A grade that isn't graded (skipped or failed) has score: None, as its legacy entry's already is for a skip, so a reader averaging scores can't count it as 0 by accident; how a skip counts in a run's result is the result rules' call (G7, G8).
  • Legacy entry: verifications[verifier_id] is {results, score}, then a dict output's keys, so a verifier whose entry carries more (a test's command and exit code, a judge's format fields) keeps it; one that wrote fields before results, as rubrics does, keeps the same contents in another key order. Rubrics' second (-trajectory-mistakes) entry is G2's call.
  • Base settings: verifier_id (defaults to the step id for a verifier new on the base; moved ones keep theirs), min_passing_score (default 1.0, config_problems refuses anything outside (0, 1]) and score_aggregator (a per-class default), each serialized only when not default, so no stored step document changes.
  • verifier_settings defaults to the step's own fields, one whose JSON is over 1 KB kept as {sha256, bytes}, plus the resolved score_aggregator, default or not, so a changed default shows as drift; min_passing_score stays on the grade itself. A subclass adds what the run changes (the step_params it applies, a judge's model) in G2.
  • Output offload: over 16 KB to the object store, from a worker thread, as verifier_output_ref: {object_url, bytes, sha256}; if the store refuses it, the output stays inline with verifier_output_offload_error and the step still passes.
  • Grade rows carry id, score, passed, weight and skipped. passed is whether the row passes under scoring rules 1, an explicit True result as ALL_PASS and ANY_PASS count it; the other defaults are the ones aggregate_score already reads a row with, so the rows pass and score as the verifier's did. The legacy rows keep the verifier's result; G10 derives both from scores and thresholds.
  • Pilot: agent_prompt_response_verifier moves onto the base. The other built-ins move in G2 (FD-5922).
  • Plugin API: the base goes into BASES; Verification, Grade, VerifierStatus, ScoreAggregator and aggregate_score into USED (agentenvhub-sdk's cua_evaluate already imports the last two).
  • Shared helpers: the package-version lookup moves into plugins/_requirements.py (distribution_holding), which bundle/requires.py now calls too, and canonical JSON into utils/canonical.py, which the bundle ledger now uses (its digests are unchanged).

Not in this PR, decided 2026-11-17: what a verifier reads (reads) lands with its first consumer, the freeze (G11) or regrade (E10), as a per-instance method over a VerifierReads enum (stored_run, live_env, live_agent_sandbox, collected_artifacts, env_snapshot); the paths a verifier uses are I5's used_paths (FD-6051). skip_on_agent_failure, keep_processes and hidden_tests come with G8, G11 and G12. Adding any of them later is not a plugin-API break.

Verified

  • The pilot's legacy entry is byte-identical to main's: the expected strings in test_agent_prompt_response_verifier.py were recorded by running the same inputs on main (810e277) before the move, for its ALL_PASS, weighted-average and prompt_error paths; its stored to_dict() is equal too, and recorded_scores reads the same score.
  • test_verifier_base.py: every grade holds exactly the fields Grade names; the full grade for graded, skipped, failed and raise_after outcomes; a direct score winning; one synthesized row when a verifier reports none; a dict output's keys joining the legacy entry, and a non-dict output's not; the pass rule at a min_passing_score; config_problems; round-trips with base fields omitted at their defaults; settings compaction; output inline, offloaded from a worker thread, and kept inline when the store refuses it; grade rows passing and scoring as the verifier's rows did under every aggregator; the grade and legacy entry surviving the step journal's replay; a subclass without verify refused by the plugin validator.
  • tst/unit/task/episode/run_test.py: a real Task.run through the stores keeps both grades on the stored run, including a failed one from a tolerated verifier.
  • The full unit tier at 96590d2 on a Linux test host: 16431 passed, 3 failed, all from the host (version_lookup_benchmark needs a git checkout; two mcp_cli_builder tests run a generated CLI on a system Python without httpx). ruff check clean. check_plugin_api.py --base origin/main: no break to the plugin surface.

Flaky tests fixed on the way

Tests this PR's CI failed on, none from its change:

  • test_a2a_agent_clock.py::test_what_the_follower_showed_of_its_config_before_never_passes_for_the_new_one[before1] (CodeBuild, assert 200 == 404). sync_time takes a follower status as this config's only when the status file's st_mtime_ns postdates the config it wrote; the test wrote the earlier status just before, and on a kernel that stamps files in coarse ticks the two shared an mtime. The earlier status is now a second older, as a follower's is. Made the status no older than the config and the unchanged test fails exactly as CodeBuild did; 1 of CodeBuild's last 200 builds hit it. Product code unchanged: <= would refuse a status the follower writes in the same tick after reading the new config, which is worse.
  • computer_env_test.py::test_an_env_that_changes_only_its_display_and_name_reuses_the_image (integration-local-slow (8/12), assert 85.58 < 60). It bounded the whole ComputerEnv.put at 60 s, but a reused put still boots and probes the image to mark it ready. The timing bound is dropped; the checks that it reused (the same image and installed packages, reused, not emulated) stay.
  • disk_test.py (3 tests) and container_stopped_test.py (integration-local, 429 Too Many Requests from public.ecr.aws for busybox:1.37). The fast tier builds the disk tests' root filesystem with no mirror step, so an ECR Public rate-limit window fails it. The rootfs now builds FROM mirror.gcr.io/library/busybox:1.37@<the same digest>, which mirror.gcr.io serves for amd64 and arm64; all four pass on a Linux test host from it.

For the docs

The user guide's verifier page should cover: VerifierTaskStep and Verification for plugin authors, the named grade's fields, min_passing_score, and that verifications stays as it is.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

Adds named verifier grades beside the existing verification results, with agent_prompt_response_verifier as the first built-in user.

  • Verifier steps now save a named grade beside their legacy result.
  • Bundle code now uses shared helpers for package lookup and canonical JSON.
  • Test setup changes make clock and image checks less dependent on CI timing.

No new actionable issues found in the latest change.

Reviews (18) · Last reviewed commit: "test(computer): the disk tests' root fil..." · Reviewed by Greptile

… grade

A verifier on the new base implements verify() and returns a Verification;
the base scores it and writes context.metadata["grades"][<step id>] beside
the legacy verifications entries, which stay byte for byte as before. An
exception from verify() is a failed grade, raised again; raise_after keeps a
recorded verdict graded and still fails the step; a skip carries its reason.
agent_prompt_response_verifier moves onto the base as the pilot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 03:47
earakely-scale and others added 3 commits November 17, 2026 19:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hared helpers

The base works out a verification's status and score once, for both its grade and its legacy entry, and builds the
grade as a Grade TypedDict, which readers import. The package-version lookup and canonical JSON it needs move to
plugins/_requirements.py and utils/canonical.py, which bundle/requires.py and the ledger now share. The base's tests
go through execute and the step journal's replay rather than its private helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lace of legacy_entries

A verifier whose legacy verifications entry holds more than its results and score (a test's command and exit code, a
judge's format fields) returns those as its output, which the grade keeps too, rather than spelling out whole entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 18, 2026 09:14
…, not result

The grade's copy of each criterion row carries passed, whether the row passes under scoring rules 1 (an explicit True
result, as ALL_PASS and ANY_PASS count it), as the grade itself does one level up. The legacy verifications rows keep
the verifier's result unchanged for the readers that take it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0-g1-a-verifiertaskstep-base-writes

# Conflicts:
#	src/agent_env/bundle/ledger.py
earakely-scale and others added 3 commits November 18, 2026 09:47
…nfig it's checked against

The test wrote the follower's earlier status and then had sync_time write the new config, which a status must
postdate (by st_mtime_ns) to count. On a kernel whose file timestamps advance in coarse ticks, the two writes can share
an mtime, and the earlier status of the same gateway and name then passed for the new config: once in CodeBuild's
last 200 runs. The earlier status is now a second older, as a follower's earlier status is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot how long the put took

The test also bounded ComputerEnv.put at 60 s, which reuses the image but still boots and probes it to mark it ready;
on a busy runner that took 85 s (one slow shard on #761). Reuse itself is what its other checks assert: the same image,
the same installed packages, reused and not emulated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings name the aggregator

A skipped or failed grade's score is None rather than 0, as its legacy entry's already is, so a reader averaging scores
can't count a skip or a broken verifier as a zero; how a skip counts in a run's result is the result rules' call.
verifier_settings always names the score_aggregator the step scores with, default or not, so a changed default shows
as drift, and leaves out min_passing_score, which the grade holds itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 3 commits November 18, 2026 10:19
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0-g1-a-verifiertaskstep-base-writes

# Conflicts:
#	src/agent_env/task_step/task_steps/verifiers/agent_prompt_response_verifier.py
…r.io's busybox

The fast integration tier pulls its bases straight from their registries, with no mirror step, and ECR Public answered
429 for busybox on #761's run, failing the three disk tests and the stopped-container test. mirror.gcr.io serves Docker
Hub's busybox under the same digest for amd64 and arm64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@andrewpark-scaleai andrewpark-scaleai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. read verifier.py in full and the pilot diff, the legacy entry stays the same and the grade is additive. small things for g2, none blocking:

  • _output runs json.dumps(output) after verify returns and outside the try, so a plugin output with a datetime/set/Path fails the step with TypeError and writes neither the grade nor the legacy entry. default=str or catching it into a failed grade would cover it. the pilot returns no output so nothing hits it today.
  • the pilot's prompt_error path still returns score=0, so its grade says graded and passed: false for a run where nothing was judged. fine for keeping the legacy entry byte-identical, but anything reading grades before g2 will count agent errors as failed verdicts.
  • min_passing_score defaults to 1.0 for every aggregator, so a weighted 0.857 is not passed. worth a line in the docs for anyone moving a weighted verifier.
  • a row's default score treats any truthy result as 1.0 but passed needs result is True, so result: 1 scores 1.0 and isn't passed. matches how ALL_PASS already counts, just flagging for g10.

@earakely-scale
earakely-scale merged commit 11a7c3a into main Nov 18, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5920-g1-a-verifiertaskstep-base-writes branch November 18, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants