Repository navigation
feat(rubrics_verifier): hooks for a subclass judge, and a step can say it isn't the task's reward - #719
Merged
Conversation
reward_of counted every step with a verifier_id as a scoring step, so a step whose score only checks the run (a user sim judge) made build_app and training ask for reward=, and as a task's only such step made every episode's reward raise LookupError. TaskStep.can_be_reward (default True) lets a step class opt out: reward_of never picks it, a task whose only score is one is unscored, and naming it as reward= is refused with a reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All off by default; an existing rubrics_verifier sends the same request and writes the same entry. - _prompt_llm_judge takes an optional system message and a cached_prefix (marked for Anthropic's prompt cache on a Claude model), and returns the call's usage and cost. - _criteria_batches is a method a subclass can override; batches must cover every criterion once. - unjudged_rows: rows no judge grades, merged after the judge's and scored with them. - judge_trajectory_content: text the prompt shows as the inline trajectory, so a subclass's run sits before the criteria (needs use_trajectory=False). - The format spec declares batchable (rubric_binary, rubric_partial, as before) and checks_bulk_verdicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rahchitty/rubrics-verifier-judge-hooks # Conflicts: # src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
…s need a batchable format judge_trajectory_content and unjudged_rows now run after the no_screenshots skip, so a skipped run never calls them. A _criteria_batches override that splits the criteria into several calls is refused for a format graded in one call (rubric_evidence and its mistakes variant), whose reasoning, checks and mistakes a merge of the batches would drop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 16, 2026
…e bulk-verdict check #719 refuses custom batches for a format whose rows don't stand alone. The user sim format's rows do (no per-call reasoning, checks or mistakes), so it's marked batchable, and a short pass reason is normal here, so bulk verdicts aren't flagged. Two tests follow #712's and #719's latest rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s set The flag is read only by reward_of, which already finds scoring steps by getattr (verifier_id), so it reads can_be_reward the same way with a default of True. A step that checks the run sets it on its own class; TaskStep, which every step type shares, stays unchanged. The flag's explanation moves to reward_of's docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
I pushed 5169c90 on top of your branch.
Tests: |
earakely-scale
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of three PRs for #682, from Edgar's review there: build the user sim judge as a
rubrics_verifiersubclass. This one adds only the hooks that subclass needs, plus a fix to how a task's reward is picked. The output format (#712) and the subclass come next.Every change is off by default. An existing
rubrics_verifiertask sends the same judge request and writes the same entry as before.What changed
_prompt_llm_judgesystem=(sent as a system message first) andcached_prefix=(the start of the prompt every call of the run shares). On a Claude model the prefix goes in its own text block markedcache_control: ephemeral. A prompt that doesn't start with the prefix is sent whole.usageandcost(LiteLLM's token counts and price; None when missing) so a caller can record them. The step itself records neither._prompt_llm_judgeand callingsuper()with them._criteria_batches(criteria, size)is now a method a subclass can override, for example to put a check's task items in that check's call. By default it still splits byjudge_batch_size, or makes one call when that's unset. The step refuses batches that drop a criterion, repeat one, or leave a batch empty.unjudged_rows(context, prompt_response)is a new async hook for rows no judge grades. It runs once per run, before the judge. Its rows go after the judge's in the entry and count in the score and infail_task_when_not_passing. They stay out of the trajectory-mistakes entry. A row whose id matches a criterion's is refused. Default: no rows.judge_trajectory_content(context, prompt_response)is a new async hook: text the judge prompt shows where the inline template puts the trajectory ($trajectory_content). It needsuse_trajectory=False. A subclass can put the run there, before the criteria, so the run stays in the part of the prompt that every batch shares.judge_prompt_sectionsputs it after the criteria. Default: None, so the prompt is unchanged.Batching settings now live on the format spec.
JudgeOutputFormatSpechas two new fields:batchable: True for rubric_binary and rubric_partial, as before. It replaces the hard-coded_BATCHABLE_FORMATSset.checks_bulk_verdicts: True everywhere by default. A format whose pass reasons may rightly be short (under 20 characters) can turn the bulk-verdict retry off.Reward: a step can say its score isn't the task's reward. Today
reward_oftreats every step with averifier_idas a scoring step. A judge that checks the run, like the user sim judge, therefore makesbuild_appand training ask forreward=. If it is the only such step, every episode raisesLookupError. The new class-level flagTaskStep.can_be_reward(default True) fixes this. With False,reward_of:reward=with a message saying why.recorded_scoresis unchanged, so the step's score is still recorded.Tests
tst/unit/task_step/verifiers/test_rubrics_verifier_subclass_hooks.py,tst/unit/openenv/reward_test.py, and four cases intest_rubrics_verifier_litellm_contract.py. Those four go through the reallitellm.acompletionwithmock_response: the default request is one plain user message, the system and cache blocks have the right shape, prefixes that can't be cached are sent whole, and usage and cost are returned.tst/unit/+packages/agentenv-protocol/tests/): 15811 passed, 70 skipped, 3 failed. The 3 failures are not from this branch: they fail the same way on origin/main on the same machine. They aretest_docker_image_github_clone.py::test_a_token_signs_the_clone_in_from_its_credentials_file_and_no_other_helper_keeps_it(both cases time out on a git clone) andtest_disk_recipe.py::test_a_windows_install_boots_its_iso_first_pressing_enter_and_finds_its_answers_on_a_removable_drive.ruff checkis clean.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge.Summary
Adds hooks for subclass judges and lets steps opt out of supplying the task reward.
Reviews (3) · Last reviewed commit: "reward: TaskStep keeps no can_be_reward;..." · Reviewed by Greptile