Skip to content

feat(rubrics_verifier): hooks for a subclass judge, and a step can say it isn't the task's reward - #719

Merged
sarah-chitty-1 merged 5 commits into
mainfrom
sarahchitty/rubrics-verifier-judge-hooks
Nov 17, 2026
Merged

sarah-chitty-1 merged 5 commits into
mainfrom
sarahchitty/rubrics-verifier-judge-hooks

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

First of three PRs for #682, from Edgar's review there: build the user sim judge as a rubrics_verifier subclass. This one adds only the hooks that subclass needs, plus a fix to how a task's reward is picked. The output format (#712) and the subclass come next.

Every change is off by default. An existing rubrics_verifier task sends the same judge request and writes the same entry as before.

What changed

_prompt_llm_judge

  • New optional system= (sent as a system message first) and cached_prefix= (the start of the prompt every call of the run shares). On a Claude model the prefix goes in its own text block marked cache_control: ephemeral. A prompt that doesn't start with the prefix is sent whole.
  • It now also returns usage and cost (LiteLLM's token counts and price; None when missing) so a caller can record them. The step itself records neither.
  • The step's own calls pass neither new argument. A subclass opts in by overriding _prompt_llm_judge and calling super() with them.

_criteria_batches(criteria, size) is now a method a subclass can override, for example to put a check's task items in that check's call. By default it still splits by judge_batch_size, or makes one call when that's unset. The step refuses batches that drop a criterion, repeat one, or leave a batch empty.

unjudged_rows(context, prompt_response) is a new async hook for rows no judge grades. It runs once per run, before the judge. Its rows go after the judge's in the entry and count in the score and in fail_task_when_not_passing. They stay out of the trajectory-mistakes entry. A row whose id matches a criterion's is refused. Default: no rows.

judge_trajectory_content(context, prompt_response) is a new async hook: text the judge prompt shows where the inline template puts the trajectory ($trajectory_content). It needs use_trajectory=False. A subclass can put the run there, before the criteria, so the run stays in the part of the prompt that every batch shares. judge_prompt_sections puts it after the criteria. Default: None, so the prompt is unchanged.

Batching settings now live on the format spec. JudgeOutputFormatSpec has two new fields:

  • batchable: True for rubric_binary and rubric_partial, as before. It replaces the hard-coded _BATCHABLE_FORMATS set.
  • checks_bulk_verdicts: True everywhere by default. A format whose pass reasons may rightly be short (under 20 characters) can turn the bulk-verdict retry off.

Reward: a step can say its score isn't the task's reward. Today reward_of treats every step with a verifier_id as a scoring step. A judge that checks the run, like the user sim judge, therefore makes build_app and training ask for reward=. If it is the only such step, every episode raises LookupError. The new class-level flag TaskStep.can_be_reward (default True) fixes this. With False, reward_of:

  • never picks the step;
  • treats a task whose only score is such a step as unscored (None);
  • refuses the step as reward= with a message saying why.

recorded_scores is unchanged, so the step's score is still recorded.

Tests

  • New: tst/unit/task_step/verifiers/test_rubrics_verifier_subclass_hooks.py, tst/unit/openenv/reward_test.py, and four cases in test_rubrics_verifier_litellm_contract.py. Those four go through the real litellm.acompletion with mock_response: the default request is one plain user message, the system and cache blocks have the right shape, prefixes that can't be cached are sent whole, and usage and cost are returned.
  • Full unit suite (tst/unit/ + packages/agentenv-protocol/tests/): 15811 passed, 70 skipped, 3 failed. The 3 failures are not from this branch: they fail the same way on origin/main on the same machine. They are test_docker_image_github_clone.py::test_a_token_signs_the_clone_in_from_its_credentials_file_and_no_other_helper_keeps_it (both cases time out on a git clone) and test_disk_recipe.py::test_a_windows_install_boots_its_iso_first_pressing_enter_and_finds_its_answers_on_a_removable_drive. ruff check is clean.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Adds hooks for subclass judges and lets steps opt out of supplying the task reward.

  • Rubric subclasses can group criteria and place run text in judge prompts.
  • Rubric subclasses can add rows that count toward the saved score.
  • Direct judge calls can send a system message and cache a shared prompt prefix.
  • Task steps can keep a run-check score out of reward selection.

Reviews (3) · Last reviewed commit: "reward: TaskStep keeps no can_be_reward;..." · Reviewed by Greptile

sarah-chitty-1 and others added 2 commits November 16, 2026 10:36
reward_of counted every step with a verifier_id as a scoring step, so a step whose score only checks the run
(a user sim judge) made build_app and training ask for reward=, and as a task's only such step made every
episode's reward raise LookupError. TaskStep.can_be_reward (default True) lets a step class opt out: reward_of
never picks it, a task whose only score is one is unscored, and naming it as reward= is refused with a reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All off by default; an existing rubrics_verifier sends the same request and writes the same entry.
- _prompt_llm_judge takes an optional system message and a cached_prefix (marked for Anthropic's prompt cache
  on a Claude model), and returns the call's usage and cost.
- _criteria_batches is a method a subclass can override; batches must cover every criterion once.
- unjudged_rows: rows no judge grades, merged after the judge's and scored with them.
- judge_trajectory_content: text the prompt shows as the inline trajectory, so a subclass's run sits before the
  criteria (needs use_trajectory=False).
- The format spec declares batchable (rubric_binary, rubric_partial, as before) and checks_bulk_verdicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 16, 2026 20:13
sarah-chitty-1 and others added 2 commits November 16, 2026 14:26
…rahchitty/rubrics-verifier-judge-hooks

# Conflicts:
#	src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
…s need a batchable format

judge_trajectory_content and unjudged_rows now run after the no_screenshots skip, so a skipped run never
calls them. A _criteria_batches override that splits the criteria into several calls is refused for a
format graded in one call (rubric_evidence and its mistakes variant), whose reasoning, checks and mistakes
a merge of the batches would drop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 16, 2026
…e bulk-verdict check

#719 refuses custom batches for a format whose rows don't stand alone. The user sim
format's rows do (no per-call reasoning, checks or mistakes), so it's marked batchable,
and a short pass reason is normal here, so bulk verdicts aren't flagged. Two tests
follow #712's and #719's latest rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s set

The flag is read only by reward_of, which already finds scoring steps by
getattr (verifier_id), so it reads can_be_reward the same way with a default
of True. A step that checks the run sets it on its own class; TaskStep, which
every step type shares, stays unchanged. The flag's explanation moves to
reward_of's docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator

🤖 Drafted by Claude Code, reviewed by Edgar Arakelyan before posting.

I pushed 5169c90 on top of your branch. TaskStep no longer declares can_be_reward:

  • task_step.py: the can_be_reward: ClassVar[bool] = True line and its comment are gone, so this PR leaves the base of every step type unchanged.
  • reward.py: no code change. reward_of already reads the flag with getattr(step, "can_be_reward", True), just as it finds scoring steps with getattr(step, "verifier_id", None). Its docstring now explains the flag, which the TaskStep comment used to do.
  • Steps that set the flag are unchanged. A step whose score checks the run still sets can_be_reward: ClassVar[bool] = False on its own class, as UserSimJudge in feat(verifiers): user_sim_judge, a rubrics_verifier that fails a run the user sim spoiled #682 and _RunCheck in reward_test.py already do. Nothing else changes.

Tests: reward_test.py (5) and test_rubrics_verifier_subclass_hooks.py with test_rubrics_verifier_litellm_contract.py (25) pass, and ruff is clean.

earakely-scale added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 merged commit d63bcdf into main Nov 17, 2026
50 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/rubrics-verifier-judge-hooks branch November 17, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants