Skip to content

feat(verifiers): user_sim_judge, a rubrics_verifier that fails a run the user sim spoiled - #682

Merged
sarah-chitty-1 merged 68 commits into
mainfrom
sarahchitty/fd-5490-user-sim-judge-v2
Nov 17, 2026
Merged

sarah-chitty-1 merged 68 commits into
mainfrom
sarahchitty/fd-5490-user-sim-judge-v2

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Merges after #719, the rubrics_verifier judge hooks ( _prompt_llm_judge(system=, cached_prefix=) with usage and cost, _criteria_batches, unjudged_rows, judge_trajectory_content, can_be_reward) and #712 (the user_sim_checks judge output format). Both are merged into this branch; review the last commit for this PR's own change.

A new verifier step, user_sim_judge, grades the user sim of a run (the AI playing the user), not the assistant (FD-5490), for always_on runs and prompt_agent chats with the built-in user sim (deploy_user_sim_agent). When the user sim slips in a way that could change the assistant's grade, the task fails, so the run is done again rather than scored.

Built on rubrics_verifier

UserSimJudge is a RubricsVerifierTaskStep subclass, as Edgar suggested, on the always_on_agent_verifier pattern:

  • Pinned: the direct judge, use_trajectory off, output_format: user_sim_checks.
  • The run (the rebuilt instructions and every check-in) is judge_trajectory_content, so everything up to its end is the same in every call and is marked for the prompt cache. The run's first call goes alone so the rest read it from the cache. A system message says the judge grades the user sim.
  • The criteria are the 7 checks plus the task's rubric_items, one call per check with the items its families own (_criteria_batches, at most 12 items a call). Auto items, na_when: always items and line items whose line can't be read come from unjudged_rows, with no call.
  • Deleted: the step's own model call, model defaults, step_params handling, answer-retry loop and concurrency. rubrics_verifier does them now. A run's user_overrides.judge_model picks the model, as for every judge.
  • user_sim_judge_core keeps the checks, families, item model, render_run and the counters (standard library only). It also builds one run's criteria (plan_run) and reads the rows (user_sim_error).
  • The load checks reuse rate_user_satisfaction's run-step finder and its built-in-user-sim check.
Check Families
guidance A1, A3
oversharing G1, G2
delegation A4
style F1–F4
rogue G3, G5
made_up A2
ending G4

Task items keep every rule they had: line tags read from the run's tagged_lines, rubric_persona_sha256 (a mismatch skips the items, and the checks still run), na_when, static_na_reason, auto items counted by max_words / max_sentences, and cut or missing answers making an auto item error. The judge's fields as psi-benchmark-tasks writes them are unchanged: run_step_id, judge_model, rubric_items, rubric_persona_sha256.

What a task sets

Only these (beyond every step's id, type, version, depends_on and retry_config):

  • run_step_id, required: the always_on or prompt_agent step whose user sim it judges. The judge never guesses the run.
  • rubric_items and rubric_persona_sha256.
  • judge_model and judge_timeout_seconds, both optional (rubrics_verifier's defaults).
  • fail_task_when_not_passing, optional, true by default.
  • verifier_id.

Everything else is fixed, and a task that sets it is refused at load with a plain message:

  • checks and error_checks: the judge always asks all 7 checks, and every check but style can fail a run. An item's own can_error works as before.
  • fail_task_on_error: false: the default is true, so nobody needs to write it.
  • Any rubrics_verifier setting the judge fixes (use_trajectory, output_format, use_agent_judge, judge_batch_size, criteria, prompt_id ...), unless it is given its fixed value.

Removed on purpose: the judge's earlier checks and error_checks let a task skip a check or stop one from failing a run. They're gone so every task is judged the same way and there are fewer ways to misconfigure it; a check that doesn't apply comes back n/a, and a noisy check gets fixed for every task. If a real task needs per-task control, add them back.

What it writes

Everything goes in one place, verifications[verifier_id]. It has format: user_sim_checks, the rows and a score, as rubrics_verifier writes them. can_be_reward is False, so the score is never the task's reward.

  • Rows: each row has its verdict, reason, cited turns, could_change_grade, assistant_asked, weight and, for items, asked_with.
  • user_sim_error: yes, no or unknown, with the failing rows and fails_task.
  • Run details: the instructions check, including whether the rebuilt prompt's sha256 matches the recorded one; the persona fingerprint status; the notes; calls, usage and cost.

The gate

fail_task_when_not_passing replaces mark_user_sim_error. It is on by default. A task that still sets mark_user_sim_error is refused at load, with a message pointing to fail_task_when_not_passing.

  • The rule: the step raises VerifierNotPassingError, naming the failing rows, only when user_sim_error is yes and the rebuilt instructions match the recorded prompt_sha256.
  • Yes: some check or task item failed and the judge says that fail could change the grade. This one rule covers checks and items.
  • Style never gates. Its rows have weight 0. An item can gate when its can_error is true, or by default when its family's check gates.
  • assistant_asked is recorded on each row and never changes the gate. This is a benchmark: a grade-changing slip spoils the run whoever caused it, so the run is done again.
  • Reward steps: with the gate on, every step that can be the task's reward must depend on the judge, directly or through other steps. Otherwise the task is refused at load.

A judge that can't reach its model fails loudly

A setup error raises at once: "the user sim judge can't reach its model: …". The setup errors are:

  • agent-env's ConfigError (no key, missing config);
  • LiteLLM's AuthenticationError, PermissionDeniedError, NotFoundError (an unknown model), BadRequestError (and its kinds), UnprocessableEntityError and BudgetExceededError.

The run's other calls aren't made. Every other failure is taken as flaky and handled as below: Timeout, RateLimitError, ServiceUnavailableError, InternalServerError, BadGatewayError, APIConnectionError, or an answer still malformed after its retries.

Unknown never fails

A broken judge answer is asked again with the format's correction, up to 3 more times. A failed model call is retried as rubrics_verifier does. If a check is still undecided, its rows are error and user_sim_error is unknown. The entry is flagged "user sim judge undecided", the step doesn't raise, and the assistant's score is untouched. If no check at all was decided, the flag instead says "the judge couldn't decide any check — check its setup".

This is deliberately different from rubrics_verifier, which fails a task when its judge gives up. Here only a slip the judge actually found fails a run, so a flaky judge call never throws away a good run.

Tests

  • test_user_sim_judge.py (93 tests, fake model): every earlier test ported, plus:
    • the gate fails on yes with matching instructions;
    • no fail on unknown (4 tries, then flagged);
    • no fail when the instructions don't match;
    • style never gates;
    • assistant_asked is recorded but doesn't gate;
    • with a grader in the task, the grader is the reward;
    • the load refusals: a missing run_step_id, checks / error_checks, fail_task_on_error: false, a fixed setting given another value;
    • each setup error raising, each flaky error leaving the run undecided, and the all-undecided flag;
    • the litellm request: system message, cached prefix, model-call retries.
  • tst/unit/task_step, task/episode, training, cli and openenv pass. The 2 test_mcp_cli_builder failures are local only: the generated script's Python has no click.
  • A free check on the two stored end-to-end runs (always_on and chat), with a fake model, reads their real records. Those runs' instructions no longer rebuild, because their personas still have the removed never_supply section, so on them a slip is recorded and doesn't fail the task.

Not done: calibration against a person's labels.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issue was found.

Summary

Adds user_sim_judge to check whether the simulated user spoiled a run.

  • The user-sim judge checks whether a run could have changed the assistant’s grade.
  • User-sim run records now keep a fingerprint of the persona and opening.

The supplied previous threads are unnumbered, so they receive no numbered ledger entries. Their fixes remain present. Review used the full PR range because the current and previous review commits have different merge bases with the target.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Read the recorded run] --> B[Rebuild and compare instructions]
  B --> C[Plan seven checks and task items]
  C --> D[Ask the judge and compute automatic items]
  D --> E{Grade-changing slip found?}
  E -->|No or unknown| F[Keep the assistant's score unchanged]
  E -->|Yes| G{Gate on and instructions match?}
  G -->|Yes| H[Fail the task before reward steps]
  G -->|No| I[Record the slip only]
Loading

Reviews (27) · Last reviewed commit: "Merge branch 'main' of github.com:scalea..." · Reviewed by Greptile

sarah-chitty-1 and others added 14 commits November 12, 2026 13:55
…an always_on run

Started Nov 10 on the old user-sim branch and moved onto main unchanged, apart from the user sim modules' new
task_steps/agent/ path. Not reviewed or calibrated; the design is still open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for a 1-5 rating after the last read (FD-5491)

A description's satisfaction lines go in the prompt as S1... A user sim with them rates the run once it's over,
recorded in metadata["user_sim_satisfaction"][step_id], never delivered; without them nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ended, recorded like always_on's (FD-5491)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…with one judge model, and records only

- Both kinds of run with the built-in user sim: an always_on step's stored timeline, or a prompt_agent step's
  user_sim_reads, each read whole from its object_url. run_step_id names the step; unset, the run must have one.
- The user sim's instructions are rebuilt from its user_sims record with user_sim_versions.setup, as
  deploy_user_sim_agent wrote them, and checked against the recorded prompt_sha256.
- One judge_model, defaulting to rubrics_verifier's (the config's judge role, else claude-sonnet-4-6); the two-judge
  split is gone. user_sim_error is yes / no / unknown (a check that could make it one couldn't be judged).
- Record only: mark_user_sim_error (off by default) sets the prompt response's error_type to user_sim_error, only
  when the instructions match the recorded hash. The step doesn't fail the task by default.
- Tests: a fake judge, one always_on and one prompt_agent run, every check's verdict, task items, the hash check,
  errored runs skipped, marking off by default and on when opted in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, in always_on and prompt_agent (FD-5491)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (FD-5491)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deploy_user_sim_agent's opening_prompt also takes a list of text and file parts, in prompt_agent's parts
format; a plain string works as before. A file part may carry a one-line metadata.description for the user
sim, which can't open files. The parts are checked when the task loads.

- always_on sends the parts on the first wake in order: the joined text as the user's labelled line, each
  file in its place, without the description; the graded conversation's first user turn has the files.
- The user sim's prompt and first read say what it attached: "(you attached itinerary.pdf: Mom and Dad's
  flight itinerary)", or the file name alone with no description. Text-only openings are byte-identical.
- prompt_agent with the built-in user sim compares the files too: the same files in the same order after
  seeding, and its error names both lists.
- Files go through the same sharing and seed filling as any prompt file part (apply_seed, lifted out of
  PromptAgentTaskStep._apply_seed, which now calls it).
- The record keeps "opening" as the text, so text readers are unchanged, and adds "opening_parts" (as
  written) for an opening given as parts. The training harness's first prompt is the opening's text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atisfaction; always_on and prompt_agent no longer rate (FD-5491)

The satisfaction lines move from the persona to the new step, so the user sim's system prompt never has them and
the run behaves as tested. The step runs after the always_on or prompt_agent step it rates and asks the user sim
once, in the session the run used, with what the user never read; a malformed answer gets one retry. The rating is
recorded at metadata["user_sim_satisfaction"][run step id] and sent nowhere. always_on and prompt_agent are as they
were before the first build, apart from the opening-files work, and always_on's record now names its user sim
session (user_sim.context_id), which the step asks in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt, its load checks, and no S lines in the prompt (FD-5491)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tep's (FD-5491)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k (G4)

- guidance: an unanswered question fails only when the assistant needs it answered; giving
  something again when asked is now a fail line. Self-contradiction moves to made_up.
- oversharing: something the assistant should know from its notes, credentials, and answering
  a may-I-share question against PRIVATE lines or rules.
- delegation: a broken "ask first" or "never" item must be objected to (the user sim's prompt
  always says so); a "tell you after" item left out of the final report must be asked about once.
- rogue: letting the assistant off for a broken limit or skipped step.
- style: same-length messages, described traits and caricature, missing pushback or reactions,
  and acting out of step with its situation (F4).
- ending (G4): a new check, run by default and able to mark a run. The step tells the judge
  when the run ended at the user sim's max_replies, which its instructions never mention.
- docstring: A5 is the separate rate_user_satisfaction step's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tty/fd-5490-user-sim-judge-v2

# Conflicts:
#	src/agent_env/task/episode/rules.py
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 14, 2026 00:30
parse_answer kept a "fail" whose cited turns were all outside the graded
ones (turns=[] but could_change_grade still true), an unsupported failure
that could mark the run. A check's or task item's fail must now cite at
least one graded turn; otherwise the answer is not usable, so the judge is
asked once more and a second such answer gives the check (or item) the
verdict "error". Pass and n/a answers are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 3 commits November 13, 2026 16:38
…ser sim's file descriptions

- rate_user_satisfaction: a failed_steps entry marked retried is a failure a retry
  recovered from, so it no longer skips the rating.
- prompt_agent with the built-in user sim: file parts copied from the opening_prompt
  are sent and stored without metadata.description, as always_on sends them; the
  user sim still gets the descriptions from its record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no failure is asked again

The answer format put the verdict first, so the judge committed to "fail" and then argued itself out of it ("No clear
failure found"); in 2 of 4 real runs that alone made the run a user sim error. The prompt and schema now ask for the
reason, then turns, then verdict and could_change_grade (items alike), and say to decide after reasoning, within the
40-word limit. As a backstop, a fail whose reason says no failure was found is not usable, so it is asked again once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 3 commits November 13, 2026 17:39
…o failure; name the error-note limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An item has id, family, kind (hard / general / auto), check (text still
loads as its old name), name, short, na_when and, for auto, metric and max.
Hard items are told to the judge as exact rules, general ones fail only on a
clear instance. Auto items are counted on the graded answers with no model
call. na_when always is n/a with its static_na_reason and no call; never
asks an n/a again. A call takes at most MAX_ITEMS_PER_CALL (12) items; the
rest go in item-only calls. rubric_task_version is compared with the run's
task instance version: on a mismatch the items aren't graded and the entry
says so, the general checks still run. Bad items are refused with a clear
error per item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e; the opening wake names its files; runs record each tag's line

- An opening with files adds one line under the "(you attached …)" note in W2: the user sim remembers only what's
  said of each file, not its contents, and says so or points the assistant to the file. Text-only openings are
  byte-identical (the golden prompt tests are untouched).
- always_on's timeline wake event records the files it carries, by name and uri, never their descriptions.
- deploy_user_sim_agent records tagged_lines ({"W1": ..., "D9": ...}) in metadata["user_sims"][name], so an answer's
  `used` resolves against the lines that run's user sim saw after a persona is reordered; rate_user_satisfaction
  records its S-tag map with the rating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 4 commits November 13, 2026 18:05
…d (T) and things the assistant should already know (C)

From the hand-written user sim template's standing_knowledge and assumed_context. Asked again for something
already told, the user sim reminds the assistant, a little annoyed, and gives it again; it never brings up what the
assistant should already know, and corrects it briefly when it gets one wrong. Both are validated like the other
sections; a persona without them renders as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t's own path; auto items take only na_when always

The judge rebuilt the prompt from the record's opening text alone, so an opening with files never matched
prompt_sha256 (no "(you attached ...)" notes, no ATTACHED_KNOWN line). deploy_user_sim_agent's person() is now a
module function, and recorded_person() builds it from the record (opening_parts when present); the judge calls that
and user_sim_versions.setup, so the files and the new already_told / should_already_know sections rebuild as deployed.

An auto item is counted, never judged, so auto_result ignored any na_when but always: a conditional na_when did
nothing and na_when never still gave n/a with no messages. Any other value is now refused when the task loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, not the task version

Any edit to a hub task makes a new version, so rubric_task_version skipped a rubric after the next edit even when the
user sim's person didn't change. user_sim_common.persona_fingerprint hashes the persona's description, permissions and
keep_private and the opening_prompt as written (recipe in its docstring, pinned by a test, since
psi-benchmark-tasks's converter computes it too); deploy_user_sim_agent records it as persona_sha256. The judge's
rubric_persona_sha256 replaces rubric_task_version: a mismatch skips the task items and still runs the checks, a
match grades them, a run with no fingerprint is unknown and grades them. The task-instance lookup is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 16, 2026 13:12
…gent judge's timeout is undecided

Task.run fills the seed into every step before it runs, so deploy_user_sim_agent fingerprinted
the seeded persona and opening while rubric_persona_sha256 is the unseeded one: a seeded task's
items were never graded. The step now takes the fingerprint when it's built, as written, and
substitute_seed_in_steps hands it to the seeded copy (seeded_from). The test runs it through
Task.run with a seed.

With a run-wide use_agent_judge, the judge call can raise TimeoutError, JudgeTimeoutError or
AgentUnreachableError; those now leave the call's rows undecided like the other flaky errors,
and setup errors still raise. judge_timeout_retries and judge_max_turns, new on main, are fixed
settings too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 16, 2026 13:49
…ver raised

A result whose id isn't text ({"results": [{"id": []}]}) reached the
shared id check, which hashes each id and raised TypeError, failing the
task instead of sending the answer back for correction. Each id is now
checked first and a non-text one is a row problem. The same goes for a
reason that isn't text and turns that aren't a list. An infinite turn
number (OverflowError) is dropped like any other entry that isn't a turn,
and `results` that isn't a list is named as such.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1

Copy link
Copy Markdown
Collaborator Author

Thanks, done in your three-PR split:

  1. feat(rubrics_verifier): hooks for a subclass judge, and a step can say it isn't the task's reward #719: the rubrics_verifier hooks (system / cached_prefix with usage and cost, an overridable _criteria_batches, unjudged_rows, judge_trajectory_content, and batchable / checks_bulk_verdicts on the format spec), plus TaskStep.can_be_reward, which fixes the reward_of bug.
  2. feat(verifiers): user_sim_checks judge output format, for grading the user sim #712: USER_SIM_CHECKS.
  3. feat(verifiers): user_sim_judge, a rubrics_verifier that fails a run the user sim spoiled #682: UserSimJudge(RubricsVerifierTaskStep). Its own model call, retries and concurrency are deleted. The result is in verifications[verifier_id], and its settings are down to run_step_id, rubric_items, rubric_persona_sha256, judge_model, judge_timeout_seconds and fail_task_when_not_passing. The pinned rubrics_verifier settings are refused.

One deliberate difference: when the judge can't decide, it records "unknown" instead of failing the task, following your "only yes fails". The agent-judge path for long runs is a follow-up (FD-5937): today's runs are about 1k tokens per check-in, and the cached single prompt is cheaper.

sarah-chitty-1 and others added 9 commits November 16, 2026 14:25
…rahchitty/rubrics-verifier-judge-hooks

# Conflicts:
#	src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
…sked again, not read as a turn

int() read `true` and `1.9` as turn 1, so a fail could gate on a turn the judge never cited. Any cited turn
that isn't an integer is now a row problem for the correction prompt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s need a batchable format

judge_trajectory_content and unjudged_rows now run after the no_screenshots skip, so a skipped run never
calls them. A _criteria_batches override that splits the criteria into several calls is refused for a
format graded in one call (rubric_evidence and its mistakes variant), whose reasoning, checks and mistakes
a merge of the batches would drop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…:scaleapi/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…e bulk-verdict check

#719 refuses custom batches for a format whose rows don't stand alone. The user sim
format's rows do (no per-call reasoning, checks or mistakes), so it's marked batchable,
and a short pass reason is normal here, so bulk verdicts aren't flagged. Two tests
follow #712's and #719's latest rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its last "but"

"No clear failure was found at turn 1, but at turn 2 she leaked the password" supports
a fail; it was sent back because the no-failure phrase matched anywhere in the last
sentence. The conclusion is now the last sentence after its last contrast word.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
sarah-chitty-1 and others added 7 commits November 16, 2026 14:58
…onclusion

"No clear failure was found, although turn 2 was awkward" concludes nothing failed; only
what follows but / however / yet is read as the conclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… again

"No clear failures were found except at turn 2, where she leaked the password" is a fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…s set

The flag is read only by reward_of, which already finds scoring steps by
getattr (verifier_id), so it reads can_be_reward the same way with a default
of True. A step that checks the run sets it on its own class; TaskStep, which
every step type shares, stays unchanged. The flag's explanation moves to
reward_of's docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rahchitty/fd-5490-user-sim-judge-v2

# Conflicts:
#	tst/unit/task_step/verifiers/test_rubrics_verifier_subclass_hooks.py
…rahchitty/fd-5490-user-sim-judge-v2

# Conflicts:
#	src/agent_env/task_step/task_steps/verifiers/judge_utils/judge_output_format.py
@sarah-chitty-1
sarah-chitty-1 merged commit 555f1f0 into main Nov 17, 2026
35 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/fd-5490-user-sim-judge-v2 branch November 17, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants