Repository navigation
feat(verifiers): user_sim_judge, a rubrics_verifier that fails a run the user sim spoiled - #682
Merged
Merged
Conversation
…an always_on run Started Nov 10 on the old user-sim branch and moved onto main unchanged, apart from the user sim modules' new task_steps/agent/ path. Not reviewed or calibrated; the design is still open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for a 1-5 rating after the last read (FD-5491) A description's satisfaction lines go in the prompt as S1... A user sim with them rates the run once it's over, recorded in metadata["user_sim_satisfaction"][step_id], never delivered; without them nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ended, recorded like always_on's (FD-5491) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…with one judge model, and records only - Both kinds of run with the built-in user sim: an always_on step's stored timeline, or a prompt_agent step's user_sim_reads, each read whole from its object_url. run_step_id names the step; unset, the run must have one. - The user sim's instructions are rebuilt from its user_sims record with user_sim_versions.setup, as deploy_user_sim_agent wrote them, and checked against the recorded prompt_sha256. - One judge_model, defaulting to rubrics_verifier's (the config's judge role, else claude-sonnet-4-6); the two-judge split is gone. user_sim_error is yes / no / unknown (a check that could make it one couldn't be judged). - Record only: mark_user_sim_error (off by default) sets the prompt response's error_type to user_sim_error, only when the instructions match the recorded hash. The step doesn't fail the task by default. - Tests: a fake judge, one always_on and one prompt_agent run, every check's verdict, task items, the hash check, errored runs skipped, marking off by default and on when opted in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, in always_on and prompt_agent (FD-5491) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (FD-5491) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deploy_user_sim_agent's opening_prompt also takes a list of text and file parts, in prompt_agent's parts format; a plain string works as before. A file part may carry a one-line metadata.description for the user sim, which can't open files. The parts are checked when the task loads. - always_on sends the parts on the first wake in order: the joined text as the user's labelled line, each file in its place, without the description; the graded conversation's first user turn has the files. - The user sim's prompt and first read say what it attached: "(you attached itinerary.pdf: Mom and Dad's flight itinerary)", or the file name alone with no description. Text-only openings are byte-identical. - prompt_agent with the built-in user sim compares the files too: the same files in the same order after seeding, and its error names both lists. - Files go through the same sharing and seed filling as any prompt file part (apply_seed, lifted out of PromptAgentTaskStep._apply_seed, which now calls it). - The record keeps "opening" as the text, so text readers are unchanged, and adds "opening_parts" (as written) for an opening given as parts. The training harness's first prompt is the opening's text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atisfaction; always_on and prompt_agent no longer rate (FD-5491) The satisfaction lines move from the persona to the new step, so the user sim's system prompt never has them and the run behaves as tested. The step runs after the always_on or prompt_agent step it rates and asks the user sim once, in the session the run used, with what the user never read; a malformed answer gets one retry. The rating is recorded at metadata["user_sim_satisfaction"][run step id] and sent nowhere. always_on and prompt_agent are as they were before the first build, apart from the opening-files work, and always_on's record now names its user sim session (user_sim.context_id), which the step asks in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt, its load checks, and no S lines in the prompt (FD-5491) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tep's (FD-5491) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tty/fd-5491-satisfaction-step
…k (G4) - guidance: an unanswered question fails only when the assistant needs it answered; giving something again when asked is now a fail line. Self-contradiction moves to made_up. - oversharing: something the assistant should know from its notes, credentials, and answering a may-I-share question against PRIVATE lines or rules. - delegation: a broken "ask first" or "never" item must be objected to (the user sim's prompt always says so); a "tell you after" item left out of the final report must be asked about once. - rogue: letting the assistant off for a broken limit or skipped step. - style: same-length messages, described traits and caricature, missing pushback or reactions, and acting out of step with its situation (F4). - ending (G4): a new check, run by default and able to mark a run. The step tells the judge when the run ended at the user sim's max_replies, which its instructions never mention. - docstring: A5 is the separate rate_user_satisfaction step's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tty/fd-5490-user-sim-judge-v2 # Conflicts: # src/agent_env/task/episode/rules.py
parse_answer kept a "fail" whose cited turns were all outside the graded ones (turns=[] but could_change_grade still true), an unsupported failure that could mark the run. A check's or task item's fail must now cite at least one graded turn; otherwise the answer is not usable, so the judge is asked once more and a second such answer gives the check (or item) the verdict "error". Pass and n/a answers are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ser sim's file descriptions - rate_user_satisfaction: a failed_steps entry marked retried is a failure a retry recovered from, so it no longer skips the rating. - prompt_agent with the built-in user sim: file parts copied from the opening_prompt are sent and stored without metadata.description, as always_on sends them; the user sim still gets the descriptions from its record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no failure is asked again
The answer format put the verdict first, so the judge committed to "fail" and then argued itself out of it ("No clear
failure found"); in 2 of 4 real runs that alone made the run a user sim error. The prompt and schema now ask for the
reason, then turns, then verdict and could_change_grade (items alike), and say to decide after reasoning, within the
40-word limit. As a backstop, a fail whose reason says no failure was found is not usable, so it is asked again once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o failure; name the error-note limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An item has id, family, kind (hard / general / auto), check (text still loads as its old name), name, short, na_when and, for auto, metric and max. Hard items are told to the judge as exact rules, general ones fail only on a clear instance. Auto items are counted on the graded answers with no model call. na_when always is n/a with its static_na_reason and no call; never asks an n/a again. A call takes at most MAX_ITEMS_PER_CALL (12) items; the rest go in item-only calls. rubric_task_version is compared with the run's task instance version: on a mismatch the items aren't graded and the entry says so, the general checks still run. Bad items are refused with a clear error per item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e; the opening wake names its files; runs record each tag's line
- An opening with files adds one line under the "(you attached …)" note in W2: the user sim remembers only what's
said of each file, not its contents, and says so or points the assistant to the file. Text-only openings are
byte-identical (the golden prompt tests are untouched).
- always_on's timeline wake event records the files it carries, by name and uri, never their descriptions.
- deploy_user_sim_agent records tagged_lines ({"W1": ..., "D9": ...}) in metadata["user_sims"][name], so an answer's
`used` resolves against the lines that run's user sim saw after a persona is reordered; rate_user_satisfaction
records its S-tag map with the rating.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d (T) and things the assistant should already know (C) From the hand-written user sim template's standing_knowledge and assumed_context. Asked again for something already told, the user sim reminds the assistant, a little annoyed, and gives it again; it never brings up what the assistant should already know, and corrects it briefly when it gets one wrong. Both are validated like the other sections; a persona without them renders as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/fd-5490-user-sim-judge-v2
…t's own path; auto items take only na_when always The judge rebuilt the prompt from the record's opening text alone, so an opening with files never matched prompt_sha256 (no "(you attached ...)" notes, no ATTACHED_KNOWN line). deploy_user_sim_agent's person() is now a module function, and recorded_person() builds it from the record (opening_parts when present); the judge calls that and user_sim_versions.setup, so the files and the new already_told / should_already_know sections rebuild as deployed. An auto item is counted, never judged, so auto_result ignored any na_when but always: a conditional na_when did nothing and na_when never still gave n/a with no messages. Any other value is now refused when the task loads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, not the task version Any edit to a hub task makes a new version, so rubric_task_version skipped a rubric after the next edit even when the user sim's person didn't change. user_sim_common.persona_fingerprint hashes the persona's description, permissions and keep_private and the opening_prompt as written (recipe in its docstring, pinned by a test, since psi-benchmark-tasks's converter computes it too); deploy_user_sim_agent records it as persona_sha256. The judge's rubric_persona_sha256 replaces rubric_task_version: a mismatch skips the task items and still runs the checks, a match grades them, a run with no fingerprint is unknown and grades them. The task-instance lookup is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rahchitty/fd-5490-user-sim-judge-v2
…gent judge's timeout is undecided Task.run fills the seed into every step before it runs, so deploy_user_sim_agent fingerprinted the seeded persona and opening while rubric_persona_sha256 is the unseeded one: a seeded task's items were never graded. The step now takes the fingerprint when it's built, as written, and substitute_seed_in_steps hands it to the seeded copy (seeded_from). The test runs it through Task.run with a seed. With a run-wide use_agent_judge, the judge call can raise TimeoutError, JudgeTimeoutError or AgentUnreachableError; those now leave the call's rows undecided like the other flaky errors, and setup errors still raise. judge_timeout_retries and judge_max_turns, new on main, are fixed settings too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rahchitty/fd-5490-user-sim-judge-v2
…ver raised
A result whose id isn't text ({"results": [{"id": []}]}) reached the
shared id check, which hashes each id and raised TypeError, failing the
task instead of sending the answer back for correction. Each id is now
checked first and a non-text one is a row problem. The same goes for a
reason that isn't text and turns that aren't a list. An infinite turn
number (OverflowError) is dropped like any other entry that isn't a turn,
and `results` that isn't a list is named as such.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-5490-user-sim-judge-v2
Collaborator
Author
|
Thanks, done in your three-PR split:
One deliberate difference: when the judge can't decide, it records "unknown" instead of failing the task, following your "only yes fails". The agent-judge path for long runs is a follow-up (FD-5937): today's runs are about 1k tokens per check-in, and the cached single prompt is cheaper. |
…rahchitty/rubrics-verifier-judge-hooks # Conflicts: # src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
…sked again, not read as a turn int() read `true` and `1.9` as turn 1, so a fail could gate on a turn the judge never cited. Any cited turn that isn't an integer is now a row problem for the correction prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s need a batchable format judge_trajectory_content and unjudged_rows now run after the no_screenshots skip, so a skipped run never calls them. A _criteria_batches override that splits the criteria into several calls is refused for a format graded in one call (rubric_evidence and its mistakes variant), whose reasoning, checks and mistakes a merge of the batches would drop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…:scaleapi/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…e bulk-verdict check #719 refuses custom batches for a format whose rows don't stand alone. The user sim format's rows do (no per-call reasoning, checks or mistakes), so it's marked batchable, and a short pass reason is normal here, so bulk verdicts aren't flagged. Two tests follow #712's and #719's latest rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its last "but" "No clear failure was found at turn 1, but at turn 2 she leaked the password" supports a fail; it was sent back because the no-failure phrase matched anywhere in the last sentence. The conclusion is now the last sentence after its last contrast word. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…onclusion "No clear failure was found, although turn 2 was awkward" concludes nothing failed; only what follows but / however / yet is read as the conclusion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… again "No clear failures were found except at turn 2, where she leaked the password" is a fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…api/agentenv-framework into sarahchitty/fd-5490-user-sim-judge-v2
…s set The flag is read only by reward_of, which already finds scoring steps by getattr (verifier_id), so it reads can_be_reward the same way with a default of True. A step that checks the run sets it on its own class; TaskStep, which every step type shares, stays unchanged. The flag's explanation moves to reward_of's docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
approved these changes
Nov 17, 2026
…rahchitty/fd-5490-user-sim-judge-v2 # Conflicts: # tst/unit/task_step/verifiers/test_rubrics_verifier_subclass_hooks.py
…rahchitty/fd-5490-user-sim-judge-v2 # Conflicts: # src/agent_env/task_step/task_steps/verifiers/judge_utils/judge_output_format.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges after #719, the rubrics_verifier judge hooks (
_prompt_llm_judge(system=, cached_prefix=)with usage and cost,_criteria_batches,unjudged_rows,judge_trajectory_content,can_be_reward) and #712 (theuser_sim_checksjudge output format). Both are merged into this branch; review the last commit for this PR's own change.A new verifier step,
user_sim_judge, grades the user sim of a run (the AI playing the user), not the assistant (FD-5490), for always_on runs andprompt_agentchats with the built-in user sim (deploy_user_sim_agent). When the user sim slips in a way that could change the assistant's grade, the task fails, so the run is done again rather than scored.Built on rubrics_verifier
UserSimJudgeis aRubricsVerifierTaskStepsubclass, as Edgar suggested, on thealways_on_agent_verifierpattern:use_trajectoryoff,output_format: user_sim_checks.judge_trajectory_content, so everything up to its end is the same in every call and is marked for the prompt cache. The run's first call goes alone so the rest read it from the cache. A system message says the judge grades the user sim.rubric_items, one call per check with the items its families own (_criteria_batches, at most 12 items a call). Auto items,na_when: alwaysitems and line items whose line can't be read come fromunjudged_rows, with no call.step_paramshandling, answer-retry loop and concurrency. rubrics_verifier does them now. A run'suser_overrides.judge_modelpicks the model, as for every judge.user_sim_judge_corekeeps the checks, families, item model,render_runand the counters (standard library only). It also builds one run's criteria (plan_run) and reads the rows (user_sim_error).rate_user_satisfaction's run-step finder and its built-in-user-sim check.Task items keep every rule they had:
linetags read from the run'stagged_lines,rubric_persona_sha256(a mismatch skips the items, and the checks still run),na_when,static_na_reason, auto items counted bymax_words/max_sentences, and cut or missing answers making an auto itemerror. The judge's fields as psi-benchmark-tasks writes them are unchanged:run_step_id,judge_model,rubric_items,rubric_persona_sha256.What a task sets
Only these (beyond every step's id, type, version, depends_on and retry_config):
run_step_id, required: the always_on or prompt_agent step whose user sim it judges. The judge never guesses the run.rubric_itemsandrubric_persona_sha256.judge_modelandjudge_timeout_seconds, both optional (rubrics_verifier's defaults).fail_task_when_not_passing, optional, true by default.verifier_id.Everything else is fixed, and a task that sets it is refused at load with a plain message:
checksanderror_checks: the judge always asks all 7 checks, and every check but style can fail a run. An item's owncan_errorworks as before.fail_task_on_error: false: the default is true, so nobody needs to write it.use_trajectory,output_format,use_agent_judge,judge_batch_size,criteria,prompt_id...), unless it is given its fixed value.Removed on purpose: the judge's earlier
checksanderror_checkslet a task skip a check or stop one from failing a run. They're gone so every task is judged the same way and there are fewer ways to misconfigure it; a check that doesn't apply comes back n/a, and a noisy check gets fixed for every task. If a real task needs per-task control, add them back.What it writes
Everything goes in one place,
verifications[verifier_id]. It hasformat: user_sim_checks, the rows and a score, as rubrics_verifier writes them.can_be_rewardis False, so the score is never the task's reward.could_change_grade,assistant_asked, weight and, for items,asked_with.user_sim_error: yes, no or unknown, with the failing rows andfails_task.The gate
fail_task_when_not_passingreplacesmark_user_sim_error. It is on by default. A task that still setsmark_user_sim_erroris refused at load, with a message pointing tofail_task_when_not_passing.VerifierNotPassingError, naming the failing rows, only whenuser_sim_erroris yes and the rebuilt instructions match the recordedprompt_sha256.can_erroris true, or by default when its family's check gates.assistant_askedis recorded on each row and never changes the gate. This is a benchmark: a grade-changing slip spoils the run whoever caused it, so the run is done again.A judge that can't reach its model fails loudly
A setup error raises at once: "the user sim judge can't reach its model: …". The setup errors are:
ConfigError(no key, missing config);AuthenticationError,PermissionDeniedError,NotFoundError(an unknown model),BadRequestError(and its kinds),UnprocessableEntityErrorandBudgetExceededError.The run's other calls aren't made. Every other failure is taken as flaky and handled as below:
Timeout,RateLimitError,ServiceUnavailableError,InternalServerError,BadGatewayError,APIConnectionError, or an answer still malformed after its retries.Unknown never fails
A broken judge answer is asked again with the format's correction, up to 3 more times. A failed model call is retried as rubrics_verifier does. If a check is still undecided, its rows are
erroranduser_sim_erroris unknown. The entry is flagged"user sim judge undecided", the step doesn't raise, and the assistant's score is untouched. If no check at all was decided, the flag instead says "the judge couldn't decide any check — check its setup".This is deliberately different from rubrics_verifier, which fails a task when its judge gives up. Here only a slip the judge actually found fails a run, so a flaky judge call never throws away a good run.
Tests
test_user_sim_judge.py(93 tests, fake model): every earlier test ported, plus:assistant_askedis recorded but doesn't gate;run_step_id,checks/error_checks,fail_task_on_error: false, a fixed setting given another value;tst/unit/task_step,task/episode,training,cliandopenenvpass. The 2test_mcp_cli_builderfailures are local only: the generated script's Python has noclick.never_supplysection, so on them a slip is recorded and doesn't fail the task.Not done: calibration against a person's labels.
🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new blocking issue was found.Summary
Adds
user_sim_judgeto check whether the simulated user spoiled a run.The supplied previous threads are unnumbered, so they receive no numbered ledger entries. Their fixes remain present. Review used the full PR range because the current and previous review commits have different merge bases with the target.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Read the recorded run] --> B[Rebuild and compare instructions] B --> C[Plan seven checks and task items] C --> D[Ask the judge and compute automatic items] D --> E{Grade-changing slip found?} E -->|No or unknown| F[Keep the assistant's score unchanged] E -->|Yes| G{Gate on and instructions match?} G -->|Yes| H[Fail the task before reward steps] G -->|No| I[Record the slip only]Reviews (27) · Last reviewed commit: "Merge branch 'main' of github.com:scalea..." · Reviewed by Greptile