Repository navigation
feat(verifiers): user_sim_checks judge output format, for grading the user sim - #712
Merged
Merged
Conversation
… user sim A new JudgeOutputFormat, USER_SIM_CHECKS, so the user sim judge (#682) can become a rubrics_verifier subclass. The judge grades the user sim, not the assistant: the framing and the run come first and the criteria last, so calls on one run can share a cached head. Each row gives reason, turns, verdict (pass / fail / n/a) and could_change_grade. Diagnosis ports the current judge's answer checks: a fail must cite a graded turn, a fail whose reason concludes nothing failed is asked again, and so is an n/a where na_when is never. A row lowers the score only on a fail that could change the grade; a weight 0 row is recorded and never gates. Additive: JudgeResponseDiscrepancy gains row_problems (empty by default), and rubrics_verifier's final error names them. Other formats are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…only Every row now gives assistant_asked: true when the assistant asked for or invited the behaviour the row grades (asked the user to write a message, or to give a fact it should look up), false otherwise. It is in the schema and the prompt, required on every row (a missing or non-bool one is asked again), and recorded only: it never changes a row's score or a gate. A test also shows that a task item follows the same rule as a check: a fail lowers the score only with could_change_grade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 16, 2026
…itty/fd-5490-user-sim-judge-v2
…ver raised
A result whose id isn't text ({"results": [{"id": []}]}) reached the
shared id check, which hashes each id and raised TypeError, failing the
task instead of sending the answer back for correction. Each id is now
checked first and a non-text one is a row problem. The same goes for a
reason that isn't text and turns that aren't a list. An infinite turn
number (OverflowError) is dropped like any other entry that isn't a turn,
and `results` that isn't a list is named as such.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sked again, not read as a turn int() read `true` and `1.9` as turn 1, so a fail could gate on a turn the judge never cited. Any cited turn that isn't an integer is now a row problem for the correction prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 16, 2026
…e bulk-verdict check #719 refuses custom batches for a format whose rows don't stand alone. The user sim format's rows do (no per-call reasoning, checks or mistakes), so it's marked batchable, and a short pass reason is normal here, so bulk verdicts aren't flagged. Two tests follow #712's and #719's latest rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its last "but" "No clear failure was found at turn 1, but at turn 2 she leaked the password" supports a fail; it was sent back because the no-failure phrase matched anywhere in the last sentence. The conclusion is now the last sentence after its last contrast word. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onclusion "No clear failure was found, although turn 2 was awkward" concludes nothing failed; only what follows but / however / yet is read as the conclusion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… again "No clear failures were found except at turn 2, where she leaked the password" is a fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second of three PRs that rebuild the user sim judge (#682) as a
rubrics_verifiersubclass, as Edgar's review of #682 suggested. This one adds the output format only. Nothing uses it yet, and the other formats don't change.What it adds
JudgeOutputFormat.USER_SIM_CHECKSinjudge_output_format.py:Templates. They frame the judge as grading the user sim (the AI playing the user), never the assistant. The framing and the run come first and the criteria last, so every call on one run shares the same head for a prompt cache. The run goes in the template's
$trajectory_content, as the user sim judge renders it. In the no-trajectory template, the run comes in the sections after the criteria.Schema. Each row gives
id,reason,turns,verdict(pass / fail / n/a),could_change_gradeandassistant_asked, in that order, so the judge reasons before it decides.assistant_asked. It's true when the assistant asked for or invited the behaviour the row grades (for example, asked the user to write a message, or to give a fact it should look up), and false otherwise. It's required on every row and only recorded: it never changes a row's score or a gate.Diagnosis. These are the checks the current judge's
parse_answermakes, ported with their tests. Each one sends the answer back with the correction prompt:could_change_gradeorassistant_askedthat isn't true or false;na_whenisnever.Turns outside the graded ones are kept apart, and a pass can't change the grade. The diagnosis reads the graded turns from each criterion's
graded_turns.Scoring. Checks and task items follow the same rule: a row lowers the score only on a fail with
could_change_grade. A row with weight 0 (style) is recorded with the judge's verdict but never lowers the score, under any aggregator.Shared code changes (additive)
JudgeResponseDiscrepancygainsrow_problems, which is empty by default and only this format sets.rubrics_verifier's final retry error names those problems, instead of "returned N results but expected N criteria".Tests
tst/unit/task_step/verifiers/judge_utils/test_user_sim_checks_format.pyhas 25 tests for the templates, schema, parser, diagnosis, correction prompt and scoring. Two of them run the format throughrubrics_verifier's retry loop. The verifier tests (536) andtst/unit/task_step(3220) pass, and so doesruff check.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge, with no new actionable issues found.Summary
Adds
USER_SIM_CHECKStemplates, schema, answer checks, correction prompts, and scoring.Reviews (7) · Last reviewed commit: "fix(verifiers): 'except' starts a user_s..." · Reviewed by Greptile