Skip to content

feat(verifiers): user_sim_checks judge output format, for grading the user sim - #712

Merged
sarah-chitty-1 merged 8 commits into
mainfrom
sarahchitty/user-sim-checks-format
Nov 17, 2026
Merged

sarah-chitty-1 merged 8 commits into
mainfrom
sarahchitty/user-sim-checks-format

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

The second of three PRs that rebuild the user sim judge (#682) as a rubrics_verifier subclass, as Edgar's review of #682 suggested. This one adds the output format only. Nothing uses it yet, and the other formats don't change.

What it adds

JudgeOutputFormat.USER_SIM_CHECKS in judge_output_format.py:

  • Templates. They frame the judge as grading the user sim (the AI playing the user), never the assistant. The framing and the run come first and the criteria last, so every call on one run shares the same head for a prompt cache. The run goes in the template's $trajectory_content, as the user sim judge renders it. In the no-trajectory template, the run comes in the sections after the criteria.

  • Schema. Each row gives id, reason, turns, verdict (pass / fail / n/a), could_change_grade and assistant_asked, in that order, so the judge reasons before it decides.

  • assistant_asked. It's true when the assistant asked for or invited the behaviour the row grades (for example, asked the user to write a message, or to give a fact it should look up), and false otherwise. It's required on every row and only recorded: it never changes a row's score or a gate.

  • Diagnosis. These are the checks the current judge's parse_answer makes, ported with their tests. Each one sends the answer back with the correction prompt:

    • a verdict outside pass / fail / n/a, or a could_change_grade or assistant_asked that isn't true or false;
    • a fail that cites no graded turn;
    • a fail whose reason's last sentence says nothing failed ("No clear failure found");
    • an n/a on a criterion whose na_when is never.

    Turns outside the graded ones are kept apart, and a pass can't change the grade. The diagnosis reads the graded turns from each criterion's graded_turns.

  • Scoring. Checks and task items follow the same rule: a row lowers the score only on a fail with could_change_grade. A row with weight 0 (style) is recorded with the judge's verdict but never lowers the score, under any aggregator.

Shared code changes (additive)

  • JudgeResponseDiscrepancy gains row_problems, which is empty by default and only this format sets.
  • rubrics_verifier's final retry error names those problems, instead of "returned N results but expected N criteria".

Tests

tst/unit/task_step/verifiers/judge_utils/test_user_sim_checks_format.py has 25 tests for the templates, schema, parser, diagnosis, correction prompt and scoring. Two of them run the format through rubrics_verifier's retry loop. The verifier tests (536) and tst/unit/task_step (3220) pass, and so does ruff check.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with no new actionable issues found.

Summary

Adds USER_SIM_CHECKS templates, schema, answer checks, correction prompts, and scoring.

  • The new user-sim check format grades the simulated user, not the assistant.

Reviews (7) · Last reviewed commit: "fix(verifiers): 'except' starts a user_s..." · Reviewed by Greptile

… user sim

A new JudgeOutputFormat, USER_SIM_CHECKS, so the user sim judge (#682) can
become a rubrics_verifier subclass. The judge grades the user sim, not the
assistant: the framing and the run come first and the criteria last, so
calls on one run can share a cached head. Each row gives reason, turns,
verdict (pass / fail / n/a) and could_change_grade. Diagnosis ports the
current judge's answer checks: a fail must cite a graded turn, a fail whose
reason concludes nothing failed is asked again, and so is an n/a where
na_when is never. A row lowers the score only on a fail that could change
the grade; a weight 0 row is recorded and never gates.

Additive: JudgeResponseDiscrepancy gains row_problems (empty by default),
and rubrics_verifier's final error names them. Other formats are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 16, 2026 18:29
…only

Every row now gives assistant_asked: true when the assistant asked for or
invited the behaviour the row grades (asked the user to write a message, or
to give a fact it should look up), false otherwise. It is in the schema and
the prompt, required on every row (a missing or non-bool one is asked
again), and recorded only: it never changes a row's score or a gate.

A test also shows that a task item follows the same rule as a check: a
fail lowers the score only with could_change_grade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 3 commits November 16, 2026 13:49
…ver raised

A result whose id isn't text ({"results": [{"id": []}]}) reached the
shared id check, which hashes each id and raised TypeError, failing the
task instead of sending the answer back for correction. Each id is now
checked first and a non-text one is a row problem. The same goes for a
reason that isn't text and turns that aren't a list. An infinite turn
number (OverflowError) is dropped like any other entry that isn't a turn,
and `results` that isn't a list is named as such.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sked again, not read as a turn

int() read `true` and `1.9` as turn 1, so a fail could gate on a turn the judge never cited. Any cited turn
that isn't an integer is now a row problem for the correction prompt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 16, 2026
…e bulk-verdict check

#719 refuses custom batches for a format whose rows don't stand alone. The user sim
format's rows do (no per-call reasoning, checks or mistakes), so it's marked batchable,
and a short pass reason is normal here, so bulk verdicts aren't flagged. Two tests
follow #712's and #719's latest rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its last "but"

"No clear failure was found at turn 1, but at turn 2 she leaked the password" supports
a fail; it was sent back because the no-failure phrase matched anywhere in the last
sentence. The conclusion is now the last sentence after its last contrast word.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onclusion

"No clear failure was found, although turn 2 was awkward" concludes nothing failed; only
what follows but / however / yet is read as the conclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… again

"No clear failures were found except at turn 2, where she leaked the password" is a fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 merged commit de64938 into main Nov 17, 2026
61 of 63 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/user-sim-checks-format branch November 17, 2026 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants