Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
08153f1
WIP(FD-5490): user_sim_judge, a verifier that grades the user sim of …
sarah-chitty-1 Nov 12, 2026
32b50f0
feat(user sim): an optional satisfaction section, and always_on asks …
sarah-chitty-1 Nov 13, 2026
dec766b
feat(prompt_agent): the built-in user sim rates the chat once it has …
sarah-chitty-1 Nov 13, 2026
68c7cd5
feat(FD-5490): user_sim_judge judges always_on and prompt_agent runs …
sarah-chitty-1 Nov 13, 2026
e526294
test(user sim): the satisfaction section and the rating after the run…
sarah-chitty-1 Nov 13, 2026
47ed2c2
docs(user sim): the satisfaction section and the rating after the run…
sarah-chitty-1 Nov 13, 2026
e9e8200
feat(user sim): the user's first message can attach files (FD-5494)
sarah-chitty-1 Nov 13, 2026
2abc53c
docs(user sim): a first message with files (FD-5494)
sarah-chitty-1 Nov 13, 2026
3edb450
feat(user sim): the rating after the run is its own step, rate_user_s…
sarah-chitty-1 Nov 13, 2026
58e95a5
test(user sim): rate_user_satisfaction after always_on and prompt_age…
sarah-chitty-1 Nov 13, 2026
d2c3ead
docs(user sim): the rating at the end is the rate_user_satisfaction s…
sarah-chitty-1 Nov 13, 2026
c062ba5
Merge commit '3e16e8cceeb133321de189151bdf6dfd2675f538' into sarahchi…
sarah-chitty-1 Nov 14, 2026
8146f00
feat(FD-5490): user_sim_judge wording from review, and an ending chec…
sarah-chitty-1 Nov 14, 2026
9f1ca6d
Merge commit '3e16e8cceeb133321de189151bdf6dfd2675f538' into sarahchi…
sarah-chitty-1 Nov 14, 2026
c39bcbc
user_sim_judge: a fail that cites no graded turn is asked again
sarah-chitty-1 Nov 14, 2026
cb22f19
fix(user_sim): rate a run a retry recovered; prompt_agent drops the u…
sarah-chitty-1 Nov 14, 2026
b310862
user_sim_judge: name the two error-note length limits
sarah-chitty-1 Nov 14, 2026
6c621b5
user_sim_judge: reason before verdict, and a fail whose reason finds …
sarah-chitty-1 Nov 14, 2026
d3688de
user_sim_judge: only a reason's last sentence decides that it found n…
sarah-chitty-1 Nov 14, 2026
3c14761
user_sim_judge: task rubric items take usersim.yaml's criterion shape
sarah-chitty-1 Nov 14, 2026
69cc36b
fix(user_sim): the user sim knows only what's said of an attached fil…
sarah-chitty-1 Nov 14, 2026
7928477
feat(user_sim): two optional description sections, things already tol…
sarah-chitty-1 Nov 14, 2026
ae2729c
Merge branch 'sarahchitty/fd-5491-satisfaction-step' into sarahchitty…
sarah-chitty-1 Nov 14, 2026
111a988
user_sim_judge: rebuild the user sim's prompt by deploy_user_sim_agen…
sarah-chitty-1 Nov 14, 2026
48591e4
user_sim_judge: a rubric is tied to the user sim's persona fingerprin…
sarah-chitty-1 Nov 14, 2026
3f44808
user_sim_judge: an auto item can't pass an answer the record kept onl…
sarah-chitty-1 Nov 14, 2026
97c2ff1
fix(user_sim): one section for what the user never gives (keep_privat…
sarah-chitty-1 Nov 14, 2026
4878943
Merge branch 'sarahchitty/fd-5491-satisfaction-step' into sarahchitty…
sarah-chitty-1 Nov 14, 2026
f3b520d
fix(user_sim): never_supply's look-up facts belong in what_to_look_up…
sarah-chitty-1 Nov 14, 2026
bae4986
Merge branch 'sarahchitty/fd-5491-satisfaction-step' into sarahchitty…
sarah-chitty-1 Nov 14, 2026
00439a6
user_sim_judge: guidance holds the user sim to its T and C lines; a t…
sarah-chitty-1 Nov 14, 2026
8c299bd
test(user_sim): Hannah's look-up line drops the sentence the fixed op…
sarah-chitty-1 Nov 14, 2026
bb40e87
Merge branch 'sarahchitty/fd-5491-satisfaction-step' into sarahchitty…
sarah-chitty-1 Nov 14, 2026
8190289
user_sim_judge: an auto item can't pass when the record dropped some …
sarah-chitty-1 Nov 14, 2026
57facda
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 16, 2026
45fed68
user_sim_judge: a reply still waiting when the run ended is shown as …
sarah-chitty-1 Nov 16, 2026
2e562f8
feat(verifiers): user_sim_checks judge output format, for grading the…
sarah-chitty-1 Nov 16, 2026
b89f67e
reward: a step can say its score isn't the task's reward
sarah-chitty-1 Nov 16, 2026
f9c7499
rubrics_verifier: hooks for a subclass judge
sarah-chitty-1 Nov 16, 2026
4db2603
feat(verifiers): user_sim_checks rows give assistant_asked, recorded …
sarah-chitty-1 Nov 16, 2026
e43fb83
Merge branch 'sarahchitty/rubrics-verifier-judge-hooks' into sarahchi…
sarah-chitty-1 Nov 16, 2026
73314aa
Merge branch 'sarahchitty/user-sim-checks-format' (#712) into sarahch…
sarah-chitty-1 Nov 16, 2026
c978600
user_sim_judge: rebuilt as a rubrics_verifier that fails the task on …
sarah-chitty-1 Nov 16, 2026
10d1531
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 16, 2026
70e7384
user_sim_judge: fewer settings, run_step_id required, and a judge tha…
sarah-chitty-1 Nov 16, 2026
15445c0
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 16, 2026
d57124b
user_sim_judge: a seeded task keeps its persona fingerprint, and an a…
sarah-chitty-1 Nov 16, 2026
fd0d1f8
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 16, 2026
3e07f44
fix(verifiers): a malformed user_sim_checks answer is asked again, ne…
sarah-chitty-1 Nov 16, 2026
43fb9cd
Merge branch 'sarahchitty/user-sim-checks-format' into sarahchitty/fd…
sarah-chitty-1 Nov 16, 2026
71d72fa
Merge remote-tracking branch 'origin/main' into sarahchitty/user-sim-…
sarah-chitty-1 Nov 16, 2026
263a341
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 16, 2026
01a773f
fix(verifiers): a user_sim_checks turn that isn't a whole number is a…
sarah-chitty-1 Nov 16, 2026
9eb4595
rubrics_verifier: the subclass hooks skip a skipped run; split batche…
sarah-chitty-1 Nov 16, 2026
3b811fc
Merge branch 'sarahchitty/rubrics-verifier-judge-hooks' of github.com…
sarah-chitty-1 Nov 16, 2026
5ff62d2
Merge branch 'sarahchitty/user-sim-checks-format' of github.com:scale…
sarah-chitty-1 Nov 16, 2026
ce765de
user_sim_checks: its rows can be asked over several calls, without th…
sarah-chitty-1 Nov 16, 2026
b0c6558
fix(verifiers): a user_sim_checks reason's conclusion is what follows…
sarah-chitty-1 Nov 16, 2026
5b2602d
Merge branch 'sarahchitty/user-sim-checks-format' of github.com:scale…
sarah-chitty-1 Nov 16, 2026
cc53d9b
fix(verifiers): 'although' doesn't start a user_sim_checks reason's c…
sarah-chitty-1 Nov 16, 2026
845f7ab
Merge branch 'sarahchitty/user-sim-checks-format' of github.com:scale…
sarah-chitty-1 Nov 16, 2026
f6c77f7
user_sim_judge: name the cost's rounding (COST_DECIMALS)
sarah-chitty-1 Nov 16, 2026
82a3f2d
fix(verifiers): 'except' starts a user_sim_checks reason's conclusion…
sarah-chitty-1 Nov 16, 2026
d281086
Merge branch 'sarahchitty/user-sim-checks-format' of github.com:scale…
sarah-chitty-1 Nov 16, 2026
5169c90
reward: TaskStep keeps no can_be_reward; reward_of reads it where it'…
earakely-scale Nov 17, 2026
bbdb2ab
Merge #719's head (5169c9009) into #682: TaskStep keeps no can_be_reward
earakely-scale Nov 17, 2026
3cd49bb
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 17, 2026
a851462
Merge branch 'main' of github.com:scaleapi/agentenv-framework into sa…
sarah-chitty-1 Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/agent_env/task/episode/rules.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ def _rubrics_verifier(step: RubricsVerifierTaskStep, outside: Outside) -> list[s
"browser_release_context", "browser_set_cookies", "browser_sign_in", "browser_set_local_storage", "build_mcp_cli",
"combine_universe_verdicts", "configure_scheduled_checks", "deploy_env", "deploy_sandbox", "end_browser_recording",
"end_computer_recording", "start_browser_recording", "start_computer_recording", "env_outcome_verifier",
"modify_env_tool_access", "reset_env", "review",
"modify_env_tool_access", "reset_env", "review", "user_sim_judge",
"run_container_unit_tests_verifier", "run_docker_container", "set_env_clock_rate", "snapshot_env", "snapshot_computer_env",
"sync_env_clock",
"validation_gate_aggregator", "verify_env_card", "verify_env_core_protocol", "verify_mcp_tool_schema",
Expand Down
5 changes: 4 additions & 1 deletion src/agent_env/task/task.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,10 @@ def _sub(value):
out: list["TaskStep"] = []
for step in steps:
data = _sub(step.to_dict())
out.append(attach_retry_config(type(step).from_dict(data), data))
seeded = attach_retry_config(type(step).from_dict(data), data)
if (keep := getattr(seeded, "seeded_from", None)) is not None: # a step that keeps something as written
keep(step)
out.append(seeded)
return out


Expand Down
2 changes: 2 additions & 0 deletions src/agent_env/task_step/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]:
from agent_env.task_step.task_steps.configure_scheduled_checks import ConfigureScheduledChecksTaskStep
from agent_env.task_step.task_steps.install_agent import InstallAgentTaskStep
from agent_env.task_step.task_steps.verifiers.rubrics_verifier import RubricsVerifierTaskStep
from agent_env.task_step.task_steps.verifiers.user_sim_judge import UserSimJudge
from agent_env.task_step.task_steps.verifiers.always_on_agent_verifier import AlwaysOnAgentVerifier
from agent_env.task_step.task_steps.verifiers.run_container_unit_tests_verifier import RunContainerUnitTestsVerifierTaskStep
from agent_env.task_step.task_steps.run_docker_container import RunDockerContainerTaskStep
Expand Down Expand Up @@ -148,6 +149,7 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]:
_get_type(ConfigureScheduledChecksTaskStep): ConfigureScheduledChecksTaskStep,
_get_type(InstallAgentTaskStep): InstallAgentTaskStep,
_get_type(RubricsVerifierTaskStep): RubricsVerifierTaskStep,
_get_type(UserSimJudge): UserSimJudge,
_get_type(AlwaysOnAgentVerifier): AlwaysOnAgentVerifier,
_get_type(RunContainerUnitTestsVerifierTaskStep): RunContainerUnitTestsVerifierTaskStep,
_get_type(RunDockerContainerTaskStep): RunDockerContainerTaskStep,
Expand Down
55 changes: 40 additions & 15 deletions src/agent_env/task_step/task_steps/agent/deploy_user_sim_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@
- ``prompt_set``: the system prompt and model are on the agent, so always_on sets neither; ``multi_day``: the done line
the prompt was written with, which always_on refuses to run against a run of the other kind.
- ``model``, ``model_source``, ``prompt_sha256``, ``user_version``, ``setup``: for the run's record.
- ``persona_sha256``: ``user_sim_common.persona_fingerprint`` of the persona and ``opening_prompt`` as written, which
user_sim_judge compares a rubric's ``rubric_persona_sha256`` with.
- ``tagged_lines``: each tag the system prompt shows with its line as written there (``{"W1": "...", "D9": "..."}``),
so an answer's ``used`` resolves against the lines this run's user sim saw, whatever later edits to the persona move.

Expand Down Expand Up @@ -102,7 +104,8 @@
from agent_env.entity.scopes import FieldScope
from agent_env.task_step.context import TaskStepContext
from agent_env.task_step.task_steps.agent import user_sim_versions
from agent_env.task_step.task_steps.agent.user_sim_common import attached, opening_parts, opening_problems, opening_text
from agent_env.task_step.task_steps.agent.user_sim_common import (attached, opening_parts, opening_problems, opening_text,
persona_fingerprint, recorded_opening)
from agent_env.task_step.task_steps.deploy_agent import DeployAgentTaskStep

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -188,6 +191,31 @@ def persona_problems(persona: Any) -> list[str]:
return out


def person(key: str, persona: dict, opening: list[dict], seed: dict, multi_day: Optional[bool]) -> dict:
"""The user as the built-in user sim's ``setup`` takes them: ``persona``'s fields, the ``opening`` parts with the
run's ``<seed>`` placeholders filled in (as always_on fills them), and the done line ``multi_day`` picks. The step
writes its prompt from this, and user_sim_judge rebuilds that prompt from the step's record (``recorded_person``)."""
from agent_env.task_step.task_steps.prompt_agent import apply_seed

name = persona["display_name"]
parts = apply_seed(opening, seed)
text, files = opening_text(parts), attached(parts)
return {"key": key, "name": name, "first": persona.get("first_name") or name.split()[0],
"description": persona.get("description"), "permissions": persona.get("permissions"),
"keep_private": persona.get("keep_private"),
"opening": {"text": text, **({"files": files} if files else {})} if text else None,
"done_ends_run": not multi_day}


def recorded_person(us: dict, seed: dict, multi_day: Optional[bool] = None) -> dict:
"""``person`` from the step's ``metadata["user_sims"]`` record, with its ``opening_parts`` when it has them, so
``setup`` gives the prompt the step wrote; ``multi_day`` is the record's, else the one given."""
persona = {**{k: us.get(k) for k in PERSONA_KEYS},
"display_name": us.get("display_name") or us.get("name") or "the user"}
return person(us.get("name"), persona, recorded_opening(us), seed,
us["multi_day"] if us.get("multi_day") is not None else multi_day)


class DeployUserSimAgentTaskStep(DeployAgentTaskStep):
"""Deploys the agent that plays the task's user, its persona set at deploy, and records the user under
``agent_name`` in ``metadata["user_sims"]``."""
Expand Down Expand Up @@ -224,9 +252,16 @@ def __init__(self, id: str, version: Optional[int], *, persona: Optional[dict] =
self.model = model
self.multi_day = multi_day
self.opening_prompt = opening_prompt
# Who the user is as the task writes it, the seed not filled in: what the judge's rubric_persona_sha256 is.
self.persona_sha256 = persona_fingerprint(persona, opening_prompt)

_OWN = ("persona", "model", "multi_day", "opening_prompt")

def seeded_from(self, written: "DeployUserSimAgentTaskStep") -> None:
"""This step is ``written`` with the run's seed filled in (``substitute_seed_in_steps``): keep the fingerprint
of the persona and opening as written."""
self.persona_sha256 = written.persona_sha256

def to_dict(self) -> dict:
base = super().to_dict()
base.pop("system_prompt", None)
Expand All @@ -250,20 +285,8 @@ def _resolve_agent_ref(self, context: TaskStepContext) -> tuple[str, Optional[in

def person(self, context: TaskStepContext) -> dict:
"""The user as the built-in user sim's ``setup`` takes them, as always_on hands them over."""
name = self.persona["display_name"]
parts = self._opening(context)
text, files = opening_text(parts), attached(parts)
return {"key": self.agent_name, "name": name, "first": self.persona.get("first_name") or name.split()[0],
"description": self.persona.get("description"), "permissions": self.persona.get("permissions"),
"keep_private": self.persona.get("keep_private"),
"opening": {"text": text, **({"files": files} if files else {})} if text else None,
"done_ends_run": not self.multi_day}

def _opening(self, context: TaskStepContext) -> list[dict]:
"""``opening_prompt`` as parts with the run's ``<seed>`` placeholders filled in, as always_on fills them."""
from agent_env.task_step.task_steps.prompt_agent import apply_seed

return apply_seed(opening_parts(self.opening_prompt), context.metadata.get("seed") or {})
return person(self.agent_name, self.persona, opening_parts(self.opening_prompt),
context.metadata.get("seed") or {}, self.multi_day)

def _setup(self, context: TaskStepContext) -> dict:
return user_sim_versions.setup(user_sim_versions.VERSION, self.person(context))
Expand Down Expand Up @@ -319,6 +342,8 @@ async def _set_user(self, context: TaskStepContext, agent, setup: dict) -> TaskS
**({"opening_parts": opening_parts(self.opening_prompt)} if isinstance(self.opening_prompt, list) else {}),
"user_version": user_sim_versions.VERSION, "model": model, "model_source": source,
"prompt_sha256": hashlib.sha256(prompt.encode()).hexdigest(), "prompt_set": True,
# Who the user is, as written: what user_sim_judge's rubric_persona_sha256 is compared with.
"persona_sha256": self.persona_sha256,
# Each tag the prompt shows with its line, so an answer's ``used`` ("D9") names what this run saw.
"tagged_lines": user_sim_versions.tagged_lines(self.person(context)),
"multi_day": self.multi_day,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -133,22 +133,7 @@ def from_dict(cls, data: dict) -> RateUserSatisfactionTaskStep:
# ---- the load check ---------------------------------------------------------------------------------------------
def _run_step(self, steps: list[TaskStep]) -> tuple[Optional[PromptAgentTaskStep], list[str]]:
"""The step this one rates, or why there's none."""
where = f"{self.type} '{self.id}'"
if self.run_step_id is not None:
run = next((s for s in steps if s.id == self.run_step_id), None)
if run is None:
return None, [f"{where}: run_step_id '{self.run_step_id}' names no step of the task"]
if not isinstance(run, PromptAgentTaskStep):
return None, [f"{where}: run_step_id '{self.run_step_id}' is a {run.type} step; name the always_on or "
"prompt_agent step to rate"]
return run, []
runs = [s for s in steps if isinstance(s, PromptAgentTaskStep)]
if not runs:
return None, [f"{where}: the task has no always_on or prompt_agent step to rate"]
if len(runs) > 1:
return None, [f"{where}: the task has several always_on and prompt_agent steps "
f"({', '.join(s.id for s in runs)}); set run_step_id to the one to rate"]
return runs[0], []
return run_step_of(self, steps)

def task_problems(self, steps: list[TaskStep]) -> list[str]:
run, found = self._run_step(steps)
Expand All @@ -159,8 +144,7 @@ def task_problems(self, steps: list[TaskStep]) -> list[str]:
found.append(f"{where} comes before step '{run.id}', the run it rates; put it after")
if self.depends_on is not None and run.id not in {d.task_step_id for d in self.depends_on}:
found.append(f"{where}: its depends_on must name '{run.id}', the run it rates, so it runs once that's over")
user = _user_of(run)
if user is None or not any(isinstance(s, DeployUserSimAgentTaskStep) and s.agent_name == user for s in steps):
if not plays_built_in_user_sim(run, steps):
found.append(f"{where}: the user of {run.type} step '{run.id}' isn't a user sim a deploy_user_sim_agent "
"step deploys; only that user sim can rate the run")
earlier = next((s for s in steps[:steps.index(self)] if isinstance(s, RateUserSatisfactionTaskStep)
Expand Down Expand Up @@ -294,6 +278,34 @@ async def _trajectory(self, agent: DeployedAgent, session: _Session, task_id: st
return None


# ---- the load check, shared with user_sim_judge ----------------------------------------------------------------------
def run_step_of(step: TaskStep, steps: list[TaskStep], does: str = "rate") -> tuple[Optional[PromptAgentTaskStep], list[str]]:
"""The always_on or prompt_agent step ``step`` works on (its ``run_step_id``'s, else the task's only one), or why
there's none. ``does`` names what ``step`` does to the run, in the messages."""
where = f"{step.type} '{step.id}'"
if step.run_step_id is not None:
run = next((s for s in steps if s.id == step.run_step_id), None)
if run is None:
return None, [f"{where}: run_step_id '{step.run_step_id}' names no step of the task"]
if not isinstance(run, PromptAgentTaskStep):
return None, [f"{where}: run_step_id '{step.run_step_id}' is a {run.type} step; name the always_on or "
f"prompt_agent step to {does}"]
return run, []
runs = [s for s in steps if isinstance(s, PromptAgentTaskStep)]
if not runs:
return None, [f"{where}: the task has no always_on or prompt_agent step to {does}"]
if len(runs) > 1:
return None, [f"{where}: the task has several always_on and prompt_agent steps "
f"({', '.join(s.id for s in runs)}); set run_step_id to the one to {does}"]
return runs[0], []


def plays_built_in_user_sim(run: PromptAgentTaskStep, steps: list[TaskStep]) -> bool:
"""Whether ``run``'s user is a user sim a deploy_user_sim_agent step of the task deploys."""
user = _user_of(run)
return user is not None and any(isinstance(s, DeployUserSimAgentTaskStep) and s.agent_name == user for s in steps)


# ---- the run, as the run step recorded it ----------------------------------------------------------------------------
def _user_of(run: PromptAgentTaskStep) -> Optional[str]:
"""The agent a run step talks to as its user: always_on's ``user_sim``; prompt_agent's ``user_agent_name``, unless
Expand Down
18 changes: 18 additions & 0 deletions src/agent_env/task_step/task_steps/agent/user_sim_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
from __future__ import annotations

import copy
import hashlib
import json
from datetime import datetime, timezone
from typing import Any, Optional

Expand All @@ -20,6 +22,22 @@ def person_of(us: dict) -> dict:
"permissions": us.get("permissions"), "keep_private": us.get("keep_private")}


def persona_fingerprint(persona: dict, opening_prompt: Any) -> str:
"""Which user sim a run had: the hex sha256 of who the user is and what they first sent, as written (the seed not
filled in). deploy_user_sim_agent records it as ``persona_sha256``, and user_sim_judge grades a rubric written for
a fingerprint only on a run that has it. psi-benchmark-tasks's converter computes it too, so the recipe is fixed::

sha256(json.dumps({"description": persona.get("description"), "permissions": persona.get("permissions"),
"keep_private": persona.get("keep_private"), "opening_prompt": opening_prompt},
sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")).hexdigest()

``opening_prompt`` is the step's as written: its text, or its list of parts."""
body = json.dumps({"description": persona.get("description"), "permissions": persona.get("permissions"),
"keep_private": persona.get("keep_private"), "opening_prompt": opening_prompt},
sort_keys=True, separators=(",", ":"), ensure_ascii=False)
return hashlib.sha256(body.encode("utf-8")).hexdigest()


def task_zone(us: dict, clocks: list[dict], where: str) -> str:
"""The user's time zone, else the one sync_env_clock gave the world; refused when the two disagree. ``where`` names
the step in the error."""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1662,6 +1662,10 @@ class JudgeOutputFormatSpec:
parse_response=_parse_user_sim_checks_response,
diagnose_response=diagnose_user_sim_checks_response,
format_correction_prompt=format_user_sim_checks_correction,
# Each row stands alone (no per-call reasoning, checks or mistakes), so a judge may ask its rows over several
# calls; and a short pass reason is normal here, so bulk verdicts aren't flagged.
batchable=True,
checks_bulk_verdicts=False,
),
}

Expand Down
Loading
Loading