Repository navigation
feat(prompt_agent): an opt-in gate on user-sim messages - #597
kaihackney-scale wants to merge 11 commits into
Conversation
prompt_agent's new user_sim_gate checks each LLM user-sim message before the agent receives it: phrase patterns for the assistant's side of a conversation, and optionally a small judge model that decides only whose voice the message is in. A message that fails is held back and the user-sim is asked to rewrite it, up to max_rewrites times; on_fail then fails the step, ends the conversation, or sends it anyway. Every verdict is recorded in context.metadata["user_sim_gate"]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e its cutoffs
`user_sim_gate={}` now enables the gate with its defaults; only None leaves it off. The patterns' length and
count cutoffs, and how much of a judge's reason and error a verdict keeps, are named class constants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review. Context from outside this diff, in case it's useful: there's a second open PR adding a step type that subclasses
So in either merge order, an Related, same direction: #567 addresses the same failure mode from the prompt side — Generated by Claude Code |
…te where it would be ignored The check-and-rewrite loop moves to user_sim_gate.run_gate, with a rewrite callback, so a step type that runs its own conversation can apply the gate; prompt_agent's user side now calls it. A PromptAgentTaskStep subclass that replaces prompt_agent's conversation refuses user_sim_gate at construction, instead of accepting and ignoring it, unless it sets USER_SIM_GATE_SUPPORTED = True. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main reworked prompt_agent's user side (the built-in user sim, end_after). The gate keeps its history wrapper in ask(), main's end_after short-circuit opens _answer(), and the gate still checks peer user-sim messages. A gate set with the built-in user sim is warned about, as with a non-user-sim peer, since those messages aren't checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p errors fall back to the patterns - AlwaysOnTaskStep.from_dict passes user_sim_gate on, so a saved step is refused as one built in code is, rather than running with the setting dropped. - A judge whose setup fails (its key or endpoint) no longer ends the conversation: like a failed judge call, the patterns' verdict stands and the error is recorded on each verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps checking The fallback rebuilt the gate without its judge, which a judge-only gate (patterns off) refuses, so its setup error ended the conversation. The gate's check now takes judge_unavailable instead: the judge isn't called, the patterns' verdict stands (a pass with patterns off), and the reason is recorded on the verdict, as for a judge call that fails. run_gate passes it only when set, so a check override with the old signature still works. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…agent and always_on prompt_agent applies user_sim_gate to the user deploy_user_sim_agent deployed: a held-back answer is rewritten in its session and parsed as its own answer, and the read's record keeps the first answer and lists each rewrite. always_on applies it at each read before an answer is held: one that still fails is never delivered and ends the run (user_sim_invalid; an error with on_fail fail). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ite counts toward max_replies A conversation the gate cut off with on_fail fail now records error_type user_sim_invalid on its PromptResponse, so verifiers skip it as they skip a user-sim error. A built-in user sim's rewrite replaces the held answer in the reply count, so a done rewritten into a reply counts, and max_replies ends the conversation after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-gate # Conflicts: # src/agent_env/task_step/task_steps/prompt_agent.py
…es its judge Through an endpoint, a Claude judge now drops the base's trailing /v1 (its client appends /v1/messages), and a bare name litellm can't place goes as openai/<name>. Before, both failed every call, and the gate fell back to its patterns alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
prompt_agentandalways_onget an opt-inuser_sim_gate. When the user side is an LLM user-sim, each of its messages ischecked before the agent receives it. That covers a peer user-sim and the built-in one
deploy_user_sim_agentdeploys:patterns(free): phrases and shapes that belong to the assistant's side, such as a report of workdone, a formatted deliverable, offers to do the work, or service phrases.
judge_model(optional): one model call over the last turns and the candidate. It decides only whosevoice the message is in. On its own it fails only messages of
judge_min_charsor more; shorter onesalso need a pattern.
A message that fails isn't sent. The user-sim is told why and asked to rewrite, up to
max_rewritestimes. If it still fails,
on_faildecides what happens:fail(the default) ends the conversation and fails the step, after thePromptResponseis recorded;endends the conversation only;sendrecords the verdicts and sends the message anyway.Every verdict is recorded in
context.metadata["user_sim_gate"][step_id], and an ended run is recorded incontext.metadata["user_sim_invalid"].The built-in user sim answers in JSON (
outcome,message,used), so its rewrite is parsed as its own answer:donewith text ends the conversation after the agent answers it, as before;doneends the conversation, as it would have.The read's record (
user_sim_reads) keeps what the user sim first answered, for anything grading the user sim, and lists each rewrite undergate_rewrites.In
always_oneach answer the user sim would send is checked at its read, before it's held for its response time:user_sim_invalid. Withon_fail: failit also setserror_type, so verifiers skip the run, as for a user-sim error.Her last read's answer, which never reaches the agent, isn't checked.
A step type that subclasses
PromptAgentTaskStepbut runs its own conversation would otherwise acceptuser_sim_gateand ignore it. Instead it refuses the setting at construction, unless it applies the gate and setsUSER_SIM_GATE_SUPPORTED = True, asalways_onnow does. For that, the check-and-rewrite loop isuser_sim_gate.run_gate(gate, text, history, rewrite):rewrite(note)returns the user-sim's new message, or None if the user side ended.prompt_agentuses the same function.{"user_sim_gate": {"judge_model": "<model>", "max_rewrites": 2, "on_fail": "fail"}}Why
An LLM playing the user can slip into the assistant's role. It answers its own question, reports work as
done, or offers to do the agent's job. After that, the rest of the run no longer tests the agent. Checking
each message, with a rewrite, keeps those messages out of the conversation, whether or not the user-sim
cooperates.
How it was tested
tst/unit/task_step/test_user_sim_gate.py: the patterns, judge parsing, the rendered history and therewrite note.
tst/unit/task_step/test_prompt_agent_user_side.py: 8 new tests over the real_UserSiderequest flow:failandend;run_gateon its own (rewrite, give up,send, stop);tst/unit/task_step/test_prompt_agent_built_in_user_sim.py: the built-in user sim held back and rewritten, failingafter its rewrites, and a rewrite that says nothing.
tst/unit/task_step/test_always_on_user_sim_gate.py: rewrite and deliver, fail (an error end),end, a pass withno rewrite, and no gate.
disagreements, checked against known cases):
with the next scripted request.
snapshot).
replay cover it.
always_on: two runs of a multi-day task, 4 wakes each, on the gate'spatterns plus a judge. One used the persona as written; the other's persona was told to answer like a status report.
only in the unit tests.
ruff checkis clean.No screenshots: the change has no UI.
Off by default: a step without
user_sim_gatebehaves exactly as before, andto_dictleaves the field out.🤖 Generated with Claude Code


Confidence Score: 5/5
The latest routing changes appear safe to merge.Summary
Adds an opt-in check that holds back assistant-like user-sim messages and asks for rewrites before delivery.
Reviews (10) · Last reviewed commit: "fix(user_sim_gate): route the judge call..." · Reviewed by Greptile