Skip to content

feat(prompt_agent): user-sim fixes: resume a cut-short turn as the user, keep the sim's own trajectory - #637

Open
kaihackney-scale wants to merge 9 commits into
mainfrom
kai/usersim-resume-prompt
Open

kaihackney-scale wants to merge 9 commits into
mainfrom
kai/usersim-resume-prompt

Conversation

@kaihackney-scale

@kaihackney-scale kaihackney-scale commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

prompt_agent: two user-sim fixes

This PR now carries two changes to how prompt_agent runs a user-sim. The second was #474, folded in so they're reviewed and released together.

  1. Resume a cut-short user-sim turn as the user, not as the agent (below).
  2. Keep the user-sim's own trajectory (section 2, at the end).

1. Resume a cut-short user-sim turn as the user

What

  • The Claude Code agent takes a resume_prompt (ClaudeCodeConfig). When a transient API error cuts a turn short, run_claude resumes the session with it, in place of the default _RESUME_PROMPT.
  • prompt_agent sends a user-sim USER_SIM_RESUME_PROMPT. It's negotiated against the agent's card like model and output_format, so an agent that doesn't take it is never sent it, and nothing changes for it.
  • So does deploy_user_sim_agent, for the built-in user sim it deploys (always_on, and prompt_agent with user_agent_name set to it), with its model and answer format. The prompt lives in user_sim_common, which both steps share.

Why

The default resume prompt asks the model to "continue the original request… and finish it." That's right for an agent doing a task. A user-sim's request is the agent's reply, though, so a resumed user-sim carries on as the agent. It reports work as done, says it lacks the tools to do the task, or answers the question it just asked. From then on, the conversation no longer tests the agent.

How it was tested

  • Unit tests:
    • a given resume_prompt replaces the default one on the resume;
    • the card's agent-config lists resume_prompt;
    • a user-sim whose card takes it is sent it, and one whose card doesn't is sent nothing, from prompt_agent and from deploy_user_sim_agent alike;
    • the solver is never sent it.
  • Replay at real user-sim messages that went out of role, with the real claude CLI and each task's own system prompt, the default reply schema, tools and files. Each turn was cut at its first step and resumed, and a separate grader labeled each reply:
    • with the default resume prompt, most resumed turns came back out of role, in nearly every context;
    • with the user-sim prompt, none did;
    • on contexts where the user-sim had stayed in role, none did either.
  • E2E through this branch's run_claude: the real CLI, with the first attempt cut and reported as a timed-out API call, so the bridge's own retry logic decides to resume and does. Every run resumed with the prompt it was given. With the default prompt, nearly all resumed turns went out of role; with the user-sim prompt, none did.
  • Checks: ruff check and the plugin-API check pass. The tst/unit/task_step and tst/unit/agents suites pass, except two test_mcp_cli_builder tests that fail locally because the generated script's interpreter has no click, which is unrelated to this change.

Rollout

An agent picks this up only once its image is built from a release with this change. Until then its card doesn't list resume_prompt, so it isn't sent.

2. Keep the user-sim's own trajectory (from #474)

Why

When a multi-turn conversation uses a user-sim, only the agent's trajectory is kept. The user-sim's own run isn't. So when a user-sim goes wrong, nobody can see what it did on that turn: what it read, what it reasoned, or which tools it called. Any fix to user-sim behavior has to be checked against exactly that.

What

When the user-sim's card serves urn:agentenv:trajectory/v1, each of its answers' trajectories is stored under <trajectory prefix>/usersim/. Each one is recorded in metadata.usersim_trajectories with:

  • the turn;
  • the message id the conversation sent it under;
  • where it was stored.

It's best effort:

  • a failed fetch is logged and the conversation goes on;
  • a fetch that stores nothing is logged and not recorded.

Human peers and user-sims without the extension are unchanged. The only cost is one trajectory fetch per user-sim answer, and only when its card offers one.

An earlier revision of this PR also added user_message_template (wrapping the agent's reply) and user_sim_guard (a length/markdown retry). Both are removed:

  • across many tasks, a length/markdown check flags ordinary user messages: detailed instructions, pasted emails, lists of files;
  • the framing's effect isn't established.

They'll come back, if at all, as part of a measured design.

Tests

  • tst/unit/task_step/test_prompt_agent_user_side.py covers three cases:
    • each answer's trajectory is stored under the step's prefix and recorded per turn, under the id its message was sent with;
    • a fetch that fails is tolerated;
    • a fetch that stores nothing is tolerated.
  • pytest tst/unit/task_step -k prompt_agent: 232 passed. ruff check . is clean.

End to end

I ran a clone of a real multi-turn prod task with a user-sim on this branch's step code, using agent-env task run --k 1. All 11 steps completed.

  • metadata.usersim_trajectories lists all three user-sim turns, each pointing under <trajectory prefix>/usersim/.
  • All three objects are in S3 (82.6, 35.7 and 32.2 KB) and parse as the user-sim's Claude Code stream.
  • That run also had the since-removed template and guard settings set; the screenshot shows them. The trajectory path doesn't depend on either.
1-run-and-records

The sandboxes were terminated after the run.

Folded in as one commit (6d7d002), the net diff of #474. The task-step and agents suites pass on the combined branch, apart from the same two test_mcp_cli_builder tests.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the previous delay finding is addressed.

Summary

This PR gives user simulators a role-specific resume prompt and saves their own trajectories.

  • User-sim retries can continue as the user after an interruption.
  • Peer user-sim answers can leave a saved trajectory for each turn.

Reviews (8) · Last reviewed commit: "fix(prompt_agent): bound the user-sim tr..." · Reviewed by Greptile

When a transient API error cuts a Claude Code turn short, the bridge resumes
the session with a prompt that asks the model to finish the request it was
given. A user-sim's request is the agent's reply, so a resumed user-sim
carried on as the agent.

The Claude Code agent now takes a resume_prompt, and prompt_agent sends a
user-sim one that keeps it the user, negotiated against the agent's card
like model and output_format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaihackney-scale
kaihackney-scale requested a review from a team as a code owner November 12, 2026 16:21
t and others added 3 commits November 16, 2026 11:55
deploy_user_sim_agent sends USER_SIM_RESUME_PROMPT when the user's card takes resume_prompt, as prompt_agent does for a peer user sim; a card that takes none gets none. The prompt moves to user_sim_common, which both steps share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-resume-prompt

# Conflicts:
#	src/agent_env/task_step/task_steps/agent/deploy_user_sim_agent.py
#	src/agent_env/task_step/task_steps/prompt_agent.py
#	tst/unit/task_step/test_deploy_user_sim_agent.py
t and others added 4 commits November 16, 2026 13:14
The user-sim resume prompt keeps its tested wording and adds one sentence: when what the user sim was last sent asks for something else, such as rate_user_satisfaction's rating in the same session, it gives that, in the format asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-resume-prompt

# Conflicts:
#	src/agent_env/agents/agents/claude-code/a2a_server.py
#	tst/unit/agents/claude_code/test_card_contract.py
Keep both import sets in deploy_user_sim_agent (the resume prompt here,
persona_fingerprint and recorded_opening from main).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the user-sim's card serves urn:agentenv:trajectory/v1, each of its
answers' trajectories is stored under <trajectory prefix>/usersim/ and
recorded in metadata.usersim_trajectories (turn, message id, where).
Best effort: a failed fetch is logged and the conversation goes on.

Folded in from #474 so the user-sim prompt_agent fixes ship together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaihackney-scale kaihackney-scale changed the title feat(prompt_agent): resume a cut-short user-sim turn as the user feat(prompt_agent): user-sim fixes: resume a cut-short turn as the user, keep the sim's own trajectory Nov 17, 2026
…meout

The next turn waits on it and the trace is optional, so a slow trajectory
endpoint now costs at most _USER_SIM_TRAJECTORY_FETCH_SECONDS, as
_user_sim_trajectory already does. Test: a fetch that hangs is cut off and the
conversation goes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant