Repository navigation
feat(prompt_agent): user-sim fixes: resume a cut-short turn as the user, keep the sim's own trajectory - #637
Open
kaihackney-scale wants to merge 9 commits into
Open
kaihackney-scale wants to merge 9 commits into
kaihackney-scale wants to merge 9 commits into
Conversation
When a transient API error cuts a Claude Code turn short, the bridge resumes the session with a prompt that asks the model to finish the request it was given. A user-sim's request is the agent's reply, so a resumed user-sim carried on as the agent. The Claude Code agent now takes a resume_prompt, and prompt_agent sends a user-sim one that keeps it the user, negotiated against the agent's card like model and output_format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deploy_user_sim_agent sends USER_SIM_RESUME_PROMPT when the user's card takes resume_prompt, as prompt_agent does for a peer user sim; a card that takes none gets none. The prompt moves to user_sim_common, which both steps share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-resume-prompt # Conflicts: # src/agent_env/task_step/task_steps/agent/deploy_user_sim_agent.py # src/agent_env/task_step/task_steps/prompt_agent.py # tst/unit/task_step/test_deploy_user_sim_agent.py
The user-sim resume prompt keeps its tested wording and adds one sentence: when what the user sim was last sent asks for something else, such as rate_user_satisfaction's rating in the same session, it gives that, in the format asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-resume-prompt # Conflicts: # src/agent_env/agents/agents/claude-code/a2a_server.py # tst/unit/agents/claude_code/test_card_contract.py
Keep both import sets in deploy_user_sim_agent (the resume prompt here, persona_fingerprint and recorded_opening from main). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the user-sim's card serves urn:agentenv:trajectory/v1, each of its answers' trajectories is stored under <trajectory prefix>/usersim/ and recorded in metadata.usersim_trajectories (turn, message id, where). Best effort: a failed fetch is logged and the conversation goes on. Folded in from #474 so the user-sim prompt_agent fixes ship together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…meout The next turn waits on it and the trace is optional, so a slow trajectory endpoint now costs at most _USER_SIM_TRAJECTORY_FETCH_SECONDS, as _user_sim_trajectory already does. Test: a fetch that hangs is cut off and the conversation goes on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
prompt_agent: two user-sim fixes
This PR now carries two changes to how
prompt_agentruns a user-sim. The second was #474, folded in so they're reviewed and released together.1. Resume a cut-short user-sim turn as the user
What
resume_prompt(ClaudeCodeConfig). When a transient API error cuts a turn short,run_clauderesumes the session with it, in place of the default_RESUME_PROMPT.prompt_agentsends a user-simUSER_SIM_RESUME_PROMPT. It's negotiated against the agent's card likemodelandoutput_format, so an agent that doesn't take it is never sent it, and nothing changes for it.deploy_user_sim_agent, for the built-in user sim it deploys (always_on, andprompt_agentwithuser_agent_nameset to it), with its model and answer format. The prompt lives inuser_sim_common, which both steps share.Why
The default resume prompt asks the model to "continue the original request… and finish it." That's right for an agent doing a task. A user-sim's request is the agent's reply, though, so a resumed user-sim carries on as the agent. It reports work as done, says it lacks the tools to do the task, or answers the question it just asked. From then on, the conversation no longer tests the agent.
How it was tested
resume_promptreplaces the default one on the resume;resume_prompt;prompt_agentand fromdeploy_user_sim_agentalike;claudeCLI and each task's own system prompt, the default reply schema, tools and files. Each turn was cut at its first step and resumed, and a separate grader labeled each reply:run_claude: the real CLI, with the first attempt cut and reported as a timed-out API call, so the bridge's own retry logic decides to resume and does. Every run resumed with the prompt it was given. With the default prompt, nearly all resumed turns went out of role; with the user-sim prompt, none did.ruff checkand the plugin-API check pass. Thetst/unit/task_stepandtst/unit/agentssuites pass, except twotest_mcp_cli_buildertests that fail locally because the generated script's interpreter has noclick, which is unrelated to this change.Rollout
An agent picks this up only once its image is built from a release with this change. Until then its card doesn't list
resume_prompt, so it isn't sent.2. Keep the user-sim's own trajectory (from #474)
Why
When a multi-turn conversation uses a user-sim, only the agent's trajectory is kept. The user-sim's own run isn't. So when a user-sim goes wrong, nobody can see what it did on that turn: what it read, what it reasoned, or which tools it called. Any fix to user-sim behavior has to be checked against exactly that.
What
When the user-sim's card serves
urn:agentenv:trajectory/v1, each of its answers' trajectories is stored under<trajectory prefix>/usersim/. Each one is recorded inmetadata.usersim_trajectorieswith:It's best effort:
Human peers and user-sims without the extension are unchanged. The only cost is one trajectory fetch per user-sim answer, and only when its card offers one.
An earlier revision of this PR also added
user_message_template(wrapping the agent's reply) anduser_sim_guard(a length/markdown retry). Both are removed:They'll come back, if at all, as part of a measured design.
Tests
tst/unit/task_step/test_prompt_agent_user_side.pycovers three cases:pytest tst/unit/task_step -k prompt_agent: 232 passed.ruff check .is clean.End to end
I ran a clone of a real multi-turn prod task with a user-sim on this branch's step code, using
agent-env task run --k 1. All 11 steps completed.metadata.usersim_trajectorieslists all three user-sim turns, each pointing under<trajectory prefix>/usersim/.The sandboxes were terminated after the run.
Folded in as one commit (6d7d002), the net diff of #474. The task-step and agents suites pass on the combined branch, apart from the same two
test_mcp_cli_buildertests.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; the previous delay finding is addressed.Summary
This PR gives user simulators a role-specific resume prompt and saves their own trajectories.
Reviews (8) · Last reviewed commit: "fix(prompt_agent): bound the user-sim tr..." · Reviewed by Greptile