Repository navigation
feat(task-steps): a prompt step keeps each role's usage, with the endpoint host it called - #830
Merged
Merged
Conversation
…point host it called The host read only tool_call_count from an agent's reply and dropped its tokens and cost. It now reads the turn's whole Usage with read_usage, which never raises: a value Usage refuses is kept aside, an unknown key goes to provider_details, and an agent from before the typed cache fields has its provider_details cache counts read. prompt_agent and always_on sum each role's turns into one entry, solver and user_simulator, with the model, the endpoint host (no key or URL userinfo) and the attempt epoch the scheduler now sets for each step. A count nobody reported is None, never 0, and partial marks a lower bound: turns that reported nothing, or calls a turn says went unreported. Entries are written to metadata.step_usage as each turn ends, so a step that times out keeps its finished turns, and to the step's PromptResponse.usage. metadata.model_endpoints records each role's model and host beside metadata.models, which keeps its shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
marked this pull request as ready for review
November 19, 2026 23:26
…any token count sits beside one that reported none A turn reporting only total_tokens didn't count as reporting tokens, so beside a turn that reported nothing the entry read as complete. Any token count now counts. DeployedAgent.model_endpoint_host says it's the endpoint's host as configured, before a sandbox's rewrite to reach it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad, and runners that execute steps themselves set the attempt read_usage kept a malformed usage value verbatim, and an entry kept one such record per turn, so a broken agent could bloat a run's stored context past a store's item limit. A long value is now kept as the first 200 characters of its JSON, and an entry keeps the first five turns' records with unparsed_turns counting them all. AGENTS.md's contract for embedders says a runner that calls step.execute() itself, not through Task.run(), sets ATTEMPT_EPOCH, or its usage entries carry no attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and the entry has a type prompt_agent and always_on each started a RoleUsage per role, recorded its model endpoint and rewrote metadata.step_usage after each turn at four call sites, choosing the role's spelling each time, and _UserSide reached the step's usage through a callback. A StepUsage per step now does it: start(role, model, agent), turn(role, reply), entries(). The entry is a UsageEntry TypedDict, so P10, P11 and E13 read a declared shape. Nothing a run records changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
The host read only
tool_call_countfrom an agent's reply and dropped its tokens and cost. Now each prompt step keeps one usage entry per role that ran turns in it: the solver, and the user simulator when there is one.protocol.read_usagereads the agent's wholeUsageintoTerminalResponse.usage, and it never raises.Usagerefuses is kept aside, under its name inTerminalResponse.usage_unparsed.Usagedoesn't have goes toprovider_details.provider_detailscache counts read into the typed fields.tool_call_countis read exactly as before, andextract_terminal_responsekeeps its tuple for the hub.task_step/usage.py). It coversprompt_agent,always_on, and the user simulator's turns in both: the built-in user sim, the peer user sim, andalways_on's reads. AStepUsageper step does the recording:start(role, model, agent),turn(role, reply),entries(). The entry's shape is declared as the TypedDictUsageEntry, for P10, P11 and E13 to read.metadata["step_usage"][step_id], so a step that times out or raises keeps its finished turns. That's where P11 harvests a rolled-back or failed step's usage. The same entries go on the step'sPromptResponse.usagewhen it's appended.ATTEMPT_EPOCH). Each step runs in its own asyncio task, so concurrent steps don't share it.AGENTS.md's contract for embedders says a runner that callsstep.execute()itself sets it too. The hub's worker does that (run_task_stepbypassesTask.run), so its one-line change is FD-6393; until then, hub runs' entries carryattempt: null.DeployedAgent.model_endpoint_hostis set at deploy.deploy_agenttakes it from the URL the deploy gives the agent, throughA2AAgent.model_base_url, which follows_with_model_credentials's rule.install_agenttakes the configured endpoint. It's None for an agent that takes no credentials.config.model.endpoint_hostkeeps the host and port and never a URL's userinfo.metadata.model_endpoints[role][step_id] = {model, endpoint_host}sits besidemetadata.models, which keeps its shape and legacy keys, soEvalRun.modelsis unchanged.The entry, for agreement with E13 (FD-5961)
Each counter is the sum of what the turns reported, or null if none did, never 0. The counter and cost names are
Usage.to_dict's, so P10's pricing reads an entry as it reads aUsage.rolesolveroruser_simulatormodelendpoint_hostlocalhosttohost.docker.internal), so one endpoint has one name and P10 can match it to the configured one; never a key, header value or URL userinfoattemptTask.run)turnstool_call_count,input_tokens,output_tokens,total_tokens,cache_read_tokens,cache_write_tokensinput_includes_cachecost_usdpartialunreported_calls, the proxy'sunreported); decided 2026-11-19unreported_callsunparsedunparsed_turnsunparsed: how many turns had unreadable valuesPer-turn detail stays in the A2A conversations collection, which a retry doesn't undo. Y4's per-turn ATIF fields are per turn and go beside the per-turn trajectory URLs, so they don't collide with this per-step list.
Tests
read_usage: full usage, tool calls only, unknown keys, legacy cache keys, bad values, badprovider_details, a value that isn't a mapping, andNone. Also checked:tool_call_countis read as before even when it's malformed.unreported; the attempt epoch; a step with no turns;prompt_agent, per-role sums with a user sim, tool calls only, partial sums, a step that times out after two turns keeping them, and malformed usage set aside;always_on's wakes and user-sim reads.tst/unit/task/attempt_epoch_test.py).test_model_key_scope.pythat isn't refused, the recorded URL is the one the deploy sends. And a deploy throughdeploy_agentwithhttps://alice:hunter2@…and a run key stores the host and neither secret anywhere in the stored context.E2E against the real proxy
Pydantic AI and pi, through a logging pass-through in front of Scale's LiteLLM proxy. The stored task instance was read back:
input_includes_cache: true, the pass-through's host, attempt 0, not partialPromptResponse.usageandmetadata.step_usagepartial: true,unreported_calls: 1: the proxy streams Claude without usagepartial: truewithunreported_calls: 1(the interrupted call), and the user sim's turnIn the last run the agent enforced its own 25 s limit and returned a failed turn. So the path where the host itself times out (
send_and_waitraising) is covered by the unit test, not by this run.Chaos
In process, on
prompt_agent's real run with only the agents' replies faked (~/fd5974-work/p9-chaos). There were 300 seeds, each running two prompt steps at once on one shared context, each with a solver and a user sim, up to four turns, and a random attempt epoch. Every reply carried random usage: full, tool calls only, none, malformed values, or unreported calls. Some steps timed out at a random turn.Across 600 steps there were 115 timeouts and 924 entries: 456 partial, 274 with unparsed values, 258 with unreported calls, 371 for the user sim. Every check passed:
PromptResponse.usageequalsmetadata.step_usage;With
partialbroken on purpose, the same run reports 320 problems.Docs
The FD-6263 items for P9 stand: where a run's usage lives and what each field means, null rather than 0, and the user simulator's own row. To add:
metadata.step_usageandmetadata.model_endpoints.🤖 Generated with Claude Code


Confidence Score: 5/5
The latest changes appear safe to merge; no new blocking issues were found.Summary
Prompt steps now keep each role’s reported usage, model endpoint host, and attempt number. The latest changes move that recording into the shared
StepUsagehelper.No new actionable issues found.
Reviews (7) · Last reviewed commit: "refactor(task-steps): one StepUsage per ..." · Reviewed by Greptile