Skip to content

feat(task-steps): a prompt step keeps each role's usage, with the endpoint host it called - #830

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/fd5981-step-usage
Nov 20, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/fd5981-step-usage

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

The host read only tool_call_count from an agent's reply and dropped its tokens and cost. Now each prompt step keeps one usage entry per role that ran turns in it: the solver, and the user simulator when there is one.

  • Reading a turn's usage. protocol.read_usage reads the agent's whole Usage into TerminalResponse.usage, and it never raises.
  • One entry per role per step, summed over the step's turns (task_step/usage.py). It covers prompt_agent, always_on, and the user simulator's turns in both: the built-in user sim, the peer user sim, and always_on's reads. A StepUsage per step does the recording: start(role, model, agent), turn(role, reply), entries(). The entry's shape is declared as the TypedDict UsageEntry, for P10, P11 and E13 to read.
  • Recorded as each turn ends in metadata["step_usage"][step_id], so a step that times out or raises keeps its finished turns. That's where P11 harvests a rolled-back or failed step's usage. The same entries go on the step's PromptResponse.usage when it's appended.
  • The attempt epoch. The scheduler sets it for each step in a ContextVar (ATTEMPT_EPOCH). Each step runs in its own asyncio task, so concurrent steps don't share it. AGENTS.md's contract for embedders says a runner that calls step.execute() itself sets it too. The hub's worker does that (run_task_step bypasses Task.run), so its one-line change is FD-6393; until then, hub runs' entries carry attempt: null.
  • The endpoint host. DeployedAgent.model_endpoint_host is set at deploy. deploy_agent takes it from the URL the deploy gives the agent, through A2AAgent.model_base_url, which follows _with_model_credentials's rule. install_agent takes the configured endpoint. It's None for an agent that takes no credentials. config.model.endpoint_host keeps the host and port and never a URL's userinfo.
  • metadata.model_endpoints[role][step_id] = {model, endpoint_host} sits beside metadata.models, which keeps its shape and legacy keys, so EvalRun.models is unchanged.

The entry, for agreement with E13 (FD-5961)

Each counter is the sum of what the turns reported, or null if none did, never 0. The counter and cost names are Usage.to_dict's, so P10's pricing reads an entry as it reads a Usage.

Key Content
role solver or user_simulator
model the model the agent took (null when it chose its own)
endpoint_host the host, and port, of the endpoint it was given, as configured: before a sandbox's rewrite to reach it (localhost to host.docker.internal), so one endpoint has one name and P10 can match it to the configured one; never a key, header value or URL userinfo
attempt the attempt epoch the step ran under (null outside Task.run)
turns how many turns the role finished in the step
tool_call_count, input_tokens, output_tokens, total_tokens, cache_read_tokens, cache_write_tokens summed
input_includes_cache the turns' flag when they agree, else null
cost_usd the agent's reported cost, summed
partial the sum is a lower bound: some turns reported a token count (any of the five) and others none, or a turn said some of its own calls reported none (agents' unreported_calls, the proxy's unreported); decided 2026-11-19
unreported_calls only when some turn reported them: their sum
unparsed only when some turn's usage was malformed: the first five such turns' unreadable values, a long value cut to the first 200 characters of its JSON
unparsed_turns with unparsed: how many turns had unreadable values

Per-turn detail stays in the A2A conversations collection, which a retry doesn't undo. Y4's per-turn ATIF fields are per turn and go beside the per-turn trajectory URLs, so they don't collide with this per-step list.

Tests

  • read_usage: full usage, tool calls only, unknown keys, legacy cache keys, bad values, bad provider_details, a value that isn't a mapping, and None. Also checked: tool_call_count is read as before even when it's malformed.
  • Entries:
    • sums; flags that disagree; a proxy's unreported; the attempt epoch; a step with no turns;
    • in prompt_agent, per-role sums with a user sim, tool calls only, partial sums, a step that times out after two turns keeping them, and malformed usage set aside;
    • always_on's wakes and user-sim reads.
  • Attempt epoch: a rolled-back, retried span runs under the next epoch (tst/unit/task/attempt_epoch_test.py).
  • Endpoint: for every deploy case in test_model_key_scope.py that isn't refused, the recorded URL is the one the deploy sends. And a deploy through deploy_agent with https://alice:hunter2@… and a run key stores the host and neither secret anywhere in the stored context.
  • Test stubs: four needed the new fields.

E2E against the real proxy

Pydantic AI and pi, through a logging pass-through in front of Scale's LiteLLM proxy. The stored task instance was read back:

Run What was stored
One turn, GPT a solver entry: 4,470 input (2,048 cache reads), 22 output, 1 tool call, input_includes_cache: true, the pass-through's host, attempt 0, not partial
Two turns with a user sim (both GPT) a solver entry with 2 turns and a user-simulator entry with 1 turn, each with its own counts, model and host, in both PromptResponse.usage and metadata.step_usage
pi on Claude tokens null (not 0), partial: true, unreported_calls: 1: the proxy streams Claude without usage
Turn 2 stalled 45 s past a 25 s limit the step failed (tolerated). Its entry kept turn 1's counts and turn 2, partial: true with unreported_calls: 1 (the interrupted call), and the user sim's turn

In the last run the agent enforced its own 25 s limit and returned a failed turn. So the path where the host itself times out (send_and_wait raising) is covered by the unit test, not by this run.

Chaos

In process, on prompt_agent's real run with only the agents' replies faked (~/fd5974-work/p9-chaos). There were 300 seeds, each running two prompt steps at once on one shared context, each with a solver and a user sim, up to four turns, and a random attempt epoch. Every reply carried random usage: full, tool calls only, none, malformed values, or unreported calls. Some steps timed out at a random turn.

Across 600 steps there were 115 timeouts and 924 entries: 456 partial, 274 with unparsed values, 258 with unreported calls, 371 for the user sim. Every check passed:

  • each role's stored entry equals the entry worked out independently from what that role's replies sent;
  • PromptResponse.usage equals metadata.step_usage;
  • a timed-out step appends no response but keeps its finished turns;
  • nothing raises but the timeouts;
  • concurrent steps never touch each other's entries.

With partial broken on purpose, the same run reports 320 problems.

Docs

The FD-6263 items for P9 stand: where a run's usage lives and what each field means, null rather than 0, and the user simulator's own row. To add: metadata.step_usage and metadata.model_endpoints.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The latest changes appear safe to merge; no new blocking issues were found.

Summary

Prompt steps now keep each role’s reported usage, model endpoint host, and attempt number. The latest changes move that recording into the shared StepUsage helper.

  • Prompt steps save summed usage for each role that finished turns.
  • Agent replies keep the usage fields the host can read.
  • Usage entries name the model endpoint host the agent was given.
  • Usage entries identify the attempt that ran each step.

No new actionable issues found.

Reviews (7) · Last reviewed commit: "refactor(task-steps): one StepUsage per ..." · Reviewed by Greptile

…point host it called

The host read only tool_call_count from an agent's reply and dropped its tokens and cost. It now
reads the turn's whole Usage with read_usage, which never raises: a value Usage refuses is kept
aside, an unknown key goes to provider_details, and an agent from before the typed cache fields
has its provider_details cache counts read.

prompt_agent and always_on sum each role's turns into one entry, solver and user_simulator, with
the model, the endpoint host (no key or URL userinfo) and the attempt epoch the scheduler now
sets for each step. A count nobody reported is None, never 0, and partial marks a lower bound:
turns that reported nothing, or calls a turn says went unreported. Entries are written to
metadata.step_usage as each turn ends, so a step that times out keeps its finished turns, and to
the step's PromptResponse.usage. metadata.model_endpoints records each role's model and host
beside metadata.models, which keeps its shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 19, 2026 23:26
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 23:26
Comment thread src/agent_env/task_step/usage.py Resolved
earakely-scale and others added 3 commits November 19, 2026 15:35
…any token count sits beside one that reported none

A turn reporting only total_tokens didn't count as reporting tokens, so beside a turn that reported
nothing the entry read as complete. Any token count now counts. DeployedAgent.model_endpoint_host
says it's the endpoint's host as configured, before a sandbox's rewrite to reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad, and runners that execute steps themselves set the attempt

read_usage kept a malformed usage value verbatim, and an entry kept one such record per turn, so
a broken agent could bloat a run's stored context past a store's item limit. A long value is now
kept as the first 200 characters of its JSON, and an entry keeps the first five turns' records
with unparsed_turns counting them all.

AGENTS.md's contract for embedders says a runner that calls step.execute() itself, not through
Task.run(), sets ATTEMPT_EPOCH, or its usage entries carry no attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and the entry has a type

prompt_agent and always_on each started a RoleUsage per role, recorded its model endpoint and
rewrote metadata.step_usage after each turn at four call sites, choosing the role's spelling
each time, and _UserSide reached the step's usage through a callback. A StepUsage per step now
does it: start(role, model, agent), turn(role, reply), entries(). The entry is a UsageEntry
TypedDict, so P10, P11 and E13 read a declared shape. Nothing a run records changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit ae34b28 into main Nov 20, 2026
65 of 67 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd5981-step-usage branch November 20, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant