Skip to content

feat(agents): shipped agents report cache reads and writes as Usage fields, not provider_details - #809

Merged
earakely-scale merged 3 commits into
mainfrom
edgararakelyan/fd5979-agent-usage-cache
Nov 19, 2026
Merged

earakely-scale merged 3 commits into
mainfrom
edgararakelyan/fd5979-agent-usage-cache

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

The five shipped agents now report the prompt cache in Usage's own fields, added in #791, and their input_tokens counts the cache wherever the agent can tell:

Agent Before After
pi input_tokens left the cache out; reads and writes in provider_details the cache is added back, message by message, once the reported total confirms the input left it out; cache_read_tokens, cache_write_tokens and input_includes_cache are filled in
Hermes same as pi same as pi, per turn
Pydantic AI the run's totals; Claude's cache, sent in Anthropic's keys, was dropped summed call by call: Anthropic's keys are added to a prompt smaller than the cache, and the flag is left unknown when a call can't say
OpenHands the run's totals; Claude's prompt left the cache out summed call by call, like Pydantic AI; a Responses API call's input already counts the cache
browser-use its counts already included the cache; the cache and the cost sat in provider_details the same counts, now in the Usage fields, with cost_usd

pi and Hermes don't assume their runtime's counting rule; they check it against the total the runtime reports. If the total is input + output + cache, the cache is added back. If it is input + output alone, the input already holds the cache, so nothing is added. If it matches neither, the cache is still added back but the flag is left unknown. This matters most for pi: its install pins pi-coding-agent 1.1.0, but that package depends on pi-ai ^1.1.0, so each image build picks up the latest 1.x (1.5.2 today).

When a model call returns no usage, the agent reports the counts as unknown (None) and records the call in provider_details.unreported_calls; before, it reported zero tokens. This covers Claude's streamed answers through Scale's LiteLLM proxy, which come back without usage.

Moved keys (for the release note)

  • provider_details.cache_read_tokens and provider_details.cache_write_tokens move to Usage.cache_read_tokens and Usage.cache_write_tokens for all five agents.
  • browser-use's provider_details.cost_usd moves to Usage.cost_usd.
  • input_tokens now counts the cache for pi, Hermes, and for Claude on Pydantic AI and OpenHands. A reader that used to add the cache back itself must stop doing so.
  • provider_details.unreported_calls is new.

The agents' and the proxy's protocol floor rises to 0.1.597, the first release whose Usage has the cache fields.

Why

Measured against the real proxy, the five agents disagreed on what input_tokens meant. The proxy's Chat Completions answers for Claude also leave the cache out of the prompt and carry no usage when streamed. Each runtime counts differently, so each agent normalizes its own runtime's counts, as agreed on #791.

Tests

  • Unit tests for each agent's usage, including pi's and Hermes' total check (confirmed, input already holding the cache, and neither), built from the per-call usage the proxy actually returned: GPT's cached reads, Claude's Anthropic keys, an all-zero (unreported) call, and the ambiguous case where the prompt is at least as large as the cache.
  • Integration: the Pydantic AI and OpenHands image tests (run_bundle_*_test.py) each gain a test of three turns on one deployed agent: Claude's recorded shape with a tool call, GPT's, and an answer with no usage. tst/util/fake_openai.py gains a per-request usage, None for an answer without one.

E2E against the real proxy

Each agent's smoke task ran on claude-haiku-4-5 and gpt-4.1-mini through Scale's LiteLLM proxy, behind a logging pass-through. The agent images were built from this branch, and all 10 runs passed. The table compares each turn's reported usage with the gateway's own answers for that turn, read by Usage.from_model_response:

Agent Model main: input / read / write This branch: input / read / write / flag Gateway
Pydantic AI Claude 0 / – / – unknown, unreported_calls: 2 2 streamed answers, no usage
Pydantic AI GPT 4,472 / 2,048* / 0* 4,474 / 2,048 / 0 / true 4,474 / 2,048 / 0
OpenHands Claude 23 / 10,363* / 10,562* 20,939 / 10,363 / 10,553 / true 20,939 / 10,363 / 10,553
OpenHands GPT 15,681 / 7,680* / 0* 15,681 / 7,680 / 0 / true 15,681 / 7,680 / 0
Hermes Claude 0 / 0* / 0* unknown, unreported_calls: 2 2 streamed answers, no usage
Hermes GPT 10,750 / 10,496* / 0* 21,249 / 10,496 / 0 / true 21,249 / 10,496 / 0
pi Claude 0 / 0* / 0* unknown, unreported_calls: 2 2 streamed answers, no usage
pi GPT 1,224 / 1,024* / 0* 2,248 / 1,024 / 0 / true 2,248 / 1,024 / 0
browser-use Claude 5,752 / 0* / 0* 5,775 / 0 / 0 / true 5,775 / 0 / 0
browser-use GPT 5,581 / 3,456* / 0* 9,784 / 5,632 / 0 / true 9,784 / 5,632 / 0

After the total check was added, pi and Hermes were rerun on the real proxy from the new head. pi on GPT reported 2,248 with the flag confirmed, and Hermes on GPT 21,253, both matching the gateway. On Claude, both reported unknown counts with 2 unreported calls.

* in provider_details. The main numbers come from an earlier run, so prompts differ slightly between the two runs. The Hermes gateway figures leave out its auxiliary non-streamed call (280 and 421 tokens), which Hermes keeps out of the turn's api_calls.

Chaos

Ran on the VM against the shipped Pydantic AI and OpenHands images and the fake endpoint, with two seeds of 10 turns each on one conversation. Each call carried Claude's proxy shape, GPT's shape with or without a cached read, or no usage, at random. Single 500s were retried by the agent's client. Each seed ended with a turn stalled past its 5 s timeout, then one more turn. All 40 turns matched the expected usage. Each timed-out turn failed with unknown counts, and the turn after it carried only its own call.

Notes

  • _counted and _summed are copied in the Pydantic AI and OpenHands folders rather than shared. Each image builds only from its own folder, and the protocol has no public helper for this.
  • OpenHands on a GPT model it calls through Chat Completions reports input_includes_cache: null when the cache read is smaller than the prompt. LiteLLM gives OpenHands the same counts for both shapes, so the agent can't tell which one it got.
  • Hermes still reports cost_usd: 0.0 when its cost_status is unknown. I left that to the pricing work.

Docs

The agents page should say that each shipped agent's input_tokens counts the prompt cache, which fields hold reads and writes, and what unreported_calls means.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The changes appear safe to merge; no new blocking issues were found.

Summary

The five agents move cache counts into Usage fields and improve how they report missing counts. The follow-up changes make pi and Hermes check reported totals before adding cached tokens.

  • pi reports prompt-cache tokens as part of its input count.
  • Hermes reports cache-aware input counts for each turn.
  • Pydantic AI reports cache-aware usage from each model response.
  • OpenHands reports cache-aware usage for each model call.
  • browser-use puts cache counts and cost in Usage fields.

No new actionable issues found in the changes since the previous review.

Reviews (7) · Last reviewed commit: "docs(agents): the agents guide says how ..." · Reviewed by Greptile

…ields, not provider_details

Each shipped agent (Pydantic AI, OpenHands, pi, Hermes, browser-use) now fills Usage's
cache_read_tokens, cache_write_tokens and input_includes_cache, and its input_tokens counts the
cache where the agent can tell: pi and Hermes add back the cache their runtimes take out of the
prompt; Pydantic AI and OpenHands decide call by call, adding Anthropic's cache keys to a prompt
smaller than them and leaving the flag unknown when the counts can't say. A model call that
returned no usage, as Claude's streamed answers through a LiteLLM proxy, leaves the counts unknown
and is counted in provider_details.unreported_calls instead of reading as zero tokens.

provider_details.cache_read_tokens and cache_write_tokens move to the Usage fields for all five
agents, and browser-use's cost to Usage.cost_usd. The agents' and the proxy's protocol floor is
0.1.597, the first whose Usage has the cache fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 19, 2026 08:17
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 08:17
earakely-scale and others added 2 commits November 19, 2026 08:26
…d total

pi's and Hermes' input leaves the prompt cache out and their total counts it, so the adapters
add the cache back; that rested on each runtime's current counting, and pi's library floats
within 1.x at each image build. Each now checks it against the total the runtime reported: a
total of the input, output and cache confirms the rule, one of the input and output alone says
the input already holds the cache, and one that is neither keeps the rule with
input_includes_cache unknown. The agents guide says how a call without usage is reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit e27970c into main Nov 19, 2026
38 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd5979-agent-usage-cache branch November 19, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant