fix(telemetry): a counter says what was measured - #786
Merged
Conversation
Two tools declared a limit that was true when it was written and is not any
more. Both are now measured, and the captures that settle them ship as
fixtures.
Copilot. `copilot.ts` said the exclusivity of `input` against `cache_read`
was unconfirmed, because the only capture held `cache_read: 0` — where an
inclusive and an exclusive `input` produce the same number. It named what
would close it: one session with a non-zero `cache_read`. Captured on 1.0.82:
tokenDetails input 9 · cache_read 42038 · cache_write 21404 · output 408
usage.inputTokens 63451
9 + 42038 + 21404 = 63451, and Copilot's own terminal line for that run read
`↑ 63.5k (42.0k cached, 21.4k written)`. An `input` that already counted the
cached prompt would read 63451, not 9. The four counters are disjoint, so the
report is right to add them; it would otherwise have over-counted that
session by 42038 of 63451.
OpenCode. `opencode-export.ts` said no capture ever put a large `cache.read`
beside `input` for a non-Anthropic provider — the one comparison that shows
`input` failing to shrink if it already counted the cached tokens. Captured
from `opencode export --sanitize`, providerID "opencode", modelID
"ling-3.0-flash-fin-free", three billed turns of one session:
input 28242 → 269 → 196
read 640 → 28928 → 29184
total 29089 · 29356 · 29438, each == input + output + reasoning + cache
`input` falls as `cache.read` climbs, and OpenCode's own `total` holds the
identity on all three turns, which it could not if the counters overlapped.
What each tool still cannot say is narrower, and both limitation strings now
say that instead of the old one: for OpenCode, a provider reporting prompt
tokens inclusive of the cached ones has still never been captured here.
Both guards bite. Mutating `copilot-events.ts` to fold `cache_read` into
`input` turns exactly the new case red — `expected 42047 to be 9`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two tools declared a limit that was true when it was written and is not any more. Both are now measured, and the captures that settle them ship as fixtures.
Copilot —
inputagainstcache_readcopilot.tssaid the exclusivity was unconfirmed, because the only capture heldcache_read: 0, where an inclusive and an exclusiveinputproduce the same number. The comment named what would close it: "one capture of a session with a non-zerocache_read".Captured on
@github/copilot@1.0.82:tokenDetails.inputtokenDetails.cache_readtokenDetails.cache_writetokenDetails.outputmodelMetrics.<model>.usage.inputTokens9 + 42038 + 21404 = 63451, and Copilot's own terminal line for that run read↑ 63.5k (42.0k cached, 21.4k written).An
inputthat already counted the cached prompt would read 63451, not 9. The four counters are disjoint, so the report is right to add them — it would otherwise have over-counted that one session by 42038 of 63451.OpenCode — a non-Anthropic provider whose cache is exercised
opencode-export.tssaid no capture ever put a largecache.readbesideinputfor a non-Anthropic provider, "which is the one comparison that would showinputfailing to shrink if it already counted the cached tokens". The earlier probe of a second provider never exercised its cache.Captured from
opencode export --sanitize,opencode 1.14.20, providerIDopencode, modelIDling-3.0-flash-fin-free, three billed turns of one session:inputcache.readtotalinputfalls ascache.readclimbs, andtotal == input + output + reasoning + cache.read + cache.writeholds on all three turns — which it could not if the counters overlapped.What each tool still cannot say
Narrower than before, and both limitation strings now say that instead of the old sentence. For OpenCode: a provider reporting prompt tokens inclusive of the cached ones has still never been captured here.
Those strings are user-visible — they are the
reasonaby_toolrow carries — socli/tests/fixtures/cli-owns-read/expected-envelope.jsonmoves with them.The guards bite
Mutating
copilot-events.tsto foldcache_readintoinputturns exactly the new case red, and only it:Fixtures
Both are real captures, sanitized: session ids replaced, project id zeroed, directory and title already redacted by
--sanitize, the modified-file path replaced. Neither carries a machine path or a personal name.cli/tests/fixtures/local-cost/.copilot/session-state/55555555-…/events.jsonlcli/tests/fixtures/telemetry-sink/opencode-export-non-anthropic-cache.jsonGates
clivitest — 3504/3504node --test "scripts/__tests__/**/*.test.js"— 373/373biome check src tests— the only two warnings are pre-existing, in a file this branch does not touch (telemetry-evidence-adapter.integration.test.ts)🤖 Generated with Claude Code
https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp