Skip to content

fix(telemetry): a counter says what was measured - #786

Merged
blafourcade merged 1 commit into
nextfrom
fix/a-counter-says-what-was-measured
Sep 6, 2026
Merged

fix(telemetry): a counter says what was measured#786
blafourcade merged 1 commit into
nextfrom
fix/a-counter-says-what-was-measured

Conversation

@blafourcade

Copy link
Copy Markdown
Contributor

What

Two tools declared a limit that was true when it was written and is not any more. Both are now measured, and the captures that settle them ship as fixtures.

Copilot — input against cache_read

copilot.ts said the exclusivity was unconfirmed, because the only capture held cache_read: 0, where an inclusive and an exclusive input produce the same number. The comment named what would close it: "one capture of a session with a non-zero cache_read".

Captured on @github/copilot@1.0.82:

field value
tokenDetails.input 9
tokenDetails.cache_read 42038
tokenDetails.cache_write 21404
tokenDetails.output 408
modelMetrics.<model>.usage.inputTokens 63451

9 + 42038 + 21404 = 63451, and Copilot's own terminal line for that run read ↑ 63.5k (42.0k cached, 21.4k written).

An input that already counted the cached prompt would read 63451, not 9. The four counters are disjoint, so the report is right to add them — it would otherwise have over-counted that one session by 42038 of 63451.

OpenCode — a non-Anthropic provider whose cache is exercised

opencode-export.ts said no capture ever put a large cache.read beside input for a non-Anthropic provider, "which is the one comparison that would show input failing to shrink if it already counted the cached tokens". The earlier probe of a second provider never exercised its cache.

Captured from opencode export --sanitize, opencode 1.14.20, providerID opencode, modelID ling-3.0-flash-fin-free, three billed turns of one session:

turn input cache.read total
1 28242 640 29089
2 269 28928 29356
3 196 29184 29438

input falls as cache.read climbs, and total == input + output + reasoning + cache.read + cache.write holds on all three turns — which it could not if the counters overlapped.

What each tool still cannot say

Narrower than before, and both limitation strings now say that instead of the old sentence. For OpenCode: a provider reporting prompt tokens inclusive of the cached ones has still never been captured here.

Those strings are user-visible — they are the reason a by_tool row carries — so cli/tests/fixtures/cli-owns-read/expected-envelope.json moves with them.

The guards bite

Mutating copilot-events.ts to fold cache_read into input turns exactly the new case red, and only it:

→ expected 42047 to be 9 // Object.is equality
Tests  1 failed | 10 passed (11)

Fixtures

Both are real captures, sanitized: session ids replaced, project id zeroed, directory and title already redacted by --sanitize, the modified-file path replaced. Neither carries a machine path or a personal name.

  • cli/tests/fixtures/local-cost/.copilot/session-state/55555555-…/events.jsonl
  • cli/tests/fixtures/telemetry-sink/opencode-export-non-anthropic-cache.json

Gates

  • cli vitest — 3504/3504
  • node --test "scripts/__tests__/**/*.test.js" — 373/373
  • biome check src tests — the only two warnings are pre-existing, in a file this branch does not touch (telemetry-evidence-adapter.integration.test.ts)

🤖 Generated with Claude Code

https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp

Two tools declared a limit that was true when it was written and is not any
more. Both are now measured, and the captures that settle them ship as
fixtures.

Copilot. `copilot.ts` said the exclusivity of `input` against `cache_read`
was unconfirmed, because the only capture held `cache_read: 0` — where an
inclusive and an exclusive `input` produce the same number. It named what
would close it: one session with a non-zero `cache_read`. Captured on 1.0.82:

    tokenDetails  input 9 · cache_read 42038 · cache_write 21404 · output 408
    usage.inputTokens                                                   63451

9 + 42038 + 21404 = 63451, and Copilot's own terminal line for that run read
`↑ 63.5k (42.0k cached, 21.4k written)`. An `input` that already counted the
cached prompt would read 63451, not 9. The four counters are disjoint, so the
report is right to add them; it would otherwise have over-counted that
session by 42038 of 63451.

OpenCode. `opencode-export.ts` said no capture ever put a large `cache.read`
beside `input` for a non-Anthropic provider — the one comparison that shows
`input` failing to shrink if it already counted the cached tokens. Captured
from `opencode export --sanitize`, providerID "opencode", modelID
"ling-3.0-flash-fin-free", three billed turns of one session:

    input  28242 → 269 → 196
    read     640 → 28928 → 29184
    total  29089 · 29356 · 29438, each == input + output + reasoning + cache

`input` falls as `cache.read` climbs, and OpenCode's own `total` holds the
identity on all three turns, which it could not if the counters overlapped.

What each tool still cannot say is narrower, and both limitation strings now
say that instead of the old one: for OpenCode, a provider reporting prompt
tokens inclusive of the cached ones has still never been captured here.

Both guards bite. Mutating `copilot-events.ts` to fold `cache_read` into
`input` turns exactly the new case red — `expected 42047 to be 9`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
@blafourcade
blafourcade requested a review from a team as a code owner September 6, 2026 05:47
@blafourcade
blafourcade merged commit 1f1ae9e into next Sep 6, 2026
17 checks passed
@blafourcade
blafourcade deleted the fix/a-counter-says-what-was-measured branch September 6, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant