Skip to content

feat: expose cache_creation_tokens in LLM metrics - #6655

Open
chasleslr wants to merge 2 commits into
livekit:mainfrom
chasleslr:feat/cache-creation-tokens
Open

feat: expose cache_creation_tokens in LLM metrics#6655
chasleslr wants to merge 2 commits into
livekit:mainfrom
chasleslr:feat/cache-creation-tokens

Conversation

@chasleslr

@chasleslr chasleslr commented Jul 31, 2026

Copy link
Copy Markdown

What

Surfaces prompt-cache creation (write) token counts through the metrics pipeline so downstream consumers can report on cache-creation usage.

  • Adds cache_creation_tokens to LLMMetrics and wires it from the existing CompletionUsage.cache_creation_tokens at the metrics construction site.
  • Adds input_cache_creation_tokens to the aggregate LLMModelUsage and accumulates it in ModelUsageCollector.collect().
  • Includes cache_creation_tokens in the log_metrics LLM log line.

Why

Providers already populate CompletionUsage.cache_creation_tokens (Anthropic from cache_creation_input_tokens; AWS Bedrock as of #6663), but the value was silently dropped when LLMMetrics was constructed — LLMMetrics had no such field. So cache reads were observable (prompt_cached_tokens) while cache writes were not, on either the per-request metrics_collected signal or the aggregate session_usage_updated / session.usage signal. This surfaces already-collected-but-discarded data.

Backward compatibility

All new fields default to 0. No existing construction site or consumer changes behavior. The deprecated UsageSummary / UsageCollector is intentionally left untouched.

Testing

  • tests/test_metrics_usage.py (unit): defaults-to-zero, value pass-through on LLMMetrics, and additive aggregation into LLMModelUsage via ModelUsageCollector.
  • make check passes (ruff format + lint + mypy strict).

Notes

Known limitations (follow-ups, intentionally out of scope)

  • telemetry/otel_metrics.py emits an OTel counter for prompt_cached_tokens but not yet for cache_creation_tokens.
  • For remote/cloud sessions, voice/remote_session.py (_session_usage_to_proto) cannot forward input_cache_creation_tokens until the external agent_pb protobuf gains a corresponding field.

🤖 Generated with Claude Code

@CLAassistant

CLAassistant commented Jul 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@chasleslr
chasleslr marked this pull request as ready for review July 31, 2026 17:57
@chasleslr
chasleslr requested a review from a team as a code owner July 31, 2026 17:57

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

chasleslr and others added 2 commits August 3, 2026 08:55
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chasleslr
chasleslr force-pushed the feat/cache-creation-tokens branch from 3e84098 to 25e7480 Compare August 3, 2026 12:56
@chasleslr
chasleslr marked this pull request as draft August 3, 2026 12:57
@chasleslr
chasleslr marked this pull request as ready for review August 3, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants