Feature/more llm metrics in chatmessage#5909
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if chunk.usage is not None: | ||
| data.usage = chunk.usage.model_dump() |
There was a problem hiding this comment.
🔴 AttributeError when LLM node yields a str or FlushSentinel due to unconditional chunk.usage access
chunk.usage is accessed at line 149 before the type-dispatching isinstance checks on lines 153/157/191. The LLMNode type signature (AsyncIterable[llm.ChatChunk | str | FlushSentinel]) and the existing code both confirm that chunk can be a str or FlushSentinel—neither of which has a .usage attribute. When any LLM node yields a plain string token (which is a supported and documented code path per the comment on line 152), this line will raise AttributeError: 'str' object has no attribute 'usage', crashing the entire LLM inference task.
| if chunk.usage is not None: | |
| data.usage = chunk.usage.model_dump() | |
| if isinstance(chunk, ChatChunk) and chunk.usage is not None: | |
| data.usage = chunk.usage.model_dump() | |
Was this helpful? React with 👍 or 👎 to provide feedback.
| class TokenUsage(TypedDict, total=False): | ||
| completion_tokens: int | ||
| """The number of tokens in the completion.""" | ||
| prompt_tokens: int | ||
| """The number of tokens read from the cache.""" | ||
| total_tokens: int | ||
| """The number of input tokens used (includes cached tokens).""" | ||
| prompt_cached_tokens: int | ||
| """The number of cached input tokens used.""" | ||
| cache_creation_tokens: int | ||
| """The number of tokens used to create the cache.""" | ||
| cache_read_tokens: int | ||
| """The total number of tokens used (completion + prompt tokens).""" |
There was a problem hiding this comment.
🟡 Docstrings for prompt_tokens, total_tokens, and cache_read_tokens are swapped in TokenUsage
The docstrings in the new TokenUsage TypedDict are mismatched with their field names. Comparing with the source-of-truth CompletionUsage model at livekit-agents/livekit/agents/llm/llm.py:35-51:
prompt_tokenssays "The number of tokens read from the cache" — should be "The number of input tokens used (includes cached tokens)."total_tokenssays "The number of input tokens used (includes cached tokens)" — should be "The total number of tokens used (completion + prompt tokens)."cache_read_tokenssays "The total number of tokens used (completion + prompt tokens)" — should be "The number of tokens read from the cache."
It appears the fields were reordered relative to CompletionUsage but the docstrings were left in the old positions. Since TokenUsage is a public-facing TypedDict (part of MetricsReport), wrong docs will mislead consumers interpreting metrics data.
| class TokenUsage(TypedDict, total=False): | |
| completion_tokens: int | |
| """The number of tokens in the completion.""" | |
| prompt_tokens: int | |
| """The number of tokens read from the cache.""" | |
| total_tokens: int | |
| """The number of input tokens used (includes cached tokens).""" | |
| prompt_cached_tokens: int | |
| """The number of cached input tokens used.""" | |
| cache_creation_tokens: int | |
| """The number of tokens used to create the cache.""" | |
| cache_read_tokens: int | |
| """The total number of tokens used (completion + prompt tokens).""" | |
| class TokenUsage(TypedDict, total=False): | |
| completion_tokens: int | |
| """The number of tokens in the completion.""" | |
| prompt_tokens: int | |
| """The number of input tokens used (includes cached tokens).""" | |
| total_tokens: int | |
| """The total number of tokens used (completion + prompt tokens).""" | |
| prompt_cached_tokens: int | |
| """The number of cached input tokens used.""" | |
| cache_creation_tokens: int | |
| """The number of tokens used to create the cache.""" | |
| cache_read_tokens: int | |
| """The number of tokens read from the cache.""" |
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
No description provided.