Repository navigation
fix(ai): give Claude Agent SDK generations their tool results and every output block - #1256
Conversation
…ry output block The Claude Code CLI sends each content block of a response as its own AssistantMessage, and each tool result as its own UserMessage after the generation's message_stop. The processor and PostHogClaudeSDKClient replaced the pending output on every AssistantMessage and promoted queued input at message_stop, so a generation kept only its last output block and the generation after a tool call had no $ai_input. Append output blocks per generation, append tool results to the queued input, and promote queued input when the next main-agent generation starts. A generation with nothing queued still has no input, as before. A request that is restarted before its message_stop drops its partial output. Without stream events, the result fallback reports every block of the final assistant turn. Messages a subagent sends (parent_tool_use_id set) stay out of the main generations' content.
marandaneto
left a comment
There was a problem hiding this comment.
The code looks good; no qualifying findings. Verdict: correct.
Thank you for your contribution!
SDK performance: posthog-pythonBenchmarks completed. No changes flagged for review. Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate. View all results
Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals. Each shard uses its own runner; compare Before PR against With PR within a row, not across shards. Compared 102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in |
💡 Motivation and Context
With the Claude Agent SDK integration (
PostHogClaudeAgentProcessor.queryandPostHogClaudeSDKClient), a run that uses tools produces$ai_generationevents with wrong content:$ai_input. The tool results never show up on any event.$ai_output_choiceskeeps only the last block of a response. When the model writes "Reading the file." and then calls a tool, the text is lost. With two tool calls, only the second one is kept.The cause is the order in which the Claude Code CLI sends messages. I checked it by running the bundled CLI (2.1.212, from
claude-agent-sdk0.2.121) against a local fake Messages API:AssistantMessage, and the processor replacedpending_outputon every one._instrument_queryexpected tool results beforemessage_stop, and queued input was promoted atmessage_stop. Tool results actually arrive aftermessage_stop, so they were promoted one generation late, and the last results were lost.UserMessage, and each one replaced the previous.The fix, applied in both the processor and the client:
UserMessages are merged into one user message.message_start), so it no longer matters which side ofmessage_stopthe results land on.$ai_input, as onmain. Input belongs to one generation only; it is not repeated on a follow-up generation that got no tool results.message_stop(a retry), the partial output of the unfinished attempt is dropped, and the retry keeps the attempt's input.mainit reported only the last block; it never carries output from earlier turns.parent_tool_use_idset) out of the main generations' input and output, and don't let a subagent'smessage_starttake the main generation's queued tool results. The CLI forwards those while the tool that started the subagent is running. Their tool spans are still emitted as before.PostHogClaudeSDKClient.query()now also clears any input and output left over from the previous turn.Why not group by
AssistantMessage.message_id? It doesn't exist inclaude-agent-sdk0.1.50, the minimum version inpyproject.toml, so the stream events are the only generation boundary available on every supported version.💚 How did you test it?
New tests in
posthog/test/ai/claude_agent_sdk/test_client.py. Each test runs against both the processor andPostHogClaudeSDKClient, using the message order recorded from the CLI:query().The first two fail on
main, and the subagent test fails if theparent_tool_use_idcheck is removed (it also fails onmain).The existing tests that use the old order still pass, including the multi-turn client test where tool results arrive before
message_stop.pytest posthog/test/ai: 1625 passed, 15 skipped.End to end with the real CLI and no API key: I ran the bundled
claudebinary throughPostHogClaudeAgentProcessor.query()withANTHROPIC_BASE_URLpointed at a small local server that streams a text block plus twoReadtool calls, then a final answer. Onmain, generation 2 had$ai_input = nulland generation 1's output was just the second tool call. With this change, generation 1's output hasReading.plus both tool calls, and generation 2's input has both tool results.Edge cases I considered:
UserMessagecontent still doesn't clear queued results;client.query()turn uses its own prompt;Checks:
ruff format --check,ruff check,mypy | mypy-baseline filter(no new violations), andmake public_api_check(up to date).📝 Checklist
If releasing new changes
sampo addto generate a changeset file (written by hand in the same format, assampoisn't installed here)🤖 Agent context
Autonomy: Human-driven (agent-assisted)
message_startrather than special-casing the new order, so both orders work.format_assistant_blocksre-exports as they are, so the API snapshot doesn't change.message_startonly when the previous request never finished, because an existing test sends theAssistantMessageaftermessage_stop.parent_tool_use_id.