Repository navigation
feat(context): context trace, the prompt size of every model call in a Claude Code trace - #68
Conversation
vchamarthi
left a comment
vchamarthi
left a comment
There was a problem hiding this comment.
Thanks, this is a nice, self-contained addition. Deduping calls by message id and keeping subagent windows apart are exactly the two things that make these numbers trustworthy, and the fixture in the tests exercises both. A couple of small things inline.
| name = str(block.get("name") or "") | ||
| names[str(block.get("id") or "")] = name | ||
| out.tools.setdefault(name, ToolStats()).calls += 1 | ||
| signature = (agent, name, json.dumps(block.get("input"), sort_keys=True, default=str)) |
There was a problem hiding this comment.
Is repeat meant to include retries after a failed call and re-reads after the file changed (e.g. Read, Edit, Read on the same path, or running the same test command twice)? Those all match on name and input today, so the doc's 're-reading what it already has' might overstate it. Could be worth either excluding calls whose earlier twin errored, or softening the wording in context-window.md.
| rows.append(row) | ||
|
|
||
| if args.json: | ||
| print(json.dumps(rows if len(rows) > 1 else rows[0], indent=2)) |
There was a problem hiding this comment.
Nit: with --json, one trace gives an object and several give a list, so anything scripting over this has to check the type. Always emitting a list (or always an object keyed by trace) would be a bit easier to consume, if no caller already relies on the single-object form.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #68 +/- ##
==========================================
+ Coverage 88.14% 88.59% +0.44%
==========================================
Files 198 201 +3
Lines 22616 22911 +295
Branches 3606 3663 +57
==========================================
+ Hits 19934 20297 +363
+ Misses 1999 1914 -85
- Partials 683 700 +17
🚀 New features to boost your workflow:
|
…a Claude Code trace Reads a stream-json trace into per-call prompt tokens (input + cache_creation + cache_read, one call per message id, subagent calls apart), compaction events, and tool-result size, failures and repeats per tool. Several traces print as one comparison table, so free / loaded / with-skill runs compare side by side. No model call. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Claude Code 2.1.296 writes post_tokens next to pre_tokens; a forced compaction measured 107,547 -> 99,842, so the drop is not assumed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
2ff17bb to
8f0984b
Compare
str.splitlines() also breaks on U+2028, U+2029 and U+0085, which JSON.stringify and json.dumps(ensure_ascii=False) keep raw inside strings. A record holding one was cut into invalid halves and dropped, together with its model call and tool result, with only a stderr warning and exit 0. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
A UTF-8 BOM dropped the init record (empty surface), a non-UTF-8 byte or a UTF-16 file ended in a UnicodeDecodeError traceback, and a usage value of Infinity raised an uncaught OverflowError. Decode with utf-8-sig and errors=replace, and treat OverflowError as an unexpected shape of one line. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
Ran Two defects, fixed in the commits pushed here (each test red before):
Open, your call: for a repeated |
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at a5db543. The design is right: one call per message id, subagents counted apart, and the prompt as the sum of the three input fields. The four commits fix real parsing hazards. The problems are in the edges of the two metrics the docs lean on, repeat and the repeated-id merge, and in the CLI's handling of one bad file among several.
What I ran, on a Linux review host at a5db543:
make ci-local: 19 passed, 0 failed, 4 skipped (docker, semgrep, deploy legs). 4,375 tests, changed-line coverage 98% (185 lines; uncovered:commands/context.py:60,trace.py:180,trace.py:210).- Reproductions for the code findings below, with no model call (the command reads a file).
- A mutation run over
trace.pyandcommands/context.py: 151 mutants, 67 survivetests/test_context_trace.py(about 8 of them equivalent). The gaps that matter are in the comment on the tests. - The docs example rebuilt from a 2-call trace (3+21000+387, 3+283+21387): the command prints
21,390 | 21,673 | 11% | 283as the page says.
Points I'd fix before merge: the repeat definition (trace.py:173), the repeated-id merge (trace.py:162) and the abort on one bad trace (commands/context.py:24). The rest can wait.
Nits that have no inline anchor:
- A trace name containing
|breaks the table row.context trace 'a|b.jsonl' t.jsonlprints| a|b.jsonl | 2 | 21,390 | ..., one cell more than the 10-cell header, so every value shifts right (commands/context.py:65). - The README tree lists
context/out of order. The list atdokimi/README.md:238-249is alphabetical, andcontext/sits betweendashboard/andeval/. It belongs aftercommands/.
Checked, no issues
- Dedup and subagents: blocks of one id count once with the later block winning; a subagent's calls stay out of
calls, and itslsis not a repeat of the main thread's. A subagent-only trace summarises with null prompt fields and no crash. - Odd shapes:
messageas a list or string,usageas a list,tools/mcp_serversas an int or dict, non-stringtype, badtool_use_idandcontent, a float orInfinitytoken value all end as a warning or are ignored; none raises out ofparse_trace. - Scale: a 61 MB trace with 12,000 records parses in 0.46 s at 191 MB peak RSS.
- Repo rules: the page is in
mkdocs.ymlnav anddocs/README.md; the command is registered incli.py; SPDX headers are on every new.pyfile (no.mdin the repo carries one); no schema is involved since the output is stdout only;test_cli_help.py,test_docs_links.pyandtest_docs_code_refs.pypass with the new page. A zero or negative--windowis an argparse error (exit 2).
| key = message.get("id") | ||
| if key and key in by_message: | ||
| by_message[key].update(call) # a later block of the same call: same prompt, fuller output |
There was a problem hiding this comment.
A later block of the same message id with an empty or partial usage overwrites the real prompt size with 0. update(call) replaces every key, and _call turns a missing field into 0.
m1with usage 3/100/5000, thenm1again withusage: {}: the call ends asprompt_tokens: 0, new_tokens: 0. Expected 5103.first,peakandfinalare then wrong with no warning.- Second block with only
{"output_tokens": 90}:prompt_tokens: 0, output_tokens: 90. - Second block with a different
parent_tool_use_id:agentis replaced too, and a main-thread call moves into the subagent count ('agent': 'toolu_x'). - Second block with no
usagekey at all is fine (5103), so the failure is specific to a present-but-sparse dict. - An all-zero usage block with a new id is appended as a call of its own, so
finalbecomes 0 andgrowthis measured against the wrong end: two calls at 5103 and 9103 plus one all-zero block reportcalls: 3, final_prompt_tokens: 0.
The docstring promises "same prompt, fuller output"; the code does not enforce it. Real streams repeat the cache fields on each block, so I did not see this on a genuine trace. This was raised in the earlier comment on this PR and is still open at this head.
Suggestion: merge per field with max() (prompt and new tokens) and keep the first agent; skip a call whose four usage fields are all zero.
| signature = (agent, name, json.dumps(block.get("input"), sort_keys=True, default=str)) | ||
| if signature in seen: | ||
| out.repeated_calls += 1 | ||
| seen.add(signature) |
There was a problem hiding this comment.
repeat counts any second call with the same name and input, so ordinary retry and verify loops are reported as an agent re-reading what it already has. The signature is (agent, name, input) and nothing resets it.
Bash {"command": "make test"}fails (is_error), the agent fixes the file and runs the same command again:repeated_tool_calls = 1.Read /a.py,Edit /a.py,Read /a.py:repeated_tool_calls = 1, although the file changed in between.
In a with-skill versus without-skill comparison this column will blame the agent for the very loop that makes a task pass. docs/context-window.md:37-39 describes it as "an agent re-reading what it already has". @vchamarthi asked this on the PR; the four commits do not change it, so it is still open.
Suggestion: do not count a call whose earlier twin returned is_error, and drop the recorded signatures after a call that can change state (Edit, Write, Bash). If that is out of scope, rename the column and the docs to "identical calls" so the number is not read as waste.
| try: | ||
| rec = json.loads(line) | ||
| except json.JSONDecodeError: | ||
| out.errors.append(f"line {n}: not JSON") | ||
| continue |
There was a problem hiding this comment.
json.loads can raise something other than JSONDecodeError, and that escapes as a traceback. Only JSONDecodeError is caught here, and the commit that added OverflowError handling targets the same class of problem.
- A line holding an integer of 5,000 digits raises
ValueError: Exceeds the limit (4300 digits) for integer string conversionout ofparse_trace, socontext tracedies with a traceback and no table. - A line of 100,000
[raisesRecursionError.
Both lines are valid-ish junk that a corrupt or truncated trace can contain. One such line should be a line N: not JSON warning like the others.
Suggestion: except (json.JSONDecodeError, ValueError, RecursionError):. JSONDecodeError is a ValueError, so (ValueError, RecursionError) is enough.
| except TraceError as exc: | ||
| print(f"✗ {exc}", file=sys.stderr) | ||
| sys.exit(1) |
There was a problem hiding this comment.
One unreadable trace among several discards every good result. sys.exit(1) fires inside the read loop, before the table or the JSON is printed.
context trace a.jsonl empty.jsonl b.jsonl prints only ✗ empty.jsonl: no assistant message and no init event; ... on stderr, exits 1, and writes nothing to stdout. The numbers for a.jsonl are lost, and its parse warnings were already printed. The page sells exactly this multi-file comparison, usually with four arms, so a run that crashed before its first call costs the whole table.
Suggestion: collect the error, keep reading, print the table for the good traces, and exit 1 at the end. If aborting is intended, say so in context-window.md.
| rows.append(row) | ||
|
|
||
| if args.json: | ||
| print(json.dumps(rows if len(rows) > 1 else rows[0], indent=2)) |
There was a problem hiding this comment.
Nit: --json prints an object for one trace and a list for several, and the page never says so. A script over a variable number of files has to check the type. @vchamarthi raised this and it is unchanged at this head. Always emitting a list is the simpler contract; nothing in the repo reads the single-object form yet.
| - **compact**: `compact_boundary` events. After one the prompt can drop; the | ||
| `--calls` series shows by how much, and `--json` gives each event's trigger, | ||
| `pre_tokens` and (on newer CLIs) `post_tokens`. |
There was a problem hiding this comment.
Nit: plain --json does not give each compaction's trigger, pre_tokens and post_tokens. Those events are emitted as compaction_events only with --calls (commands/context.py:30). On a trace with one compaction (pre_tokens 150000, post_tokens 9000), context trace c.jsonl --json prints "compactions": 1 and no compaction_events key.
Suggestion: say "--json --calls gives each event's ...", or emit compaction_events always.
| `surface` field of `--json` (tool, skill, agent and MCP server counts, and the | ||
| loader plugins) shows what start-up offered, so two runs that differ there are | ||
| not the same condition. |
There was a problem hiding this comment.
Nit: the page says surface holds "tool, skill, agent and MCP server counts, and the loader plugins". Only tools, skills and agents are counts; mcp_servers and plugins are sorted lists of names (trace.py:140-146). For an init with mcp_servers: [{"name": "ctx"}] the output is "tools": 1, "mcp_servers": ["ctx"], "skills": 2, "agents": 0, "plugins": [].
| assert s["repeated_tool_calls"] == 1 | ||
| assert s["tool_result_chars"] == 300 + 300 + len("File does not exist.") + 50 + 4 | ||
| assert s["largest_tool_results"][0] == {"tool": "Bash", "calls": 3, "chars": 650, "max_chars": 300} | ||
| assert trace.compactions == [{"trigger": "auto", "pre_tokens": 20815, "post_tokens": 9000, "after_call": 2}] |
There was a problem hiding this comment.
The new tests leave the merge, the repeat definition and the summary limits unpinned. I ran 151 mutants of trace.py and commands/context.py against this file; 67 pass it (about 8 are equivalent). The ones that matter:
- No test repeats a
message.idwith sparse usage, so the finding attrace.py:162is invisible, and so is the "same prompt, fuller output" contract the docstring states. repeatis only tested for theagentpart of its signature. Replacing the tool name in the signature with""(soReadandBashwith equal arguments count as repeats), or ignoring the input, passes the suite. So does any definition of retry or read-edit-read.RUNhas three tools, soheavy = ...[:3]incommands/context.py:38can become[:4]or no limit and every test still passes. The same holds for theif s.result_charsfilter and the[:5]insummary, and nothing asserts the order oflargest_tool_resultsbeyond the first entry.- Failed calls are never different from repeats in the fixture (both are 1): counting distinct failing tools instead of failed calls, or swapping the
failedandrepeatcolumns, passes. - The camelCase
compactMetadata/preTokensfallback and a barecompact_boundaryare never fed in; removing either fallback passes. - An init-only trace, an assistant-only trace and the table without
--windoware untested (commands/context.py:60,trace.py:180and:210are the uncovered lines).
Suggestion: add one test per item. For the first, m1 with _usage(3, 100, 5000) then m1 with {}; assert prompt_tokens == 5103. For the limit, four tools with results and assert the line names three.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
- Merge repeated message-id blocks per field with max() and keep the first block's thread, so a sparse later usage cannot zero the prompt; skip all-zero usage blocks instead of counting them as calls. - Count a repeat only against an earlier call that did not fail, and forget earlier calls after Bash/Edit/MultiEdit/NotebookEdit/Write, so retries and read-edit-read loops are not reported as waste. - Treat any ValueError or RecursionError from json.loads as a "not JSON" line warning instead of a traceback. - CLI: an unreadable trace among several no longer discards the rest; report it, print the others, exit 1 at the end. --json always prints a list. Escape `|` in trace names in the table. - Docs: repeat definition, --json --calls for compaction_events, what surface holds, the list output; README tree order. - Tests for each of the above, plus the summary limits/order, camelCase and bare compactions, init-only and assistant-only traces. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Summary
Until now the tracks scored what an agent produced, not what its context window held. This adds
python cli.py context trace. It reads a Claude Codestream-jsontrace and reports the prompt size at every model call. No model call.When given several traces, it prints one comparison table, so the layers of a context compare side by side: free workspace, guidance loaded, a run without the skill and a run with it. The page says how to keep the free run clean: no
CLAUDE.mdin any ancestor, andCLAUDE_CONFIG_DIRpointing at an empty directory.Evidence
Before:
python cli.py context trace run.jsonlfails withinvalid choice: 'context'. The only token data for a Claude Code run was the total from the finalresultevent, so neither the start-up load nor the peak was visible.After: the same task (fix two bugs in a small repo) run with Sonnet 5.5 in four arms, one rep each, read by the new command:
All four arms pass the task. The difference is in the context: the skill roughly halves history growth and tool-result volume, and 30 unrelated skills add about 2.7k tokens to every call. The parser was first checked against a live trace from Claude Code 2.1.296. A call written as several messages under one id counts once; on the probe trace,
first(21,390) matches that call's three usage fields summed.tests/test_context_trace.py(11 tests): one call per message id, a subagent kept apart, a compaction with its position, a failed and a repeated tool call, list-content results, bad lines, a file that is not a trace, the table, JSON output, exit code 1.make ci-local: lint, mypy, tests (4304 passed), changed-line coverage, docs build and schema-compat all pass. Docker-only rows were skipped.Merge Danger
Door: two-way. It adds a new command group and package; nothing existing reads them.
Blast Radius: none
No persisted format: the command prints to stdout, and no schema is added.
--windowis required for the fill fraction, because the trace does not state the model's window and the command does not guess it.Checklist
pytestpassesruff,mypy, and schema checks pass🤖 Generated with Claude Code