[napetrov]

Summary

repo run mixed scopes: tokens came from Claude Code's parent-only usage, cost from tree-wide total_cost_usd. Now tokens use the same rule the docker-claude parser already follows.

 events._apply(result)
   cost_usd = total_cost_usd                # parent + subagents + helper models
-  input/output_tokens = usage.*            # parent only
+  (scope, tokens) = claude_json.token_counts(rec)
+    modelUsage present -> sum over models, scope "tree"
+    else usage         -> scope "main"

result.json agent record also gains cache_read_tokens, cache_write_tokens, usage_scope, subagent_calls. An events.jsonl event made by a subagent carries agent (the parent's Agent/Task tool-use id). Contract §4.1 is updated.

Evidence

• Before: stream with one Agent call, usage = 7/2 and modelUsage main 7/2 + sub 30/9 → input_tokens=7, output_tokens=2 next to cost_usd=0.5 covering both.
  After: test_tokens_cover_the_subagents_the_cost_covers → (37, 11, cache 100, 5), usage_scope="tree", subagent_calls=1, child Read event has agent: "t1". test_tokens_without_model_usage_are_the_parents_only → "main".
• make ci-local: 18 pass. Bandit fails locally only because it scans a local .venv; with that excluded, bandit -r . -ll -x ./.venv is clean.

Merge Danger

Door: two-way

Blast Radius: reports

Token totals of new cells where the agent delegated go up compared with older cells in the same batch. The changelog says to re-run those cells before comparing. Only fields are added; none are removed.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.45%. Comparing base (6e695ce) to head (76afc3a).
⚠️ Report is 20 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #106      +/-   ##
==========================================
+ Coverage   90.24%   90.45%   +0.20%     
==========================================
  Files         206      206              
  Lines       24289    24769     +480     
  Branches     3906     4002      +96     
==========================================
+ Hits        21920    22405     +485     
+ Misses       1703     1698       -5     
  Partials      666      666              

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/claude_json.py	86.86% <100.00%> (+0.55%)	⬆️
dokimi/dokimi/repos/events.py	91.82% <100.00%> (+0.72%)	⬆️
dokimi/dokimi/repos/runner.py	92.98% <ø> (ø)

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

The changes reuse established accounting logic, include focused regression coverage, and have no identified blocking issues.

No open issues 🎉

What changed in this PR

Aligns repo run token accounting with tree-wide cost reporting when model usage is available, reusing the existing Claude parser logic.

Changes:

• Shares token aggregation and records usage scope and cache tokens.
• Counts subagent calls and attributes their events.
• Adds regression tests and documents accounting changes.

File	Description
dokimi/​tests/​test_repo_reach.py	Tests tree totals, parent-only fallback, and event attribution.
dokimi/​dokimi/​repos/​runner.py	Persists new usage and subagent fields.
dokimi/​dokimi/​repos/​events.py	Aggregates tokens and tracks subagent activity.
dokimi/​dokimi/​harnesses/​claude_json.py	Exposes shared, scope-aware token counting.
dokimi/​docs/​repo-evaluation-contract.md	Documents usage fields and event attribution.
dokimi/​changelog.d/​20261018_120000_napetrov_repo-run-tree-usage.md	Explains the correction and historical comparison caveat.

_{Review effort: Balanced}

[antonwolfy]

Review at fc932fb. The fix is right: tokens and cost now cover the same scope, and the reuse of token_counts keeps repo run in step with the docker-claude parser. Three things are worth fixing before merge, all about what the tests pin and what an empty usage reports.

What I ran, on a Linux review host at fc932fb:

• make ci-local --with-docker: 23 passed, 0 failed, 3 skipped (other-Python matrix legs, docs deploy, semgrep).
• Reproductions for the code findings, with the LLM transport mocked.
• Mutation runs against the new code: the mutants named in the inline comments pass the existing suites; dropping the parent_tool_use_id attribution, summing one model only and swapping the scope strings are all caught.

Checked, no issues

• Docs: the §4.1 text for agent, usage_scope and subagent_calls matches the code, including nested subagents (Task inside a subagent is counted, agent is the id of the direct parent call).
• Readers: repos/report.py and repos/compare.py read only cost_usd and num_turns from the agent record, so old result.json files without the new fields still load.
• Malformed input: a non-dict or empty modelUsage, and non-dict model entries, fall back to the usage numbers with scope main without raising.
• Rules: changelog fragment is under changelog.d/; result.json has no schema file, so no schema bump; ruff passes.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

• Raising TypeError here will make the claude-stream-json conversion fail hard when usage is… · New
• Importing token_counts inside _apply() adds repeated import lookups on every result record. If… · New

_{Review effort: Lite}

[napetrov]

Fixed in 9569eba: Malformed usage (non-dict) is now detected in _token_counts() before calling token_counts(), and the TypeError is properly caught and logged by the outer try/except. The conversion continues processing remaining records without hard failure.

[napetrov]

No change on this branch. A malformed usage is deliberately recorded as a per-line error ("line N: unexpected shape") and token extraction is skipped; the answer and cost of that record are kept. test_a_wrong_shaped_trace_line_is_an_error_not_a_crash pins this. My earlier reply citing 9569eba was wrong: that commit is on fix/repo-run-tree-usage-review, not on this PR branch.

[napetrov]

Fixed in 9569eba: Moved token_counts import to a module-level _token_counts() helper function. The import now happens once per call instead of on every result record, eliminating the lookup overhead for large transcripts.

[napetrov]

No change on this branch. The import runs once per run (one result record per stream), and a module-level import would make dokimi.repos.events load the whole dokimi.harnesses package (its init pulls in docker_solver and the runner). Same correction as above: 9569eba is not on this branch.

[antonwolfy]

Review at 3ea412a. Four of the five earlier threads are fixed: the empty usage case, the Task and main-branch cache pins, the lazy import and the long line. One test gap from my first review is still open, and I found a smaller one of the same kind.

What I ran, on a Linux review host at 3ea412a:

• make ci-local --with-docker: 18 passed, 0 failed, 6 skipped (network, job conditions, semgrep).
• Reproductions with the LLM transport mocked and the container runner faked.
• Mutation runs against tests/test_repo_reach.py: dropping or not usage, _AGENT_TOOLS = {"Agent"} or {"Task"}, zeroing the cache reads on either path, swapping the scope strings and dropping the agent attribution are all caught now. Two mutants survive, named below.

Still open from my earlier review:

• dokimi/dokimi/repos/runner.py:201-203: nothing tests that run_cell writes cache_read_tokens, cache_write_tokens, usage_scope and subagent_calls to result.json. Mutation check: removing those four kwargs still gives all of test_repo_run.py, test_repo_runner.py, test_repo_reach.py, test_repo_reach_edges.py, test_repo_manifest.py and test_repo_compare.py passing (337 passed). Mocking the container runtime is not needed: FakeRunner and fake_harness in tests/repo_eval_fixture.py already drive run_cell through repo run (test_repo_run.py:21-37). A FakeRunner subclass whose run writes a stream-json agent.stdout for the agent call (an Agent tool_use, a child Read with parent_tool_use_id, a result with modelUsage for two models) passes at the head, and fails with cache_read_tokens None once the kwargs are removed. It asserts result.json["agent"] has input_tokens/output_tokens equal to the model sums, cache_read_tokens, cache_write_tokens, usage_scope == "tree" and subagent_calls == 1.

Checked, no issues

• Empty usage: {"usage": {}} gives usage_scope None and all four counts None through convert. The docker-claude parser gives identical output to the base on 18 edge inputs (empty, non-dict, missing keys, bool, non-int, modelUsage as list).
• Import: import dokimi.repos.events alone no longer loads dokimi.harnesses. The load happens on the first result record.
• Malformed usage: "usage": "n/a" still yields one trace_errors entry and keeps cost, turns and answer.
• Docs and rules: the new §4.1 text matches the field names and the tree/main/null semantics. The changelog fragment name and format are right, and result.json has no schema, so there is no bump.

[xaleryb]

• Ran: the full suite on the head merged into current main (5293 passed, 21 skipped), plus ruff, mypy strict, bandit, the new tests against pre-fix source (5/5 red), mutations of the new guards, and a main-vs-head convert() differential over 68 real Claude Code stream-json traces and 13 malformed result records.
• No open must-fix or mid findings. Tokens match main on every real trace. Malformed usage gives one trace error and keeps answer/cost, with no crash. Every resolved thread is pinned by a test that goes red when its fix is reverted.
• No commits from this review.
• Left for the author (optional): with trace: text, subagent_calls is written as 0 rather than null, so a run where nothing was measured looks the same as a measured run with no delegation.

[xaleryb]

Approved at 8d019e5. Tokens match main on 68 real Claude Code traces, malformed usage is handled without a crash, every thread is pinned by a test, and CI is green.

[antonwolfy]

LGTM. The new tests close both gaps from my earlier review: result.json is now pinned end to end, and nested Task calls are counted in the test. One nit inline.

I agree with @xaleryb that trace: text writes subagent_calls: 0 where "not measured" is null for every other new field.

What I ran, on a Linux review host at 8d019e5:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped (other-Python matrix legs, docs deploy, semgrep).
• Mutation runs over test_repo_reach.py, test_repo_run.py and test_repo_runner.py: 31 mutants. Dropping any of the four run_cell kwargs, swapping cache read and write, counting only Agent or only Task, skipping nested calls, swapping the scope strings and summing one model only are all caught. The survivors are named below or are equivalent.

Checked, no issues

• Earlier threads: test_result_json_carries_the_token_fields_of_the_whole_tree drives run_cell through repo run and asserts (37, 11, 100, 5), usage_scope == "tree" and subagent_calls == 1. test_tokens_cover_the_subagents_the_cost_covers now asserts subagent_calls == 2 with a nested Task.
• Readers: repos/report.py and repos/compare.py read only cost_usd and num_turns, so old result.json files still load.
• Docs and rules: the new §4.1 text matches the field names and the tree/main/null semantics; the changelog fragment is in changelog.d/; result.json has no schema, so there is no bump.

[antonwolfy]

Nit: the failure-path defaults of the new keys are not pinned.

Mutation check: changing "usage_scope": None, "subagent_calls": None here to "usage_scope": "tree", "subagent_calls": 0 still passes test_repo_run.py, test_repo_reach.py and test_repo_runner.py (154 passed). A cell whose agent was killed before it wrote a trace would then report a measured zero delegation and a tree-wide scope, where the contract says null means no measurement.

Suggestion: in a killed-harness test, assert usage_scope and subagent_calls are None in result.json.