[napetrov]

Part of #41 (step D4).

Summary

 run_cell
   convert(trace) -> events.jsonl
+  run_process_metrics(events, guidance.globs, manifest.process_metrics)
   run_entry("grade") -> grade
-  result.json {agent, grade}
+  result.json {agent, process, grade}

# result.json
process: {tool_calls: 10, commands: 4, failed_commands: 1, denied_commands: 1,
          guidance_read: [AGENTS.md, src/AGENTS.md], counters: {bazel_cmds: 2, icpx_missing: 1}}
# repo-eval.yaml
process_metrics:
  bazel_cmds: '\bbazel(?:isk)?\b'                             # command text (default)
  icpx_missing: {regex: "icpx is not found", match: output}   # text | output | any

• New dokimi/repos/process_metrics.py; counts all events (parent and subagents). failed_commands = rc not 0/null; denied commands still count for counters (the agent asked for them), as in oneDAL's metrics.py.
• repo validate refuses a regex that does not compile, a bad match, a bad name; process_metrics needs a runnable manifest.
• entry_points.grade no longer schema-required (widening, v1 in place); check_repo requires it only when some task's grader does not start with dokimi:, and names those tasks. Generic, so it holds whether or not feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay #120 lands first.
• Task host: blocks were already validated by repo validate (feat(repo): task host requirements, unsupported instead of a false pass (#41 D3) #112); added a test that pins it.
• Contract: §2 manifest rules, §4.1 result.json process table. Nothing in §3/§5/§8.

Evidence

• Before: result.json has no process; oneDAL recomputes counts in its own metrics.py; a manifest without entry_points.grade fails the schema.
• After: tests/test_repo_process_metrics.py (each metric, counter fields, 6 regex/shape refusals, grade-optional rule, host validation) and test_repo_run.py asserting result["process"] on a real cell. make ci-local --offline: 0 failed, 15 passed, 11 skipped (network/docker). git merge-tree against repo scan: L0 rules for parity with oneDAL static_check (D6) #115 and feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay #120 heads: clean.

Merge Danger

Door: two-way

Additive key in result.json; schema widened only.

Blast Radius: small

Readers of result.json that reject unknown keys would see process; none in-tree.

Checklist

• DCO sign-off
• scriv fragment
• contract doc updated
• pytest, ruff, mypy pass

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.07692% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.98%. Comparing base (1abd834) to head (0696bb6).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/events.py	87.50%	1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #157      +/-   ##
==========================================
+ Coverage   92.43%   92.98%   +0.55%     
==========================================
  Files         240      243       +3     
  Lines       30415    30835     +420     
  Branches     5029     5081      +52     
==========================================
+ Hits        28114    28673     +559     
+ Misses       1615     1460     -155     
- Partials      686      702      +16     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	93.03% <100.00%> (+0.26%)	⬆️
dokimi/dokimi/repos/manifest.py	94.25% <100.00%> (+0.35%)	⬆️
dokimi/dokimi/repos/process_metrics.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/runner.py	95.25% <100.00%> (+0.19%)	⬆️
dokimi/dokimi/repos/events.py	92.52% <87.50%> (+0.17%)	⬆️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 343f7ac. The process-metrics code is clean and the counting rules match the contract. Two things are worth fixing before merge: the optional grade entry point has no runtime behind it at this head, and the one runner test cannot tell a working guidance_read / counters wiring from a broken one.

What I ran, on a Linux review host at 343f7ac:

• make ci-local: 25 passed, 1 failed, 6 skipped. 6,731 tests, changed-line coverage passes. The failure is pip-audit on virtualenv 21.14.7 (CVE-2026-113722) from the existing locks, fixed on main by #152.
• A reproduction of the first finding through repo run with the container runner faked.
• A mutation run: 48 mutants of process_metrics.py, manifest.py and runner.py against the three test files that cover them; 18 survive, 17 of them real gaps (the second finding).

Checked, no issues

• Schema: dropping required: ["grade"] only widens repo_manifest.v1; process_metrics is validated by both the schema and parse_counter.
• Counting rules: failed_commands ignores rc null, denied commands still reach the counters, bad regex / bad match / bad name are refused with the metric named.
• Offline: the new tests need no network or Docker.

[xaleryb]

• Ran: the full suite, ruff, mypy and bandit on the head and on the head merged with current main, plus fake-runner repros of each point.
• Open: an agent that exits before writing any trace line gets process with every count at zero, the same as a 3-turn run with no tool call; on the merged tree the same cell has uptake.measured: false. Expected: process: null under main's transcript.events or transcript.num_turns is not None condition. This is in the hunk that conflicts with main.
• Pushed:
  • 24a292f: repo run / repo tasks check refuse a repository grader with no entry_points.grade before the batch is pinned (before: every cell ran, was paid for, and graded grader_failure).
  • a296a67: a non-string process_metrics key (404:, on:) is refused (before: validate passed it, then the result.json write raised TypeError).
  • 732cb6b: a permission-denied command no longer counts in failed_commands.
  • 56fe503: guidance_read ignores absolute, ../ and denied reads.
• Left for the author: resolve the conflict with main, and the process null case.

[antonwolfy]

Review at 56fe503. Both threads from my earlier review are addressed: the merge of main brings the built-in grader dispatch, and the new runner and unit tests now pin guidance_read, counters and process: null. Three things are worth fixing before merge, plus three nits.

What I ran, on a Linux review host at 56fe503:

• make ci-local: 27 passed, 1 failed, 7 skipped. 7,173 tests. The failure is pip-audit on the lock files, which this PR does not touch; the changelog, schema-compat, docs and gitleaks jobs pass.
• Reproductions of the two guidance_read findings through events.convert and process_metrics, with no model involved.
• A mutation run: 62 mutants of process_metrics.py, manifest.py, runner.py and repo_run.py against the four test files that cover them; 11 survive, 6 of them real gaps (finding 1).

Findings that cannot be anchored inline:

1. The tests still let four regressions through. Each mutant below passes test_repo_process_metrics.py, test_repo_run.py, test_repo_builtin_graders.py and test_repo_manifest.py (260 tests):

   • BUILTIN_GRADER_PREFIX = "dokimi", and BUILTIN_GRADER_PREFIX not in t.grader in grade_problems (manifest.py:377). The tests only use dokimi:x, dokimi:review_replay and tests, so a repository grader named dokimi_tests or x-dokimi:y would skip the "grade required" error and every cell would then fail grading after the agent is paid for. Add grade_problems(m, [summary("a", "dokimi_tests"), summary("b", "x-dokimi:y"), summary("c", "dokimi")]) and expect one problem naming a, b and c.
   • e.get("kind") == "read" replaced by e.get("path") (process_metrics.py:70): an edit of AGENTS.md would count as read. The only edit event in the tests is src/lib.c. Add {"kind": "edit", "path": "AGENTS.md"} and expect it absent.
   • _in_workspace as not str(p).startswith("..") (process_metrics.py:60): only a leading ../ is tested. Add src/../AGENTS.md and a/../../x/AGENTS.md.
   • additionalProperties: false removed from the object form of a counter in repo_manifest.v1.json: {regex: x, mach: output} is then accepted and silently counts against text. Add it to the refusal cases in test_repo_manifest.py.

2. Nit: the contract still says the opposite in §4.5. repo-evaluation-contract.md:534 (outside this diff) reads "entry_points.grade is still required (repo_manifest.v1), even when every task uses a built-in grader." The schema, grade_problems, §2 and the changelog fragment now say it is not. Suggestion: reword the sentence to say entry_points.grade is optional when every task's grader starts with dokimi:.

Waiting for the author: @xaleryb's question on process for an agent that exits before writing any trace line waits for the author. At this head such a run gets process with every count at zero, the same as a short run with no tool call.

Checked, no issues

• Schema versioning: dropping required: ["grade"] and adding process_metrics only widens repo_manifest.v1; schema_compat against the base reports OK.
• Counting rules: failed_commands skips rc null and denied commands; denied_commands and the counters still see denied commands; a bad regex, match or name is refused with the metric named. The schema and parse_counter agree on every shape I tried.
• Grade-optional rule: _require_grade honours --task, runs before the pin and the image build, and a cell with only dokimi: graders and no entry_points.grade grades.
• Offline: the new tests need no network or Docker.

[antonwolfy]

I pushed a commit per nit, up to 0fbf76c, which fix the three nits from my review:

• 406d986: dokimi/dokimi/repos/manifest.py:57: removed BUILTIN_GRADER_PREFIX and made grade_problems use the imported BUILTIN_PREFIX.
• 81e80b4: dokimi/docs/repo-evaluation-contract.md:210: the name rule now says a name cannot start with a digit.
• 0fbf76c: dokimi/docs/repo-evaluation-contract.md:534: §4.5 now says entry_points.grade is optional when every task's grader starts with dokimi:.

make ci-local at 0fbf76c: 27 passed, 1 failed, 7 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving yet. Still open: the three risks from my review.

[napetrov]

Review items at 56fe503 and the merge of main, mapped to commits:

• Finding 1 (four surviving mutants): pinned in f9b314c. test_a_grader_that_only_resembles_a_built_in_one_needs_grade (dokimi_tests, x-dokimi:y, dokimi), test_only_a_read_counts_as_reading_guidance (edit of AGENTS.md), test_guidance_paths_are_normalized_before_the_workspace_test (src/../AGENTS.md, a/../../x/AGENTS.md), and the mach typo case in test_repo_manifest.py's refusal list.
• Nits (BUILTIN_GRADER_PREFIX, name rule, §4.5 grade optional): 406d986, 81e80b4, 0fbf76c (already pushed).
• Conflict with main: merged in f587046 (kept the arm uptake record and the process record in run_cell, and the FAKE_PROCESS / FAKE_TOOL_USES fixture hooks together).
• xaleryb's open point (agent that exits before any trace line gets zero counts): 135a5ed, process is null under the same transcript.events or transcript.num_turns is not None condition as uptake; asserted in test_an_agent_that_left_no_trace_is_unmeasured_one_that_called_no_tool_is_measured.
• The two inline threads (failed read, ./ paths) are answered in their threads (f9b314c).

[antonwolfy]

LGTM. Both threads from my last review are fixed and pinned: a failed read no longer counts in guidance_read, and paths are normalized before the workspace test. One nit inline.

What I ran, on a Linux review host at 1b9bda8:

• make ci-local: 26 passed, 3 failed, 7 skipped. 7,782 tests. The schema-compat, changelog, docs and CLI-surface jobs pass. pip-audit fails on the lock files, which this PR does not touch. The two gitleaks jobs fail on the tool install and the missing binary, not on a finding; gitleaks detect over the PR's 12 commits reports no leaks.
• A mutation run on process_metrics.py: 19 mutants (rc filter, denied filter, normpath, the ~, .. and absolute-path checks, failed_commands variants, sorted, any/output fields, search, tool_calls), all killed.
• Reproductions of the edge cases through events.convert and process_metrics, with no model involved.

Checked, no issues

• Schema versioning: dropping required: ["grade"] and adding process_metrics only widens repo_manifest.v1; schema_compat against the base reports OK.
• Grade-optional rule: grade_problems is used by check_repo and by _require_grade in repo run, so a repository grader without entry_points.grade is refused before the pin.
• Docs: repo-evaluation-contract.md is in the nav; the manifest rules and the process table match the code.

[antonwolfy]

I pushed a commit per nit, up to f090697, which fix the nit from my review:

• f090697: dokimi/changelog.d/20261021_140000_napetrov_d4-process-metrics.md:3: the fragment now says process is null when the agent left no readable trace.

make ci-local at f090697: 26 passed, 3 failed (host: gitleaks install, gitleaks scan, pip-audit), 7 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving until CI passes at this head.

[vlad-perevezentsev]

A command that a PreToolUse hook refused keeps the hook's message as its output ("PreToolUse:Bash hook error: ..."), so an output or any counter searches that message. With errors: {regex: error, match: output}, make all & refused by no_background.py gives counters.errors == 1, though the command never ran. Counting denied commands makes sense for text (the agent asked for it), not for output. Suggestion: skip output for events with denied: true, and say so in §2.

[napetrov]

Fixed in a6b20af: a denied command is now searched in its text only, so the hook refusal message no longer counts as the output. errors with match: output or any stays 0 for a refused make all &. Pinned by test_a_refused_command_is_searched_in_its_text_not_in_the_hook_message. Section 2 of the contract says so too.

[vlad-perevezentsev]

guidance_read counts only read events (the Read tool). An agent that runs cat AGENTS.md or grep -n rule CLAUDE.md through Bash read the guidance, but guidance_read is [], which looks like "did not read it". Either also match the command text against the guidance globs (as arm_uptake does for added files), or rename/document the field as "guidance opened with the Read tool" so an empty list is not read as a result.

[vlad-perevezentsev]

run_process_metrics runs inside the grade try and before agent.update(...). If it raises (or a manifest regex backtracks badly on 8 KB of output), the cell gets grader_failure and loses cost_usd, num_turns and tokens, so a best-effort metric decides the task result. Suggestion: compute process after agent.update, in its own try that leaves it null and adds to trace_errors.

[napetrov]

Fixed in a6b20af: process is computed after agent.update, in its own guard. A metric that raises leaves process null, adds a trace_errors entry, and the cell keeps its cost, turns and grade. Pinned by test_a_process_metrics_error_leaves_process_null_and_keeps_the_cost_and_the_grade.

[napetrov]

Thanks for the reviews. Each open item and the commit that answers it, on head 79c20c5:

• denied command output counted by a counter, and a metric error failing the cell: a6b20af (tests named in the threads)
• BUILTIN_PREFIX, the name rule, and the changelog nit: 406d986, 81e80b4, f090697

Ready for another look.

[antonwolfy]

LGTM. Both fixes since my last review hold: a refused command is searched in its text only, and a metric that raises leaves process null while the cell keeps its cost, turns and grade. One nit inline.

What I ran, on a Linux review host at 79c20c5:

• make ci-local: 29 passed, 0 failed, 7 skipped. 8,426 tests. Changed-line coverage, schema-compat, changelog, docs, gitleaks and pip-audit pass.
• A mutation run: 28 mutants of process_metrics.py and the run_cell wiring in runner.py against test_repo_process_metrics.py, test_repo_run.py, test_repo_manifest.py and test_repo_builtin_graders.py; 26 killed. The two survivors are test gaps with no effect on results: a guidance_globs predicate that honours only the first glob (no test uses two globs), and if events is not None replaced by if True (only adds a process metrics failed line to trace_errors for an unreadable trace).
• Reproductions through events.convert and process_metrics, with no model involved: tool_result pairing out of order, an orphan result, a subagent read, a denied command next to a failed one. The output is as the contract says.

@vlad-perevezentsev's point that guidance_read misses guidance read through Bash (cat AGENTS.md) waits for the author. The contract row says "of read events", so the field does what it documents; widening or renaming it is the author's call.

Checked, no issues

• Schema versioning: dropping required: ["grade"] and adding process_metrics only widens repo_manifest.v1; schema_compat against the base passes. The schema and parse_counter give the same verdict on every shape I tried.
• Grade-optional rule: _require_grade runs before the pin in repo run and repo tasks check, and uses the same dokimi: prefix as the per-task check.
• events.py refactor: on the same streams, base and head differ only by the new rc on read events; command, edit, skill and task events are identical.

[antonwolfy]

LGTM. I pushed a commit per nit, up to 654b6b1, which fix the nit from my review:

• 654b6b1: dokimi/docs/repo-evaluation-contract.md:435: process is null also when computing the metrics failed, with the error in agent.trace_errors.

make ci-local at 654b6b1: 29 passed, 0 failed, 7 skipped.

Not merging yet: @vlad-perevezentsev on guidance_read counts only Read-tool events, not guidance read through Bash (cat AGENTS.md); widen it or document/rename it waits for your answer, @napetrov.

[napetrov]

@antonwolfy @vlad-perevezentsev answering the open `guidance_read` question (Vlad's Bash `cat AGENTS.md` point).

Kept Read-tool-only and documented it, rather than parsing Bash command text. The contract row, the `process_metrics` docstring and the changelog fragment now say `[]` means no Read-tool read. To count a Bash read, add a `process_metrics` counter on the command text, e.g. `guidance_cat: '\bcat\b.*AGENTS\.md'`; counters also count denied commands.

Commit 0696bb6 (docs and docstring only, on top of 654b6b1). Tests: 121 passed on `test_repo_process_metrics.py` and `test_repo_run.py`; ruff clean. Local self-review: no blocking findings.

Still pending: reviews from @xaleryb, @jharlow-intel and @vchamarthi.