[napetrov]

Summary

New context consistency: reads repeated Claude Code stream-json traces of one task and answers three questions apart. No model calls.

Axis	Question	Numbers
Result	solves it every time?	success_rate, pass@k, pass^k
Path	same steps?	mix_jsd, order_distance, action_distance, CV of calls/turns/tokens
Contract	kept the required process?	conformance_rate, violations per rule, passed_but_nonconforming

context consistency --arm A=glob --arm B=glob [--outcomes verdicts.json] [--contract contract.yaml] [--shell split|call]
  parse_run          # main-thread steps; Bash split per chained command; sed -i / > / tee carry file_path
                     # blocked Stop hook = failed Hook:Stop step
  group by order distance -> verdict per strategy (stable success / valid diversity / unstable / stable failure)
  check_contract     # requires, forbids, before, after_last, last_ok

Third commit fixes three bugs found while using it:

 tool_result is_error on a Bash call
-  mark last chained command failed      # `make check && git diff` passed last_ok
+  mark every chained command failed     # trace does not say which one
 step piped without pipefail (`pytest | tail`)
+  masked; last_ok -> "last_ok: test (status hidden by a pipe)"
 _written
-  regex over raw text                   # `echo 'a > b'` was a write
+  shlex punctuation_chars tokens

Fourth commit wires it into repo compare (item 4 of NEXT_STEPS), opt-in, descriptive, outside Holm:

 repo compare <batch> --baseline none
+  --pass-k K         # pass^K per task = C(c,K)/C(n,K); <K cells -> unpaired
+  --path-stability   # path_distance per task = mean pairwise order distance of events.jsonl paths
 measures: gate, score, [pass^K], [path_distance], cost_usd, turns, metric:*

events.jsonl is harness-agnostic, so path stability works for any harness repo run converts (consistency.run_from_events).

Docs: docs/run-consistency.md, docs/repo-evaluation-contract.md.

Evidence

• Used on 96 + 60 real runs (haiku/sonnet, skill vs script vs Stop-hook arms). Harness success was 100% everywhere; verifier outcomes and the contract separated arms (hook vs script p = 0.0006), and split mode exposed sonnet's single-Bash sed edits that call grain hid.
• Before: make check && git diff failing → check_contract == []; pytest -q | tail -5 → [].
  After: ["last_ok: test"]; ["last_ok: test (status hidden by a pipe)"]; with set -o pipefail → [].
• repo compare test: t1 2/3→2/3, t2 2/3→3/3: gate delta moves one task by 1/3; pass^3 shows baseline 0.0 vs treatment 0.5 (t2 became reliable). Path test: baseline cells grep,pytest vs git,make → 1.0; treatment grep && pytest vs grep; pytest → 0.0.
• Full suite: 6739 passed, 22 skipped; ruff, mypy clean. (Earlier run had one test_reach_timeout_kills_the_harness_process_tree[orphan] failure under -n auto load; passes alone and on main.)

Merge Danger

Door: two-way

New subcommand and module; repo compare gains two opt-in flags, default output unchanged.

Blast Radius: small

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 95.41716% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.25%. Comparing base (4b6b964) to head (d218906).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/context/consistency.py	95.51%	8 Missing and 11 partials ⚠️
dokimi/dokimi/commands/context.py	83.83%	11 Missing and 5 partials ⚠️
dokimi/dokimi/context/shell.py	98.55%	1 Missing and 3 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #155      +/-   ##
==========================================
+ Coverage   93.21%   93.25%   +0.04%     
==========================================
  Files         270      272       +2     
  Lines       34278    35117     +839     
  Branches     5766     5964     +198     
==========================================
+ Hits        31952    32750     +798     
- Misses       1526     1548      +22     
- Partials      800      819      +19     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	93.80% <100.00%> (+0.03%)	⬆️
dokimi/dokimi/repos/compare.py	99.48% <100.00%> (+0.05%)	⬆️
dokimi/dokimi/context/shell.py	98.55% <98.55%> (ø)
dokimi/dokimi/commands/context.py	90.18% <83.83%> (-9.82%)	⬇️
dokimi/dokimi/context/consistency.py	95.51% <95.51%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 8b8bd09. The metrics (pass@k, pass^k, JSD, order distance, the C(c,K)/C(n,K) form in repo compare) are right, and the default repo compare output is unchanged. The problems are in the shell-step parsing, in how empty or unreadable input is reported, and in the strength of the tests. Inline comments cover the first ones; the test gap is below.

What I ran, on a Linux review host at 8b8bd09:

• make ci-local: 24 passed, 2 failed, 6 skipped. 6759 tests passed, 1 failed: test_unbalanced_quote_in_hook_completes_quickly (1.25 s against a 1 s limit, under 8-worker load). pip-audit reports virtualenv 21.14.7 (CVE-2026-113722) from the existing locks, fixed on main by #152. Neither comes from this PR. The changed-line coverage gate passed.
• Reproductions for the code findings, with the LLM transport mocked (none of this code calls a model).
• A mutation run over the two new test files: 49 mutants applied, 21 pass the full suite.

Tests do not pin several rules. tests/test_context_consistency.py and tests/test_repo_compare.py survive these one-line mutants:

• forbids: ok = not a changed to ok = True passes. The test contract has forbids: push, but no fixture run pushes.
• before / after_last: a[0] < b[0] changed to <=, and a[-1] changed to a[0], both pass.
• passed_but_nonconforming: dropping the r.passed and condition passes.
• Metric denominators: success_rate and pass^k over n instead of judged pass, because no fixture has an unjudged run.
• run_from_events: rc != 0 -> False and denied -> False both pass, so failure marking for --path-stability is untested.
• repo compare: if pass_k and not score_task changed to if pass_k passes (no score-task test); order_distance(runs[:2]) instead of all runs passes (the tests use two cells per arm); except ValueError: continue changed to raise passes.
  Add a fixture per rule where the rule is violated, one arm with an unjudged run, and a three-cell path case.

Checked, no issues

• Metrics: pass_at / pass_hat are guarded by k <= judged, JSD stays in 0..1, and the single-linkage grouping and CV are safe on zero or one sample.
• Holm: the new measures are outside PRIMARY, so the family size is unchanged. With the flags absent no events.jsonl is read.
• Repo rules: the new page is in mkdocs.yml nav and the docs index, the changelog fragment has the scriv format, no schema or SCHEMA_VERSIONS entry applies, and no generated files are committed.
• Docs vs code: every flag and metric name in the docs matches the CLI help and the code.

[antonwolfy]

@napetrov benchmark now fails on this PR:

ValueError: golden_manifest violation: matched 32 document(s) under .../dokimi, more than max_docs=31. include=['docs/**/*.md'] exclude=['docs/archive/**']

It passed at 8b8bd09 because main was older then. Since that run, #139 has added docs/research-register.md and set max_docs: 31 for it. This PR adds docs/run-consistency.md, and GitHub tests the PR merged into current main, so the count is 32. Merged as is, it would break benchmark on main.

Suggestion: in dokimi/benchmarks/spec.v1.yaml, set max_docs: 32 and change the comment's "Measured selection at the time of writing: 31 files" to 32. That file asks for the bump in the same change that adds the doc.

[napetrov]

Review at 8b8bd09, test gaps, all pinned in 911a507 (each checked by applying the one-line mutant and seeing a test fail):

• forbids: test_forbids_fires_on_a_run_that_pushes
• before strictness (a[0] < b[0]): test_before_needs_a_strictly_earlier_action
• after_last uses the last action: test_after_last_looks_past_the_last_action_not_the_first
• passed_but_nonconforming: test_passed_but_nonconforming_counts_passes_only
• success_rate and pass^k over judged runs, with one unjudged run: test_result_rates_count_judged_runs_only
• run_from_events rc and denied marking: test_run_from_events_marks_failed_commands_and_denied_tools
• pass_k only on gate tasks: test_pass_k_is_a_gate_measure_not_a_score_measure
• order_distance over all cells (three-cell case): test_path_distance_is_the_mean_over_every_pair_of_three_cells
• bad lines skipped, not raised: covered by the empty/unparseable and nested-line tests.
  Also: merged intel/main (a5ea88b), and benchmarks/spec.v1.yaml max_docs 31 -> 32 after recounting docs/**/*.md minus docs/archive (32), in 6148101; make benchmark-run passes locally.

[antonwolfy]

Review at 911a507. The fixes for the earlier threads hold: the "no runs" verdict, the clean error for malformed --outcomes, the refusal of a shared outcome key, the empty-trace and nested-line cases in repo compare, the max_docs bump and the --reps wording. The shell-step parser is still the weak part, and the new shell-edit rule has a side effect. Six new points below, plus one thread still open.

What I ran, on a Linux review host at 911a507:

• make ci-local: 27 passed, 1 failed, 7 skipped. 7220 tests passed. The one failure is pip-audit (gitpython 3.2.0, CVE-2026-114145, fixed in 3.2.1) from the existing locks; the PR changes no lock file. The changed-line coverage gate passed.
• Reproductions for the code findings, with no model involved (none of this code calls one).
• A mutation run over the two new test files: 72 mutants, 29 pass the full suite, 6 of those equivalent. The ones that matter are in point 2 below.

Still open from my earlier review:

• dokimi/dokimi/context/consistency.py:207: the program name is cleaned, but the step's action and command still carry the syntax words. for f in a; do black x; done gives the action Bash:black(do black x), a plain black x gives Bash:black(black x), and seq_distance between them is 1.0. So action_distance still splits identical strategies, and a matcher command: ^black does not match the loop step ([False] against [True] for the bare command). The same holds for if pytest -q; then ...; fi (Bash:pytest(if pytest -q)) and (cd x && pytest -q) (Bash:pytest(pytest))). Suggestion: build the action and command text from the words after _strip_syntax, not from the raw segment.

Findings that cannot be anchored inline:

1. repo compare --pass-k / --path-stability flood unpaired with entries that are not unpaired tasks. compare.py:292-296 adds an entry for every task that lacks the measure: score tasks (pass^K is gate-only), tasks with fewer than K cells, and tasks with fewer than two traced cells. The reason reads "no pass^2 in baseline or treatment", which is wrong for a score task. repo run defaults to --reps 1, so --path-stability on a default batch lists every task. Repro: 15 tasks with one cell per arm plus one task only in the baseline; compare(..., pass_k=2, path_stability=True) returns 31 unpaired entries, and the text shows ten "no pass^2 in baseline or treatment" lines and "and 21 more". Without the flags the same batch prints one line, zz_only_base (m-1, no cells in treatment), the only real one. Suggestion: keep a task out of unpaired when the measure does not apply to it, and report the count of tasks without the measure in one line.

2. Tests still do not pin several rules added in the fixes. Mutation check, one-line mutants that pass both new test files (140 tests):

• Matcher.__call__: dropping and set(self.tools) & _WRITE_TOOLS (so tool: Read also matches a shell write) passes.
• parse_run: except (ValueError, RecursionError) changed to except ValueError passes; only _cell_path has the nested-line test. A trace line of "[" * 100000 still aborts context consistency.
• _written: dropping prev != "1" or the (?<!\S) lookbehind in _fd_redirect, removing ">|" from the operators, len(files) >= 2 changed to >= 1 (so sed -i 's/a/b/' with no file reports a file), and tee without the pipe-or-start condition (echo tee out.txt becomes a write) all pass.
• apply_outcomes: removing the stem match passes. load_outcomes: accepting int values passes.
• Run.shape: if not out or out[-1] != k changed to if True passes, so nothing checks that repeats collapse.
  Add one case per line: _written("echo tee out.txt") == "", _written("sed -i 's/a/b/'") == "", _written("echo x >| f") == "f", a Read matcher against a sed -i step, a stem key for dir/run1.jsonl, load_outcomes on {"a": 1}, parse_run on a 200000-bracket line, and runs Read Read Read Edit / Read Edit as one strategy.

[xaleryb]

• Ran: the full suite on main eaad684 merged with this head plus the commits below, ruff, mypy, bandit, and the docs contract on synthetic traces.
• Ratchet: on the merged tree tests/test_growth_ratchet.py fails. It reports context.py::cmd_context_consistency at 17 and consistency.py::parse_run at 14, against a cap of 10. Expected: split both into helpers, then merge main so CI runs the ratchet.
• pytest -q; echo done, pytest -q || true and pytest -q followed by a newline and git status, with is_error false: last_ok: test passes. Expected: "status hidden", as pytest -q | tail -5 already gets. set -e should exempt ; and newline, but not ||.
• timeout 300 pytest, nohup pytest, nice -n 10 pytest produce Bash:timeout, Bash:nohup, Bash:nice. Unquoted X=$(git rev-parse HEAD) && pytest adds a Bash:rev-parse step. Expected: Bash:pytest only.
• tool_calls_* and failed_calls_mean count split steps. One failed call cd x && make && make test && git diff counts as 3 failed calls. Expected: count calls, or rename the measures and record --shell in the JSON.
• Commits: c60f7d2 makes saving a test log (pytest > test.log, | tee) no longer an edit; 3235f26 counts a trace given twice once; 5a11251 adds tests for outcome keys, verdicts and the shell guards.

[antonwolfy]

Review at 5a11251. The fixes since 911a507 hold: a saved test log (pytest > test.log, git diff > f.patch) is no longer an edit, and a trace given twice counts once. Four of the earlier threads are still open, and three new points below.

What I ran, on a Linux review host at 5a11251:

• make ci-local: 27 passed, 1 failed, 7 skipped. 7229 tests passed. The failure is pip-audit on the existing lock files, which the PR does not touch.
• Reproductions for the code findings, with no model involved (none of this code calls one).
• Mutants from my last review re-applied to the head: perl/tee/>|/lookbehind/len(files) >= 2/RecursionError/int outcome/Run.shape still pass both test files (see point 1).

Still open from my earlier review:

• dokimi/dokimi/context/consistency.py:227: _segments still mis-splits \;, $(...), & and case. find . -name '*.pyc' -exec rm {} \; -print gives Bash:find, Bash:-print; echo don\'t; ls is one step; make & pytest loses pytest; case $x in a) make ;; b) ls ;; esac gives only Bash:b).
• consistency.py:207: the step's action and command keep the syntax words. for f in a; do black x; done gives Bash:black(do black x) against Bash:black(black x) for black x, action_distance 1.0, and command: ^black is [False] against [True]. if pytest -q; then make; fi gives if pytest -q / then make; (cd x && pytest -q) leaves pytest -q).
• consistency.py:441 and :427: parse_contract still leaks TypeError for requires: 5, requires: [[a]] and tool: 5, and the CLI prints a traceback. requires: {a: 1} is accepted as [a].
• consistency.py:569: a trace with tool calls and no result event stays unjudged and drops out of success_rate, pass@k and pass^k. The text table shows runs but not judged, and nothing warns.

Findings that cannot be anchored inline:

1. Tests still do not pin several rules. Re-applied one-line mutants that pass test_context_consistency.py and test_repo_compare.py (149 tests):

• except (ValueError, RecursionError) in parse_run changed to except ValueError;
• (?<!\S) removed from _fd_redirect, and ">|" removed from the operators;
• len(files) >= 2 changed to >= 1 (_written("sed -i s/a/b/") then returns s/a/b/);
• the tee pipe-or-start condition made True (_written("echo tee x") returns x);
• load_outcomes accepting {"a": 1};
• Run.shape with if True (repeats no longer collapse);
• perl removed from _SHELL_WRITERS, since no test uses perl -pi.
  Add one case per line: _written("echo tee x") == "", _written("sed -i s/a/b/") == "", _written("echo x >| out.txt") == "out.txt", _written("echo a2>/dev/null 2 > f") == "f", load_outcomes on {"a": 1} raising, parse_run on a 200000-bracket line, runs Read Read Edit and Read Edit as one shape, and a perl -pi -e s/a/b/ src/a.py edit.

I agree with @xaleryb on the growth ratchet and on the timeout / nohup / nice wrapper steps. New evidence: ruff --select C901 at max 10 reports cmd_context_consistency 17 and parse_run 14, and timeout 300 pytest gives the step Bash:timeout.

[napetrov]

Review-body items at 5a11251 and the open points of @xaleryb, mapped to commits (merge of main in bba5f9c; the rest pushed as 97f9581 and f4fa3ce):

• Surviving mutants: pinned in tests/test_context_shell.py and test_repo_compare_paths.py (97f9581): echo tee x, sed -i s/a/b/ with no file, >|, the 2> lookbehind case, load_outcomes on {"a": 1}, a 200000-bracket line, Read Read Read Edit vs Read Edit as one shape, perl -pi, a Read matcher against sed -i, a stem outcome key.
• Growth ratchet: parse_run and cmd_context_consistency split into helpers, the shell reading moved to dokimi/context/shell.py (each function under the cap), and the compare tests that pushed test_repo_compare.py over 1000 lines moved to test_repo_compare_paths.py. test_growth_ratchet passes on the merged tree.
• Status hidden after ;, a newline, || and & (pytest -q; echo done, pytest -q || true): last_ok reports it as hidden; set -e exempts ; and a newline but not ||; && does not hide. test_a_later_command_hides_the_status_of_an_earlier_one, both grains.
• timeout, nohup, nice wrappers and X=$(...) && pytest: one Bash:pytest step.
• tool_calls_* / failed_calls_mean count calls, not steps (cd x && make && make test && git diff failing is 1 failed call); the JSON records the shell grain. test_a_chained_call_is_one_call.
• The unpaired flood, the action/command text, _segments, parse_contract and the no-result run are answered in their threads.
• Docs (run-consistency.md) and the changelog fragment describe all of it.

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi context consistency`
    + argument `traces`
    + option `--arm`
    + option `--contract`
    + option `--help/-h`
    + option `--json`
    + option `--outcomes`
    + option `--shell`
    + option `--threshold`
~ `dokimi repo compare` parameters changed:
    + added option `--pass-k`
    + added option `--path-stability`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[antonwolfy]

Review at e696966. All six open threads are fixed: the compound-syntax and wrapper handling, the scanner, parse_contract errors, a run with no result event, the unpaired count and the prefixed writes. The remaining problems are in the new shell scanner, which still yields wrong steps for a few common forms, and one test gap. Three bugs and two risks are inline.

What I ran, on a Linux review host at e696966:

• make ci-local: 28 passed, 1 failed, 7 skipped. 7,895 tests passed. The changed-line coverage gate passed. The one failure is pip-audit (filelock 3.32.7, setuptools 84.0.0) on the existing lock files; this PR changes no lock file.
• Reproductions for the code findings, with no model involved (none of this code calls one).
• A mutation run over the four new test files; the survivor I re-ran and kept is the run_from_events mapping below.

Checked, no issues

• Metrics: pass^K is C(c,K)/C(n,K) and is skipped when a task has fewer than K cells; path_distance is the mean over all pairs of cells (three cells give 2/3); empty, garbage and deeply nested events.jsonl lines are skipped, not raised.
• Unpaired: score tasks are left out for pass^K, and tasks with too few cells are one line with a count.
• Repo rules: the docs page is in mkdocs.yml nav and the docs index, the changelog fragments are present, no schema applies, and no generated files are committed.
• Scanner: \;, $(...), backticks, a lone &, case labels, for/if/while/(...) syntax words, sudo/env/uv run/timeout prefixes, >|, and heredoc bodies all give the right steps.

[antonwolfy]

Review at a22b8c5. My five earlier threads are fixed and pinned: the { } group, the wrapper value options, $# in a word, the &> / >& writes with multi-file sed -i, and the read/edit mapping. Three points remain: two bugs in the new forms and one test gap.

What I ran, on a Linux review host at a22b8c5:

• make ci-local: 29 passed, 0 failed, 7 skipped. 8,371 tests passed, 2 skipped. Changed-line coverage 96% (734 lines).
• Reproductions for the code findings, with no model involved (none of this code calls one).
• A mutation run over the changed shell code: 33 mutants, 8 survive the full suite; the ones that matter are in the last inline comment.

Checked, no issues

• Threads: { ruff check .; pytest; }, { make; } and { a; b; } > f.txt give no } step. echo $# > out.txt and echo ${#x} > out.txt write out.txt, echo ${#x} | grep 1 is masked, and a real # comment is still cut. echo hi &> f, &>> and >& f write f, while >&2, 2>&1 and >&- do not.
• Read/edit: tool = "Read" for every edit event, and a read/edit swap, both fail test_a_read_and_an_edit_are_different_steps_for_path_stability.
• Calls vs steps: a multi-file sed -i gives one step per file, and tool_calls_* / failed_calls_mean still count calls.

[napetrov]

Thanks for the review at a22b8c5. Each open item -> sha:

• Long-form wrapper options (sudo --user, env --unset, timeout --signal, nice --adjustment, uv run --extra): 2e11ee6, test_control_syntax_and_wrappers_are_not_steps.
• sed -i --expression=..., sed -i -es/.../, BSD sed -i '': 2e11ee6, test_written_files_of_an_in_place_edit.
• Test gap (the one-line mutants): 2e11ee6, rows in the same two tests.

Ready for another look.

[vlad-perevezentsev]

This case-label pattern also matches a one-word command that closes a subshell when a redirect or pipe follows it. (cd build && make) 2>&1 | tail -5 gives one step Bash:2>&1 with the command 2>&1 | tail -5, so make is lost, and (cd sub; pytest) > log.txt gives Bash:>. A contract requires: {tool: Bash:make} then reports that the run never built. Please peel a word) label only inside a case ... in block, and add these two lines to the tests.

[napetrov]

Fixed in 70da7a6: a word) is a case label only inside case ... in (after the header or ;;); an unopened ) elsewhere closes a subshell. (cd build && make) 2>&1 | tail -5 gives Bash:make, (cd sub; pytest) > log.txt gives Bash:pytest; both lines plus nested/after-esac cases are in test_context_shell.py.

[napetrov]

Fixed on the branch at e8f0174: a word) is a case label only inside a case ... in block, and elsewhere an unopened ) closes a subshell. The two lines from the review now give make and pytest, and both are in test_context_shell.py.

[vlad-perevezentsev]

A pattern value is compiled with str(v), so a list becomes a regex character class: command: [pytest, tox] compiles to ['pytest', 'tox'], which matches almost any command, and requires: test is met by git status. tool already refuses anything but a name or a list of names. Please refuse a non-string pattern with a ValueError, or join a list of strings into one alternation.

[napetrov]

Fixed in 70da7a6: a list of strings is joined into one alternation (command: [pytest, tox] no longer matches git status); a number, mapping or empty list is a ValueError. Tests added.

[napetrov]

Fixed on the branch at e8f0174: a list of strings is one alternation, so command: [pytest, tox] no longer matches git status, and a number, mapping or empty list is refused with a ValueError. Both cases have tests.

[vlad-perevezentsev]

Clearing the separator when only done/fi follow also clears it where the status is not the command's. pytest -q & always reports 0, but its step is not masked, so last_ok: test passes. In for f in a b; do pytest $f; done the status is the last iteration's, so a failure in the first one reads as a pass. Please keep a trailing & as hiding, and mark a step inside a for/while/until body as masked.

[napetrov]

Fixed in 70da7a6: a trailing & keeps hiding, and a step inside a for/while/until/select body is masked (not under set -e). Loop headers are read as shell words, so quoted for or $(for ...) does not mask. Rows added to the hidden-status test.

[napetrov]

Fixed on the branch at e8f0174: a trailing & keeps hiding, and a step inside a for, while, until or select body is masked unless set -e is on. Loop headers are read as shell words, so a quoted or substituted for does not mask. The rows are in the hidden-status test.

[vlad-perevezentsev]

int(usage.get("output_tokens") or 0) raises on a value that is not a number: "n/a" gives ValueError and Infinity (which json.loads accepts) gives OverflowError. parse_run has no guard for it and the command catches only TraceError, so context consistency ends in a traceback. Please skip a count that is not a finite number, or catch these errors per line as context/trace.py does.

[napetrov]

Fixed in 70da7a6: a count that is not an int or a finite float ("n/a", Infinity, NaN, bool, list) is skipped; a huge int is still counted. Tests added.

[napetrov]

Fixed on the branch at e8f0174: a usage count that is not an int or a finite float is skipped, so n/a, Infinity and NaN no longer raise, and a huge integer is still counted. The tests cover each value.

[vlad-perevezentsev]

output_tokens starts at 0, so traces without usage give output_tokens_cv == 0.0: a token spread nobody measured reads as perfectly stable. Please keep output_tokens as None when no message had a usage count, and leave such runs out of the CV, as turns_cv does.

[napetrov]

Fixed in 70da7a6: output_tokens is None when no message had a usage count, and such runs stay out of output_tokens_cv, as turns_cv does.

[napetrov]

Fixed on the branch at e8f0174: output_tokens is None when no message had a usage count, and such runs are left out of output_tokens_cv, as turns_cv does. test_traces_without_usage_have_no_token_spread pins it.

[vlad-perevezentsev]

_mean(...) or 0.0 gives a one-run strategy an inner_distance of 0.0, which reads as a tight group, but nothing was measured. Together with _summary_verdict, two passing runs on different paths read as "2 strategies, all stable success (valid diversity)" although each strategy has one run. Please keep None for a one-run strategy, and do not count "single run, passed" as stable success in the summary verdict.

[napetrov]

Fixed in 70da7a6: a one-run strategy has inner_distance: null, and only stable success groups make "valid diversity"; "single run, passed" no longer does. Documented in run-consistency.md.

[napetrov]

Fixed on the branch at e8f0174: a one-run strategy has inner_distance null, and single run, passed no longer counts as stable success for valid diversity. run-consistency.md says so, and a test pins the verdict.

[vlad-perevezentsev]

&& and ; inside [[ ... ]] and (( ... )) split the line. [[ -n $x && -f y ]] && pytest gives a step Bash:-f, and for ((i=0; i<3; i++)); do pytest; done gives Bash:i<3 and Bash:i++. These fake steps change order_distance, action_distance and mix_jsd. Please track [[/]] and ((/)) in the scanner like $(...), and add both lines to the tests.

[napetrov]

Fixed in 70da7a6: the scanner tracks [[/]] and ((/)) like $(...); (( ... )) alone is no step. Both lines are in the tests.

[napetrov]

Fixed on the branch at e8f0174: the scanner tracks [[ ]] and (( )) like $( ), so && and ; inside them no longer split a line. [[ -n $x && -f y ]] && pytest and the C-style for loop now give only pytest, and both lines are in the tests.

[antonwolfy]

Review at 2e11ee6. The three threads from my last review are fixed and pinned, but eight points from @vlad-perevezentsev's review at a22b8c5 still reproduce at this head, so this is not ready to merge. One nit inline.

I agree with @vlad-perevezentsev's eight points from a22b8c5; each still reproduces at this head.

What I ran, on a Linux review host at 2e11ee6:

• make ci-local: 28 passed, 1 failed, 7 skipped. 8,572 tests passed, 1 failed: test_unbalanced_quote_in_hook_completes_quickly (1.15 s against a 1 s limit under 8-worker load; it passes alone in 0.73 s, and the PR does not touch that code). Changed-line coverage 96% (738 lines).
• Reproductions for the code findings, with no model involved (none of this code calls one).
• A mutation run over the changed shell code: 38 mutants, 29 killed. The four mutants from my thread are all killed; the survivors are in the inline nit.

Checked, no issues

• Wrapper options: uv run --extra dev pytest, --group dev, sudo --user bob make, env --unset FOO make, timeout --signal KILL 5 make and nice --adjustment 5 make give pytest / make as the program.
• sed -i: --expression=s/a/b/ a.py, -es/a/b/ a.py, -fa.sed, BSD -i '', -i s/a/b/ f > log.txt and 2>err.log each give the one real file.

[antonwolfy]

I pushed a commit per nit, up to d2e8f51, which fix the nit from my review:

• d2e8f51: dokimi/tests/test_context_shell.py:92: added the two sed script-flag rows and the five sudo/env/time/timeout prefix rows from the suggestion.

make ci-local at d2e8f51: 29 passed, 0 failed, 7 skipped.

Not approving yet. Still open: @vlad-perevezentsev on shell.py: command -v pytest is read as running pytest (Bash:pytest step, satisfies requires/last_ok); @vlad-perevezentsev on shell.py:167: (cd build && make) 2>&1 | tail -5 gives only Bash:2>&1 and (cd sub; pytest) > log.txt gives Bash:>; make/pytest lost; @vlad-perevezentsev on shell.py: [[ -n $x && -f y ]] && pytest gives Bash:-f; for ((i=0; i<3; i++)) gives Bash:i<3 and Bash:i++; @vlad-perevezentsev on shell.py: find -exec sed -i ... {} + and xargs sed -i give no file_path, so requires: edit reports a missing edit; not documented as a limit; @vlad-perevezentsev on consistency.py:257: pytest -q & and a for ...; do pytest; done loop are not masked, so last_ok: test passes; @vlad-perevezentsev on consistency.py:161: non-numeric output_tokens ("n/a", Infinity) raises and ends context consistency in a traceback; @vlad-perevezentsev on consistency.py:513/526: no-usage traces give output_tokens_cv 0.0; one-run strategies give inner_distance 0.0 and read as 'all stable success (valid diversity)'; @vlad-perevezentsev on consistency.py:359: a list pattern value is compiled via str(v) into a character class; command: [pytest, tox] matches git status.

[napetrov]

@antonwolfy the open list in your review at d2e8f51, mapped to commits (head now a8572f0):

• command -v pytest read as running pytest: 70da7a6, it is a Bash:command step.
• (cd build && make) 2>&1 | tail -5, (cd sub; pytest) > log.txt: 70da7a6, a word) label only inside case ... in.
• [[ ... && ... ]], for ((...; ...; ...)): 70da7a6, scanner tracks [[/]] and ((/)).
• find -exec sed -i / xargs sed -i: declined as a parser change, documented as a limit in docs/run-consistency.md (70da7a6); reason in the thread.
• pytest -q & and loop bodies not masked: 70da7a6.
• non-numeric output_tokens: 70da7a6, skipped.
• no-usage output_tokens_cv 0.0, one-run inner_distance 0.0 / verdict: 70da7a6, both null, "single run, passed" is not stable success.
• list pattern compiled as a character class: 70da7a6, joined into an alternation; non-strings refused.

Also merged intel/main (bcbb0d9) and raised benchmarks/spec.v1.yaml max_docs 32 -> 33 (a8572f0): main added answer-formats.md, so with run-consistency.md the docs count is 33; make benchmark-run passes.

[antonwolfy]

Review at a8572f0. The fixes since d2e8f51 hold: each point from the last round now gives the right step or value, and I found no crash in the new code. Two things are worth fixing before merge: the trailing-& fix covers only the command right before the &, and the new masking and bracket code has several lines no test pins.

What I ran, on Linux at a8572f0:

• make ci-local: 26 passed, 7 skipped, 2 failed. 9,214 tests passed. Changed-line coverage 96% (802 lines). The two failures are Install gitleaks (the download of the binary returned HTTP 504) and Scan commits, which needs that binary; neither is caused by the PR. gitleaks detect over origin/main..HEAD by hand reports no leaks.
• Reproductions for the code findings, with no model involved (none of this code calls one).
• A mutation run over the four test files for the new code: 78 one-line mutants, 35 survive, 15 of those are equivalent. The ones that matter are in the inline test comment.

Checked, no issues

• Delta: command -v pytest is a Bash:command step; (cd build && make) 2>&1 | tail -5 and (cd sub; pytest) > log.txt give make / pytest; [[ -n $x && -f y ]] && pytest and for ((i=0; i<3; i++)) give no fake step; command: [pytest, tox] no longer matches git status; output_tokens of "n/a", Infinity, NaN, true is skipped; a one-run strategy has inner_distance null and is not "valid diversity".
• Output: context consistency in text and --json runs on traces without usage and on one-run arms; null is printed, nothing crashes.
• Repo rules: the docs page is in the mkdocs.yml nav, the changelog fragments are present, test_growth_ratchet.py passes, no schema applies, no generated files are committed.

[xaleryb]

• Ran: the full suite, ruff, mypy and bandit at the head and with main merged, plus the earlier inputs on synthetic traces. Every earlier point of ours passes.
• { pytest -q; } & and if x; then pytest -q; fi &: Bash:pytest is not masked, so last_ok: test passes. Expected: "status hidden", as pytest -q & gets (the trailing-& thread on consistency.py:264).
• Pushed:
  • 7897750: pytest -q && echo PASS || echo FAIL and pytest -q && git diff; echo done hide pytest's status. The && form of the trailing-& thread is covered too.
  • 94c521b: set -e / pipefail used to be read from anywhere in the line: a heredoc body, a comment, an echo, a grep or a later set -e. Now only a set command counts, from where it runs, so pytest -q | tail -5 after a written set -euo pipefail reads as hidden again.
• Left for the author: masking a whole backgrounded group, and the test rows from the open thread on test_context_shell.py:166.

[antonwolfy]

Review at 94c521b. The && list forms from the last round are fixed: pytest -q && echo ok &, pytest -q && echo PASS || echo FAIL and the set -e / pipefail reading now give the right result. Two of my earlier threads are still open (below), so this is not ready to merge. One nit inline.

What I ran, on Linux at 94c521b:

• make ci-local: 26 passed, 2 failed, 7 skipped. 9,231 tests passed, 2 skipped. The changed-line coverage gate passed. The two failures are Install gitleaks (the proxy returned 504 on the download) and Scan commits, which needs that binary; the PR does not cause either. gitleaks detect over origin/main..HEAD by hand reports no leaks.
• Reproductions with _bash_events, no model involved (none of this code calls one).
• A mutation run over the four test files for the new code: 126 one-line mutants, 66 survive, 18 of those equivalent.

Still open from my earlier review:

• dokimi/dokimi/context/consistency.py:293: a trailing & after a group closer still hides only the closer. { pytest -q; } & gives Bash:pytest masked=False at both grains, and if x; then pytest -q; fi & gives Bash:pytest masked=False, though the shell reports 0 for both. last_ok: test passes on them. The && form (pytest -q && echo ok &) is fixed and pinned. Left: a } or fi / done directly before & must mask its whole group, with the two rows added to the last_ok test.
• dokimi/tests/test_context_shell.py:166: the delta pins the && / || list rules and set parsing from a heredoc, comment or echo, but these one-line mutants still pass all four test files (361 tests): _masks without loops -= ... done (for f in a; do pytest $f; done; make hides make); any -> all in the separator-clearing line (pytest -q; becomes hidden); _opens_loop without select, until, (, !, else or do (select x in a b; do pytest; done, (for f in a; do pytest; done), ! while read f; do pytest $f; done); _set_options with "e" in w -> w[1] == "e" (set -ue; pytest -q; echo done reads as hidden), without the set guard (grep -e foo x; pytest -q; echo done turns errexit on), and [a-zA-Z] -> [a-z] (set -Eeuo pipefail; pytest -q | tail -5; echo done); "\n" removed from the errexit check (set -e + newline); _list_hides with j = i + 1 or any -> all; bracket without ( in the start set (([[ -f a && -f b ]]) && pytest -q gives Bash:-f); command -V pytest and command pytest -v with the flag scan widened; _verdict(0, 1); the float token guard (9.0) and max(...) for a message id seen twice (5 then 9, 9 then 5).

Checked, no issues

• Delta: _list_hides and _set_options give the right result for pytest -q && echo ok || echo bad, pytest -q && git diff; echo done, cd x && pytest -q && git status (not hidden), set -euo pipefail, set -o errexit, set +e, and set words in a heredoc body, a comment, echo 'set -e' and grep -r pipefail.
• Repo rules: the changelog fragment keeps its ### Fixed form, test_growth_ratchet.py passes, no new doc page needs a nav entry, no generated files are committed, the new tests run offline.

[xaleryb]

• Ran: the full suite, ruff, mypy and bandit at 3d7e3ac with the commits below, plus every earlier input from this thread, re-run on synthetic traces. The earlier points pass, and the backgrounded-group fix from e8f0174 holds.
• Pushed:
  • 219e803: set -e; { pytest -q; echo ")"; } & and ... echo \); } & left pytest unmasked, so last_ok passed: a quoted or escaped ) cut the group short. Only unquoted, unescaped parens are counted now.
  • 6513416: (set -e; make); pytest -q; echo done and (set -o pipefail); pytest -q | tail -5 passed last_ok, because the option leaked out of the subshell. It now ends at the ).
  • 887b46a: adds rows for two lines that every test passed without: the set-only guard in _set_options (grep -e foo x, grep -o pipefail x) and until in _opens_loop.
• Nothing is left for the author.

[xaleryb]

Approved at 887b46a: every point from our review is fixed at this head, the full suite, ruff, mypy and bandit pass on Linux, and CI is green.

[xaleryb]

Merged main after #271 landed (d218906): the one conflict was max_docs in benchmarks/spec.v1.yaml, raised to 34 there for its two pages and to 33 here for run-consistency.md; with all three the bound is 35. Full suite, ruff, mypy and bandit pass on Linux.

[xaleryb]

Approved at d218906: merged with main after #271 (max_docs 35), the full suite, ruff, mypy and bandit pass on Linux, and CI is green.