[napetrov]

Summary

A user doc for repository owners: what to put in a repository and its environment so coding agents build, test and review it well, and how to measure each change with arms in repo run. Tools stay ordinary repository artifacts; nothing is declared for Dokimi. The page says which part of the measurement loop runs today (guidance arms) and which is planned (added-file/tool arms, #41 D7; paired comparison, #42).

 dokimi/docs/
 ├── repo-evaluation-contract.md
 ├── repo-evaluation-mergeability.md
 ├── repo-evaluation-agent-modes.md
+└── preparing-a-repository.md   # §1 runnable pinned commands · §2 one AGENTS.md line per tool
+                                #  §3 true guidance + traps · §4 complete compile DB · §5 navigation tools
+                                #  §6 learned review rules · §7 what each harness loads · §8 measure per model
 dokimi/docs/README.md            # index entry under "Repository evaluation"
 dokimi/mkdocs.yml                # nav entry

Every recommendation cites the measurement behind it, with the model it was measured on. Measurements are oneDAL and ScalableVectorSearch, Claude Code / Codex, Haiku and Sonnet 5.5 plus earlier generations, 8 to 24 cells per arm. Examples:

Change	Result
pinned formatter behind a named command	Haiku 4.5: CI format check 5/16 → 12/16 (7/16 with the binary installed but not named); Haiku 5.5 already 12/16 without it
MCP navigation tool, no guidance line → with one line	Haiku 5.5 / Sonnet 5.5: called in 2/24 and 1/24 → 13/24 and 16/24 runs
compiler call graph + line, "who calls X" questions	Sonnet 5.5 0.906 → 0.968 at half the cost; Haiku 5.5 0.874 → 0.951
curated review rules, seeded recurring defects	Codex recall 0.68 → 0.875; no effect on replayed PR-specific comments
fast compile check on cross-module renames	Haiku 4.5: build breaks 5.9 → 0.4 per run; Haiku and Sonnet 5.5 pass every run in both arms (§8)

Evidence

• Checks: pytest tests/test_docs_code_refs.py tests/test_docs_links.py tests/test_docs_hooks.py, where every dokimi repo … command and path in the page resolves against the CLI and tree: 52 passed. mkdocs build --strict passes, and pre-commit passes on the changed files.
• Numbers: each figure was re-read against the run data before this revision. The MCP-without-guidance figures were re-measured on Haiku 5.5 and Sonnet 5.5 for this PR: 2/24 and 1/24 runs used the tool, with no score change, against 13/24 and 16/24 with the guidance line.

Merge Danger

Door: two-way. Docs only; one new page plus an index and a nav line.

Blast Radius: docs

The numbers are small-sample, and §8 says to re-measure them on the reader's own tasks and models. The table of which files each harness loads (§7) is tied to the measured versions and points to repo reach instead of being normative.

Checklist

• Tests added/updated and pytest passes (docs tests)
• ruff, mypy, and schema checks pass (no code changed)
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed

🤖 Generated with Claude Code

[vchamarthi]

Thanks, this is a useful page! Good call to tie each recommendation to a measured before/after and to point readers at repo scan / repo reach / repo run instead of making the numbers normative. A few small things inline.

[vchamarthi]

The header says "version measured", but only the Claude Code row has a version. Were the Codex CLI and Copilot CLI rows measured with repo reach, or taken from vendor docs? Could we add the versions, or mark those rows as documented behaviour, so readers know how much to trust them?

[napetrov]

The intro now says which rows were measured (with the harness version) and which are vendor-documented; the §7 table labels the Codex and Copilot rows that way. Up to 46d67cd.

[napetrov]

No change in this pass: already handled in 154d52e. The Codex and Copilot rows were dropped from §7, which now lists only the measured Claude Code row, and the text says the other harnesses were not measured.

[vchamarthi]

What metric are 0.906 / 0.968 here, recall or F1 on the caller set? §4 and §6 name theirs, so naming it in the column header would keep it comparable.

[napetrov]

The §5 column headers now say "F1 of the caller-file set", and the text under the table gives 24 runs per arm (12 questions, 2 repetitions). Fixed in 8f2fc6c.

[napetrov]

No change in this pass: the metric is stated under the §5 table (F1 on the set of files that call the function, from precision and recall), with 24 runs per arm. The header itself still reads "F1".

[vchamarthi]

Is there somewhere we can link for the oneDAL / ScalableVectorSearch runs, like skill-novelty-evidence.md does for the novelty numbers? Even a short evidence section with task names and cell counts per claim would let someone re-run the comparison.

[napetrov]

Fixed in 154d52e. I kept only the figures that come from kept result files: the X7 numbers (tool called in 2/24 and 1/24 runs without a line, 13/24 and 16/24 with it; the F1 and cost table in §5). Each now has an inline line naming the task set, arms, models and cell count (12 questions x 2 reps = 24 runs per arm, Haiku 5.5 and Sonnet 5.5). The other figures existed only as report text, so they are now qualitative.

[napetrov]

Correction to my earlier reply: the per-cell results were kept after all. In 37de924 I recomputed each figure from them, using the model ID the trace reported. The page links each figure to a row of the new preparing-a-repository-evidence.md. Each row gives the task names, arms, model, cells per arm and raw counts, plus the sign-test p where the page cites one. That is enough to re-run the comparison with repo run and repo compare. The guidance-defect counts in §3 now cite uxlfoundation/oneDAL#3869. Two figures stay without a number because no stored result reproduces them: the share of callers missed by the CPU-only compile database, and the indexer agreement.

[napetrov]

No change in this pass: the evidence page is linked, and each figure has a row there (preparing-a-repository-evidence.md, added in 37de924).

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.12%. Comparing base (5634912) to head (c7b63a4).
⚠️ Report is 37 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #65      +/-   ##
==========================================
+ Coverage   89.82%   91.12%   +1.30%     
==========================================
  Files         205      212       +7     
  Lines       24038    26031    +1993     
  Branches     3858     4238     +380     
==========================================
+ Hits        21592    23721    +2129     
+ Misses       1752     1643     -109     
+ Partials      694      667      -27     

see 79 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

Checked the harness rows against the vendors' own pages and the CLI on a Linux host.

• §6 says to keep review rules in the root AGENTS.md "where every harness reads them". That is false for Claude Code whenever a CLAUDE.md exists (§7 says so itself) and for Gemini CLI by default (context.fileName).
• §7 and the checklist say to run repo reach for each harness you use, but only the claude-code profile is built in: --harness codex and --harness copilot fail with "no harness profile 'codex' (built-in: claude-code)".
• §3 says repo scan reports claims that no longer match the tree, beside defects like "26 of 31 algorithms have a CPU backend"; scan checks paths, commands, identifiers and links, not statements like those.
• The Codex override row is imprecise: AGENTS.override.md replaces the AGENTS.md in its own directory, not the root.

Pushed 8e683d8 with these four corrections (and dropped the "most transferable review knowledge" superlative). Still open from the earlier review: versions for the Codex and Copilot rows, the metric behind 0.906 / 0.968, and a link to the oneDAL and ScalableVectorSearch runs.

[antonwolfy]

Review at 8e683d8. The page is well organised and the CLI claims it makes are mostly right. The problems are what it leaves out about running the loop it describes, and evidence a reader cannot follow up. I'd fix the first three inline comments before merge.

What I ran, on a Linux review host at 8e683d8:

• make ci-local: 18 passed, 1 failed, 4 skipped. 4,294 tests passed. The one failure is schema-compat, which compares against a newer origin/main than the PR's merge base (tool_preference*.v1.json are on main, not on this branch); the PR touches no schema, so I don't count it.
• pytest tests/test_docs_code_refs.py tests/test_docs_links.py tests/test_docs_hooks.py: 52 passed. mkdocs build --strict: passes, the page is in nav: and the docs index.
• dokimi repo --help, repo run --help and repo reach --harness codex --dry-run, workspace.py, the claude-code profile and the contract, to check each claim about Dokimi.
• Two-sided Fisher exact tests on the four before/after proportions in §1.

Also worth fixing

1. §8 step 3 contradicts §3. Step 3 says "an arm in which the tool was never called measures the baseline again". §3 (line 83-85) reports recall of missed call sites going from 0.67 to 0.92 with a guidance line "without the tool it advertised ever being called". The guidance line alone moved the score there, so a reader who follows step 3 would discard an arm that helped. Suggestion: say that tool use tells you whether the tool mattered, and that a guidance line can change the score without it.

Checked, no issues

• Commands and flags: repo scan, repo reach, repo run and --harness exist. Only the claude-code profile is built in, and --harness takes a profile name or a harness_profile.v1 file. The contract link covers it.
• Arms: guidance:none, guidance:raw and guidance:bridge are the only arms repo run accepts (workspace.py:46). The page is right that nothing else exists yet.
• §7 Claude Code row: "AGENTS.md only when no CLAUDE.md is in the start-up chain" matches the profile note measured on 2.1.286.
• Links and nav: the four links to repo-evaluation-contract.md resolve in the built site; the nav and index entries are present.

[antonwolfy]

Review at 46d67cd. The last round fixed most of what I raised: the repo run prerequisites are in §8, the @AGENTS.md bridge now covers subdirectories, the p-values are in §1, and the planned-arm names are gone. Two things are still worth fixing before merge: the measurements still cannot be traced to a run, and the setup_patches route that replaces the old advice does not work as a with/without comparison as written.

What I ran, on a Linux review host at 46d67cd:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker/network, job conditions). 4,294 tests passed. The PR changes no code, so changed-line coverage has nothing to measure.
• mkdocs build --strict passes. Both new anchors (#workspace-preparation-done-by-the-framework, #9-running) resolve in the built contract page, and the nested lists in §1, §3, §6 and §8 render as intended.
• workspace.py (prepare_workspace, _apply_arm) and the claude-code profile, to check the §7 and §8 claims about the arms.
• Two-sided Fisher exact tests on the four proportions in §1 and the two tool-use counts in §2.

Still open from my earlier review:

• preparing-a-repository.md:20-24: no figure links to a run, PR or doc, and there is no evidence section. The contract (lines 618-619) names its runs and the PR; skill-novelty.md links skill-novelty-evidence.md. A short table of task set, arm, model and cell count per claim would do. Unanswered since the first round.
• preparing-a-repository.md:130-133: the §5 table still names no metric, and §6 line 153 does not say which gpt-6-luna was run (skill-novelty-evidence.md calls gpt-5.6-luna retired and gpt-6-luna a different model). "Current Claude models were already near that level" (line 154) has no number. (The "Sonnet 5" in §5 is a real model, so that point is withdrawn.)
• preparing-a-repository.md:164-166: the Codex and Copilot rows are now labelled vendor-documented, but still carry no vendor version or date.

Checked, no issues

• p-values (§1): 0.032, 0.149, 0.716 and 0.569 match a recomputation. The tool-use counts in §2 are well separated (2/24 vs 13/24 and 1/24 vs 16/24, both p < 0.002).
• repo run prerequisites (§8): the manifest keys, docker, mandatory --agent-network and the refusal of a guidance-only manifest match manifest.py, repo_run.py and repo run --help.
• repo reach (§7): --model is required, claude-code is the only built-in profile, and the cost note is in the repo help.
• Bridge advice (§7): a CLAUDE.md next to each AGENTS.md matches the profile's bridge definition and the contract note on 2.1.286.
• Repo rules: nav and index entries are present, no schema is touched, and no generated artifacts are committed.

One risk and two nits are inline.

[antonwolfy]

Review at fc539a5. The last round fixed the setup_patches steps, the bridge skip rule and the "only then" wording. One new problem comes from that fix: the status box still promises what step 2 now rules out. The measurements still cannot be traced to a run, which is the main reason this is not an approval.

What I ran, on a Linux review host at fc539a5:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker/network, job conditions). 4,294 tests passed. The PR changes no code, so changed-line coverage has nothing to measure.
• mkdocs build --strict passes. pytest tests/test_docs_code_refs.py tests/test_docs_links.py tests/test_docs_hooks.py: 52 passed. The page is in nav: and the docs index.
• prepare_workspace on a scratch repository with a patch that adds a line to AGENTS.md and an .mcp.json: guidance:raw keeps both, guidance:none deletes the AGENTS.md files, guidance:bridge keeps them and writes CLAUDE.md at the root and in sub/. This matches the new §8 step 2 text.

Still open from my earlier review:

• preparing-a-repository.md:20-24: no figure links to a run, PR or doc, and there is no evidence section. The contract (lines 618-619) names its runs and the PR that added the scan. Unanswered since the first round.
• preparing-a-repository.md:130-133: the §5 table still names no metric. gpt-6-luna (line 153) is not tied to a run, and skill-novelty-evidence.md calls gpt-5.6-luna retired and gpt-6-luna a different model. "Already near that level" (line 154) has no number.
• preparing-a-repository.md:164-166: the Codex and Copilot rows are labelled vendor-documented, but still carry no vendor version or date.

Checked, no issues

• Bridge description (§7): "where no CLAUDE.md exists yet" matches _apply_arm (workspace.py:255-257).
• §2 wording: "the scores improved in those arms (§5)" no longer claims the tool had to be called.
• §8 step 2: two tasks that differ in a setup_patches entry, both under guidance:raw, is the right route for one added line. --strict-mcp-config with no --mcp-config is in profiles/claude-code.yaml:21.
• CLI claims: --model is required on repo reach and repo run, claude-code is the only built-in profile, and run refuses a guidance-only manifest.

One risk and two nits are inline.

[antonwolfy]

Review at da42719. The only change since fc539a5 is the license header, which is fine. None of the points from my last review has been addressed, so my verdict stays the same: the measurements cannot be traced to a run, and the status box still promises something §8 step 2 rules out.

What I ran, on a Linux review host at da42719:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker/network, job conditions). 4,294 tests passed. The PR changes no code, so changed-line coverage has nothing to measure.
• mkdocs build --strict passes. The nested lists in §1, §3, §4 and §6 render as nested lists, and the license comment is not visible on the page. The page is in nav: and the docs index.
• prepare_workspace on a scratch repository with a setup_patches entry that adds a line to AGENTS.md: guidance:raw keeps the line, guidance:none deletes the file, guidance:bridge writes CLAUDE.md only where none exists. This matches §7 and §8 step 2.
• Two-sided Fisher exact tests on the §1 and §2 counts: 0.032, 0.716, 0.149 and 0.569 match the page; 2/24 vs 13/24 gives p = 0.0014 and 1/24 vs 16/24 gives p = 1e-5.

Still open from my earlier review:

• preparing-a-repository.md:25-29: no figure links to a run, PR or doc, and there is no evidence section. The contract (lines 618-619) names its runs and the PR that added the scan. Unanswered since the first round.
• preparing-a-repository.md:8-11 and :208-214: the status box still says setup_patches or a distinct base can test arms that add a file or a tool. Step 2 says an MCP server added by a patch is not loaded, and the image does not follow the task base, so a pinned formatter (§1) or a navigation-server binary (§2) is not installed by a patch either. The box also still says arms that add a file are planned, while guidance:bridge writes CLAUDE.md today.
• preparing-a-repository.md:135-160: the §5 table names no metric, gpt-6-luna (line 158) is not tied to a run, and "already near that level" (line 159) has no number.
• preparing-a-repository.md:169-171: the Codex and Copilot rows are labelled vendor-documented but carry no vendor version or date.
• preparing-a-repository.md:204-214: step 2 does not say the two tasks must differ only in the setup_patches entry, or that their rows are compared in dokimi repo report.
• preparing-a-repository.md:221-231: the checklist has no item for the CLAUDE.md import, and line 227 still says "bridge exceptions".

Checked, no issues

• Arithmetic: 59/190 is 31%, 26/31 and 20/31 are 84% and 65%, 1 to 2 of 44 is 2% to 5%. All match the text.
• Links: #workspace-preparation-done-by-the-framework and #9-running resolve in the built contract page.
• Claude Code facts: claude-code is the only built-in profile and has a bridge key, --strict-mcp-config is passed with no --mcp-config, and --model is required on repo reach and repo run.

[xaleryb]

Re-checked at 249595b: the status box, the setup_patches pairing in §8 step 2 and the checklist item now match workspace.py, repo report --help and the contract, every command the page names runs as written, and mkdocs build --strict passes. Still open from the earlier rounds: §5 names no metric for 0.906/0.968 and 0.874/0.951 while §3 and §6 report recall and §4 F1, so they cannot be compared; no figure links to a run or an evidence section; the Codex and Copilot rows say "vendor-documented" but carry no version or date.

[antonwolfy]

Review at 903c199. The last round fixed three of my threads: the status box now matches §8 step 2, step 2 says the two tasks differ only in the setup_patches entry and to compare their rows in repo report, and the checklist has the CLAUDE.md import item. What keeps this from an approval is the same point as in every round: the measurements still cannot be traced to a run. One new risk on step 2 is inline.

What I ran, on a Linux review host at 903c199:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker/network, job conditions). 5,011 tests passed. The PR changes no code, so changed-line coverage has nothing to measure.
• mkdocs build --strict passes. pytest tests/test_docs_code_refs.py tests/test_docs_links.py tests/test_docs_hooks.py: 55 passed. The page is in nav: and the docs index.
• prepare_workspace through the repo's fake runner on a scratch repository: a setup_patches entry that adds a line to AGENTS.md is kept by guidance:raw and guidance:bridge and deleted by guidance:none; a distinct base commit works; repo report prints one row per (task, arm, model). One image is built for all tasks. This matches §8 step 2.
• Two-sided Fisher exact tests on the §1 and §2 counts: 0.032, 0.149, 0.716 and 0.569 match the page. 59/190, 26/31, 20/31 and 1 to 2 of 44 match the text.

Still open from my earlier review:

• preparing-a-repository.md:26-30: no figure links to a run, PR or doc, and there is no evidence section. A short table of task set, arm, model and cell count per claim would do. Unanswered since the first round.
• preparing-a-repository.md:136-139 and :159-160: the §5 table names no metric (§3 and §6 say recall, §4 says F1). gpt-6-luna is not tied to a run, and skill-novelty-evidence.md calls gpt-5.6-luna retired and gpt-6-luna a different model. "Already near that level" has no number.
• preparing-a-repository.md:171-172: the Codex and Copilot rows are labelled vendor-documented but carry no vendor version or date.
• preparing-a-repository.md:141-142: "with Sonnet 5, agents did not call it" does not say whether the guidance line was present, and does not name the fix and review tasks.

Checked, no issues

• Status box and step 2: both now say a tool needs planned arms and a guidance line can be tested today with two setup_patches tasks under guidance:raw. The image and --strict-mcp-config limits match repo_run.py and profiles/claude-code.yaml.
• CLI claims: --model is required on repo reach and repo run, claude-code is the only built-in profile, --agent-network and the guidance-only refusal match repo run, and both contract anchors resolve in the built site.
• Cross-references: every §N on the page points at an existing heading. §6 now defers to §7 on which harness loads the root AGENTS.md.

[napetrov]

On the review-body points:

• antonwolfy point 2 (§5 metric, gpt-6-luna, "near that level"): Metric named, model string confirmed, numbers added (8f2fc6c).

• antonwolfy point 4 (guidance line present, tasks named): See thread 17 (8f2fc6c).

• xaleryb point 1 (§5 metric): See thread 1 (8f2fc6c).

[napetrov]

On the review-body points:

• antonwolfy point 1: Traceability (154d52e): the figures that remain are the X7 ones, each with task set, arms, model and cell count on the line below. The rest no longer carry a number. The metric (F1 of the caller-file set) is stated at §5, "gpt-6-luna" has no numbers, and "near that level" now says only that rules added little.

• antonwolfy point 2: Codex and Copilot rows are removed from the §7 table (154d52e), since no version or measurement backs them. A sentence now says other harnesses need their own harness_profile.v1 and a repo reach measurement, and the text-trace limit is kept. The "vendor-documented" intro text is gone.

[antonwolfy]

Review at 37de924. The last round closed all seven of my open threads: the figures now link to an evidence page, and the Codex/Copilot rows are gone. The evidence page overstates what it lets a reader do, and three claims on the main page are stronger than the counts behind them. I'd fix the four risks inline before merge.

What I ran, on a Linux review host at 37de924:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker/network, job conditions). The PR changes no code, so changed-line coverage has nothing to measure.
• pytest tests/test_docs_code_refs.py tests/test_docs_links.py tests/test_docs_hooks.py: 56 passed. mkdocs build --strict passes. Both pages are in nav: and the docs index.
• Sign-test recomputation of the evidence page: 7 up 0 down gives 0.016, 5/0 gives 0.062, 3/0 gives 0.25; 53/9 and 4/9 give 5.9 and 0.4 per run; the net counts match the raw counts in E1.
• ARMS in workspace.py, repo compare (compare.py) and the task names, to check what the evidence page says can be re-run.

Checked, no issues

• Arithmetic: the E1 to E7 cell counts match their task × repetition designs, and the E1b mixed denominator (12/14) is disclosed.
• Step 2 and §7: the repo compare pairing rule, the bridge_skipped note and the text-trace limit match compare.py, workspace.py and harness_profile.v1.json.
• Repo rules: no schema is touched, no generated artifacts are committed, and the new page has the license header.

[antonwolfy]

The advice to keep rules in the root AGENTS.md is not backed by the arm that showed the gain. In E6 the rules were in the prompt (rules arm). The rulesmd arm (rules in AGENTS.md) ran only on the real-PR replays in E7, where nothing moved. The Codex row was removed from §7, so the page also no longer says which harness loads the root AGENTS.md for the model that benefited. The 0.68 to 0.875 at line 171 therefore does not measure the layout this bullet recommends.

Suggestion: say the gain was measured with the rules in the prompt, and that AGENTS.md placement was not tested on seeded defects.

[napetrov]

Fixed in 48708bc. I changed the advice: moved it from 'keep them in root AGENTS.md' to 'place them in the prompt for now'. The reason: E6 measured rules in the prompt and showed the gain (0.68→0.875 for Codex); the rulesmd arm tested rules in AGENTS.md but ran only on real-PR replays (E7), where the effect was zero. AGENTS.md placement has not been tested on seeded defects. The change is now clear: 'Place them in the prompt for now. E6 measured rules in the prompt; the rulesmd arm tested rules in AGENTS.md but only on real-PR replays (E7), where the effect was zero anyway.'

[napetrov]

No change in this pass: already done in 48708bc. The advice is now "place them in the prompt for now", and the text says AGENTS.md placement was not tested on seeded defects.

[antonwolfy]

The E2 result is credited to "a fast compile check" that the arm did not isolate. The tools arm (evidence, E1 row) is named fmt and check commands plus one AGENTS.md line. The evidence page never says check is a compile check, and the formatter command is in the same arm. The page gives no p either: pass rate moved 4/9 to 6/9, and 53 against 4 new failing units is pooled over 3 tasks. A pooled count dominated by one rename would read the same way.

Suggestion: say "named fmt and check commands", add the per-task counts or a p, and note the change is not isolated to the compile check.

[napetrov]

Fixed in 48708bc. The text now clarifies: 'the tools arm (named fmt and check commands plus one AGENTS.md line, not isolated to compile checks) cut new build breaks...pass rate 4/9 to 6/9...The change includes the formatter, so the compile-check effect alone is not known.' This removes the misleading isolation and tells readers that the effect conflates two changes.

[napetrov]

No change in this pass: already done in 48708bc. The text says the tools arm (named fmt and check, plus one AGENTS.md line) is not isolated to compile checks.

[antonwolfy]

Nit: the earlier note that n was too small to separate "named command" from "binary installed, not named" is gone. The evidence gives fmtenv→tools 5 up, 0 down, p = 0.062 (E1). The page now reads as if naming the command is the difference. Restore a short "7/16 vs 12/16, p = 0.062, not separated at this n".

[napetrov]

Fixed in 48708bc, p-value corrected in b892fe6. The caveat is restored: installing the right formatter version without naming it reached 7/16 against 5/16, which at 16 cells is not separated from noise (2 up, 0 down, p = 0.5). The first revision attached p = 0.062, which belongs to fmtenv vs tools (5 up, 0 down), not to off vs fmtenv. I also added to the first bullet: 'Name the commands explicitly so agents find them.'

[napetrov]

No change in this pass: already done in b892fe6. The caveat is restored: 7/16 vs 5/16, p = 0.5, not separated at this n.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

1 open issue

• Line 174 contains an incorrect/impossible count (“5 of 4 tasks improved”). Based on E6’s… · New

What changed in this PR

Adds a new, measurement-backed documentation page to guide repository owners in preparing repos for coding agents, plus a companion evidence page and navigation/index updates.

Changes:

• Add preparing-a-repository.md guidance doc describing recommended repo artifacts and how to measure impact with Dokimi.
• Add preparing-a-repository-evidence.md documenting the underlying experimental arms, models, and results that support the guidance.
• Wire the new pages into MkDocs navigation and the docs README index.

File	Description
dokimi/​mkdocs.yml	Adds the two new docs pages to the MkDocs nav.
dokimi/​docs/​preparing-a-repository.md	New main guidance page for repo owners, referencing measured evidence.
dokimi/​docs/​preparing-a-repository-evidence.md	New evidence page listing the runs/arms behind the figures.
dokimi/​docs/​README.md	Adds the new pages to the docs index under repository evaluation.

_{Review effort: Lite}

[Copilot]

🔵 Needs a closer look

Repetition-level p-values overstate task-level evidence and need correction before the recommendations are published.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Please compute these p-values from per-task deltas rather than pairing repetition numbers across…

dokimi/​docs/​preparing-a-repository-evidence.md:27

Please compute these p-values from per-task deltas rather than pairing repetition numbers across arms. Repeated runs share task difficulty, and matching repetition numbers does not establish a pair; treating them as independent task evidence overstates confidence in the recommendations. Under the task-level method documented in contract §9 and dokimi/dokimi/repos/compare.py:6–8, E6's three improving tasks and no worsening tasks give p = 0.25, not 0.062; E3's two give p = 0.5, not 0.25. Recompute the affected statistics and update both pages, or omit the p-values until that analysis is available.

repos/report.py:summarize supplies Wilson intervals only for pass/fail rows; score-only rows…

dokimi/​docs/​preparing-a-repository.md:244

repos/report.py:summarize supplies Wilson intervals only for pass/fail rows; score-only rows contain a mean without an interval. Readers using the two-task workaround for review recall therefore cannot apply this decision rule. Limit the Wilson guidance to pass/fail tasks and explain that score-only comparisons require separate uncertainty analysis.

This checklist contradicts §6, which recommends putting review rules in the prompt because the…

dokimi/​docs/​preparing-a-repository.md:264

This checklist contradicts §6, which recommends putting review rules in the prompt because the seeded-defect improvement was measured only with that placement. Align the checklist with §6 so readers reproduce the measured setup rather than an untested AGENTS.md placement.

1 resolved since last review

• Line 174 contains an incorrect/impossible count (“5 of 4 tasks improved”). Based on E6’s…

_{Review effort: Balanced}

[antonwolfy]

Review at 77bb5cf. The last round fixed most of what I raised: the evidence page now says which arms re-run, E4 has its inputs, the E3 and E6 counts carry p-values, and the intro matches the evidence. One risk from the AGENTS.md thread is still open, and the E2 sentence and the E1 comparison are still short of what I asked for.

What I ran, on a Linux review host at 77bb5cf:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,012 tests passed. The PR changes one test line and no code, so changed-line coverage has nothing to measure.
• tests/test_docs_links.py, test_docs_code_refs.py and test_docs_hooks.py: 56 passed. mkdocs build --strict passes and the built evidence page has id="e1" to id="e8". Mutation check: pointing a link at #e44 fails test_every_anchor_link_points_at_a_real_heading, and deleting the new _slugs line fails it too.
• Sign-test recomputation (two-sided, exact): 7/0 gives 0.016, 5/0 gives 0.062, 3/0 gives 0.25, 2/0 gives 0.5. All match the pages. The E3, E4 and E6 figures on the main page match the evidence rows.

I agree with @copilot-pull-request-reviewer's points on the cell-level p-values, the Wilson-interval rule in step 2 and the checklist line. New evidence: repos/report.py:72-77 gives ci95 only to pass/fail rows; a score-only row (the recall tasks in §3 and §6) prints a mean and no interval, so the step 2 rule has nothing to read there.

Still open from my earlier review:

• preparing-a-repository.md:264: the checklist still says to keep the review rules in the review section of AGENTS.md. §6 (line 167) now says to place them in the prompt, because only that placement was measured. Align the checklist with §6.
• preparing-a-repository.md:217-220: the sentence names fmt and check and says the effect is not isolated, but it still gives no p or per-task counts for 5.9 to 0.4 and 4/9 to 6/9. The counts are pooled over 3 tasks.
• preparing-a-repository.md:42-46: the 7 vs 5 comparison was added, but the one I asked for is not there. fmtenv→tools (7/16 to 12/16) is 5 up, 0 down, p = 0.062 (E1), and the page still reads as if naming the command is the difference.

Checked, no issues

• Anchors: every #e… link on the main page lands on a row, in the test and in the built site.
• Arms: the evidence page says only guidance:none, guidance:raw and guidance:bridge re-run, and ARMS at workspace.py:47 agrees.
• Bridge and text trace (§7): the skip rule, bridge_skipped in meta.json and the text-trace limit match workspace.py, runner.py:159, events.py:64 and harness_profile.v1.json:34.
• Repo rules: both pages are in mkdocs.yml nav: and the docs index, no schema is touched, and no generated artifacts are committed.

[antonwolfy]

Review at 5029809. The only change since 77bb5cf is that "results did not change" is gone from the E5 sentence in §5. That closes my nit there, and the remaining "0 of 20" count matches E5. Two of the three threads I left open are risks that are still unfixed. Two new nits are inline.

What I ran, on a Linux review host at 5029809:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,012 tests passed. The PR changes one test line and no code, so changed-line coverage has nothing to measure.
• tests/test_docs_links.py, test_docs_code_refs.py and test_docs_hooks.py: 56 passed. mkdocs build --strict passes. Mutation check on the anchor test: renaming #e4 to #e44, linking a missing #e9, deleting the <a id="e1b"> cell, an extra space inside an anchor and moving an anchor into a code fence each fail the suite (1 failed, 55 passed).
• Sign-test recomputation (two-sided, exact): 7/0 gives 0.016, 5/0 gives 0.062, 3/0 gives 0.25, 2/0 gives 0.5. Every count, rate, cost and cell total on the main page matches its E-row.

I agree with @copilot-pull-request-reviewer's point on the cell-level p-values and with my earlier note on the step 2 Wilson rule. New evidence: at this head §1 (line 43) still gives p = 0.016 with no mention that it is a cell-level sign test, and repos/report.py:72-77 gives ci95 only to pass/fail rows, so line 241-243 has nothing to read on score-only rows.

Still open from my earlier review:

• preparing-a-repository.md:263: the checklist still says to keep the review rules in the review section of AGENTS.md. §6 (line 166) says to place them in the prompt, because only that placement was measured on seeded defects.
• preparing-a-repository.md:216-219: the sentence now says the effect is not isolated, but 5.9 to 0.4 per run and 4/9 to 6/9 are still pooled over 3 tasks with no per-task counts or p.
• preparing-a-repository.md:42-46: the 7 vs 5 comparison is there, but fmtenv→tools (7/16 to 12/16: 5 up, 0 down, p = 0.062, E1) is not. The text still reads as if naming the command is the difference.

Checked, no issues

• §7 and §8 against the code: the three guidance arms, bridge_skipped, setup_patches running before the arm, --strict-mcp-config with no --mcp-config, the text trace limit, repo compare pairing by task id and the required --model and --agent-network all match workspace.py, report.py, compare.py and the claude-code profile.
• Repo rules: both pages are in mkdocs.yml nav: and the docs index, both have the license header, no schema is touched and no generated artifacts are committed.

[antonwolfy]

Nit: "helped in the same way" is not what E6 and E7 show. E6 has lint above off for Sonnet 5 (0.875 vs 0.750), Haiku 4.5 (0.844 vs 0.781) and Haiku 5.5 (0.875 vs 0.844), and below it for Codex (0.604 vs 0.677). E7 says no effect on real-PR replays (Sonnet 5 0.19 to 0.22, Haiku 4.5 0.13 to 0.14). The sentence has no link, and "distracted one weaker model" does not say it was Codex, whose gain was rules, not lint.

Suggestion: limit it to seeded defects, name the model that lost recall, and link [E6] and [E7].

[napetrov]

Fixed in c7b63a4: the lint sentence is now limited to seeded defects (E6), names Codex as the model that lost recall (0.677 to 0.604, its gain came from rules), and links E6 and E7.

[antonwolfy]

Nit: "16 files hold both APIs' error and pointer types" has no row in the evidence page. E8 (preparing-a-repository-evidence.md:42) lists the 46 backend files, the 26 of 31 and 20 of 31 counts and the missing script, but not this 16. The intro (line 17-19) says a figure with no stored result is given without a number.

Suggestion: add the 16 to E8, or drop the number here.

[napetrov]

Fixed in c7b63a4: E8 has no row for 16, so the number is dropped (now 'some files').

[vchamarthi]

Approved. 1 non-blocking comment.

All three earlier points are fixed: the section 7 table now lists only the measured Claude Code row, section 5 names F1 on the caller-file set, and every figure links to a row on the new evidence page. One nit on that page's harness line.

Compared each figure on the main page with its E1-E8 row and recomputed the two-sided sign-test p-values (7/0 -> 0.016, 5/0 -> 0.062, 3/0 -> 0.25, 2/0 -> 0.5); all match.

CI at send time: 0 failing, 0 pending.

[vchamarthi]

Nit: "except E6, which used the Codex CLI" reads as if all of E6 ran on Codex, but the E6 row also has four Claude models; say "except the Codex row of E6".

[napetrov]

Fixed in c7b63a4: now 'except the Codex row of E6'.