[antonwolfy]

Summary

Three points from the review of #65 that were still open when it merged. Two are risks: the checklist contradicts §6, and the E2 figures read as stronger than their counts. One is a nit: the E1 comparison I asked for is missing. Only docs change.

Changes

• Checklist vs §6 (thread). §6 says to place review rules in the prompt, because E6 measured them there and AGENTS.md placement was not tested on seeded defects. The checklist still said "in the review section of AGENTS.md". It now says the prompt (§6), and that AGENTS.md placement is not measured yet.
• E2 (thread). §8 and the E2 row now say:
  • the build-break counts (5.9 to 0.4 per run) are pooled over the 3 renames, and no per-task count is stored, so one rename can carry the whole drop;
  • the pass rate moved by 2 runs of 9, which is p = 0.5 even in the best case (2 up, 0 down).
• E1 (thread). §1 gives both steps from the evidence page: installing the pinned formatter (5 to 7 of 16, 2 up, 0 down, p = 0.5), then naming it (7 to 12, 5 up, 0 down, p = 0.062). Neither alone is separated from noise at 16 cells, so the page no longer reads as if naming the command is the difference.

Risk

None to behaviour: two doc pages.

Testing

On a Linux review host, at 7b425a5 (on main 506f7ba):

• make ci-local: 29 passed, 0 failed, 7 skipped (the Docker catalog smoke, test-latest, the cli-surface packaging step, conda-forge-deps without micromamba, docs deploy, the pip-audit report, semgrep). 7,381 tests passed, 1 skipped.
• Rebased onto main 92d92d3 (test(llm): pin provider-call budgets for judge, panel, answer and validation paths #170, tests only) before opening; with the ten follow-up branches merged together, the full suite (7,447 passed, 1 skipped), the growth check, mypy and ruff pass.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[vchamarthi]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 29faac58920b, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 3 of 80 turns, 0.2 of 40 minutes, 36k in / 961 out tokens.

Docs-only change that fixes the checklist line to point at the review prompt (§6), consistent with §6's "place them in the prompt for now". It also adds the pooled-count and p = 0.5 caveats to the E2 text, and splits the E1 comparison into its install and naming steps. I checked the quoted sign-test p-values (2 up 0 down gives 0.5, 5 up 0 down gives 0.0625, 7 up 0 down gives 0.016), and they match the E1 evidence row. I found no defects.

No issues found.

[vchamarthi]

No blockers CI at the head is not green (0 failing, 15 pending).

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.77%. Comparing base (92d92d3) to head (29faac5).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #196      +/-   ##
==========================================
- Coverage   91.78%   91.77%   -0.02%     
==========================================
  Files         223      223              
  Lines       27623    27623              
  Branches     4549     4549              
==========================================
- Hits        25354    25350       -4     
- Misses       1606     1609       +3     
- Partials      663      664       +1     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

Checklist now matches §6 and the E1/E2 claims are stated at their real strength. LGTM.