feat(repo): guidance:rev:<rev> and patch:<file> arms (D7)

[napetrov]

Part of #41 (step D7).

Summary

Two arms beyond guidance on/off. guidance:rev:<rev> answers "does this PR's AGENTS.md help?" by running the same tasks with another revision's guidance, which needed generated task twins until now. patch:<file> makes a tooling or layout change (a new script, a Makefile target) a treatment.

 repo run --arm A,B
-  A in ARMS or refuse
+  parse_arm(A)                       # before the image build: a typo or an unreadable patch fails now
+    patch:<file>   -> read the bytes once, name patch:<file name>@<sha256[:12]>
   _context(pin=True)                 # the batch keeps its eval tree
+  pin_arm(A, git_dir)                # once per batch
+    guidance:rev:<rev> -> guidance:rev:<full sha>   (a remote source fetches the branch or tag first)
   run_batch -> prepare_workspace
     1-4  base tree, drop eval_dir, init, setup_patches
     5    arm
+         rev:   delete the base's guidance files, check out the globs' files at <sha> through a
+                throwaway index (outside eval_dir), as step 1 does
+         patch: git apply on top of guidance:raw; changed files into arm_files
     6-8  amend, review_patch, expire reflog and prune

The arm string names the content, and it is what cells, result.json, meta.json and repo compare see:

given	arm (cell name part)	meta.json extra
guidance:rev:main	guidance:rev:<40 hex> (guidance-rev-<sha>)	arm_rev: {rev, sha, removed}
patch:tools/make-check.diff	patch:make-check.diff@<12 hex> (patch-make-check.diff_<12 hex>)	arm_patch: {path, sha256}

Why this grammar:

• The SHA is in the arm string. repo run pins a batch to its eval tree, not to its arms. With the SHA in the name, a branch that moved before a later invocation gives a new arm with its own cells. Neither pin nor arm changes silently. Two spellings of one commit (rev:main and rev:<sha>) run once.
• No / and no , in a name. repo compare splits BATCH/ARM on the last /, and --arm splits on ,. The patch arm keeps only the file name (reduced to [A-Za-z0-9.-], at most one _ in a row, since cell names join their parts with __) plus a digest prefix. The full host path is in meta.json.
• The patch comes from a host path given on the CLI. It is the evaluator's treatment, not the repository's, so it doesn't belong in the eval tree, and making it a task file would need a twin of each task again. The file is read once per invocation, so every cell applies the same bytes even if the file is edited mid-batch. Its sha256 sits in the arm name and in meta.json.

Isolation: both arms write files and copy no git objects. The workspace's object store gains only the blobs of the files the arm wrote, and step 8 drops the guidance the arm replaced. guidance:rev: and patch: don't combine (out of scope). runner.py is unchanged: Arm is a str, so it goes through run_batch to prepare_workspace as an arm string always has.

Evidence

New tests/test_repo_arms.py (15 tests, no docker). On main the module fails at import (parse_arm doesn't exist), and --arm guidance:rev:... is refused as an unknown arm.

test_rev_arm_copies_no_object_of_the_revision_but_the_guidance_blobs_it_uses
  rev commit: new AGENTS.md, src/AGENTS.md added, sub/AGENTS.md gone, src/calc.py changed,
              secret.py and .agent-eval/AGENTS.md added
  for fix_add (setup patch) and review_add (git am):
    workspace objects ∩ (objects of rev - objects of base) == {the 2 guidance blobs at rev}
    fsck --unreachable == ""; no "LEAKED", "pr fix", "eval guidance", or the base's old guidance bytes

To check that the test bites, I added a git fetch <source> <sha>:refs/heads/leak to _apply_rev, and the test failed (held & only_at_pr == guidance got the whole commit).

Also covered:

• the moved branch: a pinned arm keeps its commit, re-pinning gives the new SHA, and repo run adds a new cell while the old one stays
• feature/x refs
• bad revisions (-p, a..b, missing)
• the patch applying on raw, with bytes read once
• a stale patch, and one writing into eval_dir, fail as PrepError for the cell: task fix_add: arm patch:bad.diff@…: patch …/bad.diff does not apply: …
• an unreadable patch is refused before the image build
• parse_selector("b1/<arm>") round-trips
• repo compare over a batch with raw, rev and patch arms

Local: pytest 5193 passed, 2 skipped. ruff, mypy and schema_check are clean, and make ci-local shows 19 passed, 0 failed. Radon CC: _apply_arm is 13, the new functions are 10 or less.

Merge Danger

Door: two-way. The new arms are opt-in, and the existing three behave as before (same cell names, same meta.json apart from the new optional keys).

Blast Radius: small.

• meta.json can carry arm_rev or arm_patch.
• A cell name can carry a 40- or 64-hex SHA (container names are already cut to 90 characters).
• repo run now runs a repeated --arm once. Before, the arm was planned twice, and with -j above 1 both copies prepared the same cell at once.
• Contract §3 "Workspace preparation" documents both arms.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 90.44944% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.29%. Comparing base (6e695ce) to head (70420e8).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/workspace.py	89.37%	8 Missing and 9 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #117      +/-   ##
==========================================
+ Coverage   90.24%   90.29%   +0.05%     
==========================================
  Files         206      206              
  Lines       24289    24509     +220     
  Branches     3906     3953      +47     
==========================================
+ Hits        21920    22131     +211     
- Misses       1703     1707       +4     
- Partials      666      671       +5     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	90.47% <100.00%> (+1.40%)	⬆️
dokimi/dokimi/repos/workspace.py	86.64% <89.37%> (+3.08%)	⬆️

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Ignored patch files can bypass eval-directory isolation, and symlinked revision guidance can retain base content.

2 open issues

• Ignored files bypass eval_dir content validation · New
• Resolve guidance symlinks against the pinned revision · New

What changed in this PR

Adds revision-guidance and host-patch treatments to repository evaluation, implementing D7 of #41 without duplicating tasks.

Changes:

• Parses, pins, and deduplicates arms using content-based names.
• Applies treatments during workspace preparation and records their metadata.
• Adds tests, contract documentation, and a changelog entry.

File	Description
dokimi/​tests/​test_repo_arms.py	Tests preparation, naming, isolation, and comparison.
dokimi/​dokimi/​repos/​workspace.py	Parses, pins, and applies new arms.
dokimi/​dokimi/​commands/​repo_run.py	Integrates arm handling into the CLI.
dokimi/​docs/​repo-evaluation-contract.md	Documents treatment behavior and metadata.
dokimi/​changelog.d/​20261018_120000_napetrov_repo-arms-rev-patch.md	Announces the new arms.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Matched guidance symlinks bypass target resolution, leaving stale or missing guidance in prepared workspaces.

1 open issue

• Regression test does not exercise ignored-file detection · New

1 previously missed issue

In code that hasn't changed since last review

Matched guidance symlinks bypass resolution and retain stale content

dokimi/​dokimi/​repos/​workspace.py:395

The new symlink-resolution loop is unreachable for matched guidance links. Git represents symlinks as blobs, so matched links enter found and bypass this elif; links collected here are absent from found and skipped by the loop. For AGENTS.md -> docs/GUIDE.md, preparation still exports only the link, leaving stale guidance from the base or a dangling link. Collect matched symlinks independently, validate their targets, and materialize the pinned guidance without replacing unrelated base files. Update the regression test to check the prepared guidance bytes: its current or assertion passes when only the link is returned.

2 resolved since last review

• Ignored files bypass eval_dir content validation
• Resolve guidance symlinks against the pinned revision

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Host-write safety and guidance revision correctness remain unresolved.

2 open issues

• Reject symlinked destination components before writing guidance · New
• Validate terminal mode for cyclic guidance symlinks · New

1 previously missed issue

In code that hasn't changed since last review

Resolve linked-worktree revisions from the source checkout

dokimi/​dokimi/​commands/​repo_run.py:218

For a local linked-worktree source, ctx.git_dir is the common Git directory (source_git_dir, workspace.py:203–206). Resolving guidance:rev:HEAD there uses the main worktree's HEAD rather than the supplied worktree's HEAD, silently measuring the wrong guidance when they differ. Resolve local arm revisions from the source checkout; keep the common directory for reading the pinned commit's objects. Add a regression with different HEADs in the two worktrees.

1 resolved since last review

• Regression test does not exercise ignored-file detection

_{Review effort: Balanced}

[antonwolfy]

Review at 7944206. The arm grammar, the SHA-pinned names and the eval_dir guard on the patch arm hold up. The problems are in the link-resolving branch of guidance:rev:, which still has four defects, and in two untested promises. I'd fix the bugs before merge.

What I ran, on a Linux review host at 7944206:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker, semgrep, deploy, test-latest). 5199 tests, changed-line coverage 96%.
• Reproductions for the code findings, with the LLM transport mocked.
• A mutation check on _pin_arms: two mutants pass the arm, runner and compare tests (inline at repo_run.py:223).

I agree with @copilot-pull-request-reviewer's three open points (symlinked destination, cyclic link, linked-worktree HEAD); I reproduced each and put the evidence inline.

Checked, no issues

• eval_dir guard on patch:: rename, copy, empty file, binary patch, a symlink at .agent-eval/l, a quoted path with a tab, and a .gitignored file are all refused as "writes into eval_dir". .git/ paths and src/../.agent-eval/x are rejected by git apply.
• Names: guidance:rev:HEAD~1, tags and pr^ resolve to full SHAs. -x, a,b, a..b, a:b are refused. parse_selector round-trips the patch arm name.
• Docs and rules: cell-name forms match the contract. meta.json has no schema in the tree, so arm_rev / arm_patch need no SCHEMA_VERSIONS change. The changelog fragment and nav are in order. On the PR's base the new tests fail at import (cannot import name 'parse_arm').

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

6 open issues

• Reject symlinked destination components before writing guidance
• Deduplication via dict.fromkeys(pinned) is based on the pinned arm string value. For patch:*,… · New
• For remote sources, pinning any non-full-SHA guidance:rev:<rev> currently triggers a full fetch… · New
• When materializing a symlinked guidance file’s target as a regular file, the code writes bytes but… · New
• arm_files for guidance:rev:* appears to be the returned written list only, while the contract… · New
• Validate terminal mode for cyclic guidance symlinks

_{Review effort: Lite}

[napetrov]

No change in this PR, and leaving the thread open for a decision. A rev can name a branch or a tag, and a targeted refspec needs a lookup first (ls-remote), an extra round trip per remote source. Pinning runs once per batch, so the full head and tag fetch happens at most once per source. Open to a follow-up if large remotes make it slow.

[Copilot]

🔵 Needs a closer look

Truncated patch names can introduce extra cell separators and cause reports to undercount missing repetitions.

4 open issues

• arm_files for guidance:rev:* appears to be the returned written list only, while the contract…
• When materializing a symlinked guidance file’s target as a regular file, the code writes bytes but…
• For remote sources, pinning any non-full-SHA guidance:rev:<rev> currently triggers a full fetch…
• Deduplication via dict.fromkeys(pinned) is based on the pinned arm string value. For patch:*,…

1 previously missed issue

In code that hasn't changed since last review

Trailing underscore breaks missing-cell parsing and accounting

dokimi/​dokimi/​repos/​workspace.py:123

Truncation can leave a trailing underscore: a basename of 39 a characters followed by _more.diff becomes ..._@<digest>. cell_name() converts @ to another underscore, creating an extra __ separator. report.summarize() then splits missing cell names into five fields instead of four and omits them from the row's missing-repetition count. Strip trailing underscores after truncation and add a regression covering missing-cell accounting.

2 resolved since last review

• Reject symlinked destination components before writing guidance
• Validate terminal mode for cyclic guidance symlinks

_{Review effort: Balanced}

[antonwolfy]

Review at 70420e8. Five of the six threads are fixed and I checked each. The linked-guidance file still loses its mode, and the new code has two crash paths and one test that does not test its guard. I'd fix these before merge.

What I ran, on a Linux review host at 70420e8:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker, semgrep, deploy, test-latest). 5207 tests passed, 1 skipped.
• Reproductions for the code findings, with the LLM transport mocked.
• A mutation run on _pin_arms, _apply_rev, _guidance_at, _apply_patch and parse_arm against test_repo_arms.py, test_repo_runner.py and test_repo_compare.py. return pinned, fetch=False, worktree_dir=None, the [:40] cap, the set(removed) - set(written) difference, the | touched union, the out-of-tree and into-eval_dir link checks and write_text decoding are all killed. Survivors are in the inline comment on the symlink test, plus dropping -- in checkout-index, dropping or ".." in rev, the empty-patch check, --whitespace=nowarn and the rename branch of _patch_paths.

Still open from my earlier review:

• workspace.py:387: the linked guidance file's bytes are now exact, but its mode is not. write_bytes creates it as 0o664. Same revision, tools/G.md committed as 100755, AGENTS.md a link to it and src/AGENTS.md a direct 100755 file: the workspace gets AGENTS.md 0o664 and src/AGENTS.md 0o775, and the snapshot index shows 100644 against 100755. The thread named the mode. test_rev_arm_preserves_linked_file_bytes_and_mode (test_repo_arms.py:323) checks no mode, so the name overclaims. Read the mode from the ls-tree entry of the target and chmod the file, and assert it in that test.

I agree with @copilot-pull-request-reviewer on arm_files and on the trailing underscore. New evidence: guidance:rev: omits a base sub/AGENTS.md the revision lacks (removed has it, arm_files does not), and test_repo_arms.py:117 pins that against repo-evaluation-contract.md:296. A 39-character name plus _more.diff gives a cell id with five __ parts, and report.summarize then shows missing 0 for a row with 2 cells missing.

Nit. README.md:50 (outside this diff) still lists only guidance:none|raw|bridge for the repository arms.

Checked, no issues

• Cyclic links: self-link and two-link cycle both raise PrepError; a chain out of the tree or into eval_dir is refused.
• Symlinked destination: a base docs -> <outside dir> with the revision's linked docs/AGENTS.md is refused with nothing written outside the workspace; the plain-file path replaces the link.
• Revision names: HEAD^, HEAD~1, refs/heads/w, annotated tags resolve to full SHAs; w^{commit}, w@{1}, a/b/../c, w:AGENTS.md, ^w and a tag of a tree are refused with a PrepError.
• eval_dir guard on patch:: mode-only change, delete, rename into and out of it, a write through a symlink and .git/ or src/../evals paths are all refused.
• Docs and rules: cell-name forms and meta.json keys match the contract; mkdocs.yml needs no new entry; the changelog fragment and the SPDX header are in order; the new tests use local repos only.