[napetrov]

Summary

A docs page now names the code it documents, and a PR that changes that code without touching the page gets the page listed for review.

dokimi/docs/skill-novelty.md
  ---
  depends_on:                       <- new: YAML front matter, globs from the repo root
    - dokimi/dokimi/novelty/**
    - dokimi/dokimi/commands/novelty.py
  ---
  <!-- license header -->           <- unchanged, now below the front matter

PR changes dokimi/dokimi/novelty/probes.py, not the page
  -> docs-review-triggers job: table in the job summary
                              + ::warning file=dokimi/docs/skill-novelty.md::
  -> exit 0 (advisory: a stale page never fails the job)
  -> malformed depends_on: ::error file=<page>:: and exit 1
  -> bad --base (not diffable): exit 2

• scripts/check_docs_review_triggers.py --base --head: diff, match against globs, skip pages the PR touched. ** crosses /; * and ? stay inside one segment.
• .github/workflows/docs-review-triggers.yml: pull_request, contents: read, actions pinned by SHA, no PR comments, so fork PRs work.
• tests/test_docs_review_triggers.py (runs in the normal test job): every glob must match a tracked file. A glob that matches nothing means the mapping is stale, and the test fails.
• 15 pages seeded. I only listed paths a page actually describes. I left out broad overview pages (README, architecture, quickstart, user-flow, benchmarking), the two PROPOSED repo-evaluation pages, the evidence study and the difficulty rubric.
• scripts/docs/hooks.py: the list-nesting check rendered the raw file as GitHub would, so it treated the front matter list as page list items. It now skips the front matter, which GitHub shows as a table and mkdocs drops.
• Why front matter and not an HTML comment: mkdocs already strips it into page meta, and .github/.licenserc.yaml already expects the header after front matter. The cost is a small depends_on table at the top of the page on GitHub.

Evidence

• On a throwaway commit that touched novelty/probes.py, tool_preference.py and docs/tool-preference.md, the script flagged skill-novelty.md and skill-novelty-walkthrough.md. It did not flag tool-preference.md, because that page was touched in the same commit.
• Before the hook fix, mkdocs build --strict failed with 15 warnings (list item 1 on GitHub, 'dokimi/dokimi/novelty/**', is not a list item on the site). With the fix it passes, and test_front_matter_list_is_not_a_page_list covers the case.
• New tests: glob matching (9 cases), untouched page flagged, touched page not flagged, unrelated change, stale glob reported, front matter parsing, step summary and warning output, every repo glob matches a tracked file.
• make ci-local: 0 failed, 20 passed, 4 skipped (docker, main-only and semgrep). That covers pytest (5195 passed), zizmor (no findings), pre-commit, and the new job's run on this PR ("No docs page depends on a file this PR changes...").
• The skywalking-eyes license header check, run locally, reports none of the changed pages.

Merge Danger

Door: two-way. A new advisory workflow plus front matter that can be deleted with no trace in any artifact.

Blast Radius: contributors. A PR that moves or deletes code a page lists now fails test_every_depends_on_glob_matches_a_tracked_file until the glob is updated. The review job only warns. There are no package, CLI, schema or score changes, so this PR needs no changelog fragment.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.38%. Comparing base (12c4173) to head (1e258d5).
⚠️ Report is 4 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #92      +/-   ##
==========================================
+ Coverage   91.29%   91.38%   +0.08%     
==========================================
  Files         216      216              
  Lines       26565    26722     +157     
  Branches     4337     4373      +36     
==========================================
+ Hits        24252    24419     +167     
+ Misses       1645     1637       -8     
+ Partials      668      666       -2     

see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

The changes are narrowly scoped, have targeted regression coverage, and have no identified blocking issues.

No open issues 🎉

What changed in this PR

Adds documentation review reminders when a PR changes code referenced by an untouched docs page, without changing package behavior.

Changes:

• Adds dependency matching and an advisory workflow that reports pages to review.
• Seeds depends_on mappings on 15 pages and documents their use.
• Excludes front matter from list checks and adds regression and stale-mapping tests.

File	Description
dokimi/​tests/​test_docs_review_triggers.py	Tests matching, reporting, parsing, and mapping validity.
dokimi/​tests/​test_docs_hooks.py	Tests front-matter exclusion.
dokimi/​scripts/​docs/​hooks.py	Skips front matter during list comparison.
dokimi/​scripts/​check_docs_review_triggers.py	Finds and reports potentially stale pages.
dokimi/​docs/​tool-preference.md	Maps tool-preference dependencies.
dokimi/​docs/​skill-novelty.md	Maps novelty dependencies.
dokimi/​docs/​skill-novelty-walkthrough.md	Maps walkthrough dependencies.
dokimi/​docs/​repo-evaluation-contract.md	Maps repository-evaluation dependencies.
dokimi/​docs/​releasing.md	Maps release dependencies.
dokimi/​docs/​methodology.md	Maps evaluation-methodology dependencies.
dokimi/​docs/​ingestion-docling.md	Maps ingestion dependencies.
dokimi/​docs/​external-content.md	Maps external-content dependencies.
dokimi/​docs/​example-execution.md	Maps example-runner dependencies.
dokimi/​docs/​evaluating-treatments.md	Maps treatment dependencies.
dokimi/​docs/​contributing-terminal-bench-task.md	Maps task-harness dependencies.
dokimi/​docs/​contributing-metric.md	Maps metric and runner dependencies.
dokimi/​docs/​context-window.md	Maps context-window dependencies.
dokimi/​docs/​coding-harnesses.md	Maps harness dependencies.
dokimi/​docs/​adding-doc-source.md	Maps doc-source dependencies.
dokimi/​CONTRIBUTING.md	Explains dependency mappings and checks.
.github/​workflows/​docs-review-triggers.yml	Runs advisory reporting on pull requests.

_{Review effort: Balanced}

[antonwolfy]

Review at b89a033. The glob matcher and the skip-touched-page logic are right, and the hooks.py fix is the right form. Three things are worth fixing before merge, all in check_docs_review_triggers.py.

What I ran, on a Linux review host at b89a033:

• make ci-local: 20 passed, 0 failed, 4 skipped (docker, main-only, semgrep); 5196 tests passed.
• Reproductions for the findings below, run offline against the script and the tests.
• Mutation run on the new code: 66 mutants, 38 of which pass the new tests. The three that matter are in the comments below. The rest are mostly the wording of the summary and warning text.

Checked, no issues

• Glob matching: *, ?, ** and . behave as the docstring says; the nine parametrized cases kill the mutants that make * cross / or leave . unescaped.
• Workflow: pull_request, contents: read, SHA-pinned actions, BASE_REF passed through env, fetch-depth: 0 so origin/$BASE_REF exists. Nothing is posted, so a fork PR works.
• Docs build: mkdocs build --strict passes with the front matter, and the docs link and code-ref tests pass.
• Seeded globs: every glob matches a tracked file, and none is broad enough to match everything.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues

• The front matter regex requires a newline after the closing ---. If a page ends immediately after… · New
• load_pages() only scans dokimi/docs/*.md (non-recursive). If docs pages are ever organized into… · New
• stale_globs() recompiles the regex for each glob once per tracked file (glob_regex(g) is called… · New
• _FRONT_MATTER only matches LF line endings and doesn’t allow a UTF-8 BOM, while the new docs… · New
• _FRONT_MATTER only matches LF line endings and doesn’t allow a UTF-8 BOM, while the new docs… · New

_{Review effort: Lite}

[napetrov]

Fixed in a28522f: hooks.py regex updated to match check_docs_review_triggers.py — now accepts optional BOM, flexible line endings (CRLF/LF), and handles files ending without a final newline.

[napetrov]

Already in a28522f (regex matches check_docs_review_triggers.py); no further change in this round.

[napetrov]

Fixed in a28522f: same fix as thread above (lines 112-117 use the same regex).

[napetrov]

Same as the thread above: lines 112-117 already use the a28522f regex. No change here.

[Copilot]

🟢 Approval recommended

The changes are limited to documentation tooling and CI, with focused regression coverage and no blocking issues identified in static review.

5 open issues

• _FRONT_MATTER only matches LF line endings and doesn’t allow a UTF-8 BOM, while the new docs…
• _FRONT_MATTER only matches LF line endings and doesn’t allow a UTF-8 BOM, while the new docs…
• stale_globs() recompiles the regex for each glob once per tracked file (glob_regex(g) is called…
• load_pages() only scans dokimi/docs/*.md (non-recursive). If docs pages are ever organized into…
• The front matter regex requires a newline after the closing ---. If a page ends immediately after…

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Static review found no unresolved blocking issues in the scoped workflow, metadata, parsing, or regression coverage.

2 open issues

• _FRONT_MATTER only matches LF line endings and doesn’t allow a UTF-8 BOM, while the new docs…
• _FRONT_MATTER only matches LF line endings and doesn’t allow a UTF-8 BOM, while the new docs…

3 resolved since last review

• stale_globs() recompiles the regex for each glob once per tracked file (glob_regex(g) is called…
• load_pages() only scans dokimi/docs/*.md (non-recursive). If docs pages are ever organized into…
• The front matter regex requires a newline after the closing ---. If a page ends immediately after…

_{Review effort: Balanced}

[antonwolfy]

Review at a28522f. All three of my earlier threads are fixed, and the new tests for changed_files, the exit-2 path and load_pages kill the mutants I listed. Two things from the last commits are worth fixing before merge, both inline.

What I ran, on a Linux review host at a28522f:

• make ci-local: 20 passed, 0 failed, 4 skipped (docker, main-only, semgrep); 5205 tests passed.
• Reproductions for the finding below, run offline against the script.
• Mutation run on the new code: dropping --no-renames, -z, check_returncode(), return 2 -> return 0, return 1 -> return 0, the \ufeff?, \r? and end-of-string alternatives in the script's regex, and a load_pages that returns absolute paths are all killed.

Checked, no issues

• Workflow: contents: read, SHA pins match the other workflows, BASE_REF via env, fetch-depth: 0. The comment about when the job fails matches main().
• Seeded globs and front matter: every glob matches a tracked file; the front matter sits before the license header, which .licenserc.yaml allows; mkdocs build --strict passes.

[xaleryb]

• Run: the trigger script on a branch that edits code the pages name, plus the full suite, ruff, mypy and mkdocs build --strict on this head merged with main.
• Remaining: the body says "exit 0 (advisory, never fails)" / "only warns". With depends_on: set to null the script prints ::error file=dokimi/docs/methodology.md and returns rc=1; --base nosuch returns rc=2. Expected: the body describes these exit codes, since it becomes the squash message.
• Pushed: 43ed1e1 maps defaults.py, commands/benchmark.py, questions/artifact.py and schemas/questions.v*.json to methodology.md. It maps both 12,000-char loaders, eval/agent_runner.py, mcp/factory.py and sync_skilltext_profile.py to evaluating-treatments.md. Each page names this code, but before the commit an edit to any of these files flagged nothing on the page. There is a test per file.
• Left for the author: edit the body; rebase onto main (conflict in tests/test_docs_hooks.py; keeping both sides is green).

This branch conflicts with main in dokimi/tests/test_docs_hooks.py (changed by #131), so it needs a rebase.

[napetrov]

Thanks for the run and for the mappings in 43ed1e1. Point by point:

1. Run. On head 43ed1e1 (not rebased; the branch still sits on the older main base):

   • Probe commit touching all 8 mapped files (defaults.py, commands/benchmark.py, questions/artifact.py, eval/agent_runner.py, mcp/factory.py, sync_skilltext_profile.py, skills/loader.py, agent_profiles/loader.py): the script flags methodology.md, evaluating-treatments.md and adding-doc-source.md (mcp/factory.py) with ::warning lines.
   • Full pytest: 5217 passed, 2 skipped. Targeted docs tests: 52 passed. mypy on the script: clean. ruff check: clean.
   • mkdocs build --strict passes on Python 3.13 (exit 0). On Python 3.12.3 the build crashes in the HTML parser on skill-novelty-evidence.md, which this PR does not touch, so that is a local interpreter issue.

2. PR body. Fixed. The "exit 0 (advisory, never fails)" line now says findings never fail the job, malformed depends_on gives ::error file=<page>:: and exit 1, and a bad --base gives exit 2.

3. 43ed1e1. Verified, not redone. The mappings are as described and each file has a test.

4. Rebase onto main. Not done yet. The rebase rewrites the branch, so it needs a --force-with-lease push, and the conflict is in dokimi/tests/test_docs_hooks.py. Waiting on the author's go-ahead for the force push before rebasing.

[antonwolfy]

Review at 43ed1e1. The one-line ::error fix, the \ufeff escape and the nine per-file mapping tests all hold up. One test gap is worth closing before merge, and there is one nit.

What I ran, on a Linux review host at 43ed1e1:

• make ci-local: 20 passed, 0 failed, 4 skipped (docker, main-only, semgrep); 5218 tests passed, 1 skipped; the changed-line coverage job passes.
• Reproduction for the finding below, run offline against hooks.py.
• Mutation run on the delta: removing any one of the nine globs added in 43ed1e1 fails exactly one case of test_code_a_page_names_flags_that_page; reverting the ::error message to {e} fails the single-line assertion; dropping the optional BOM, \r?\n -> \n after the opening ---, and (?:\r?\n|$) -> \n or $ are all killed.

Checked, no issues

• Mappings: all nine new paths exist, and each page names the code (skills/loader.py, mcp/factory.py and sync_skilltext_profile.py in evaluating-treatments.md; benchmark run and the questions.v2 schema in methodology.md). questions.v*.json matches questions.v1.json and questions.v2.json.
• Annotation: depends_on: [a now yields a one-line ::error file=... with the parser's problem text.
• Regex parity: the regexes in hooks.py and the script agree on ---- and --- x closers, \r-only text, an unclosed block and a fenced ---.

[xaleryb]

Re-checked at 6ac643d (merged with main 61e0355). I re-ran every earlier input: a null depends_on gives ::error and exit 1, a bad --base gives exit 2, and a probe commit touching each of the nine files mapped in 43ed1e1 flags its page. I also re-ran the regex mutants for the CRLF, BOM and EOF cases in both regexes, plus the full suite, ruff, mypy, bandit and mkdocs build --strict. Nothing is open.

[antonwolfy]

Review at 965c588. Both of my open threads are fixed, and the script, workflow and mappings hold up after the merges from main. One test gap is worth closing before merge, and there is one nit.

What I ran, on a Linux review host at 965c588:

• make ci-local: 27 passed, 0 failed, 4 skipped (docker, test-latest, docs deploy, semgrep); the changed-line coverage job passes and the docs-review-triggers job passes.
• Reproductions for the findings below, run offline against the script.
• Mutation run on the new code: 30 mutants, 20 killed. The one that matters is in the inline comment; the rest are validation-branch and message variants.

Checked, no issues

• Mappings after the merges: every glob matches a tracked file (stale_globs returns []), and releasing.md, repo-evaluation-contract.md, coding-harnesses.md and skill-novelty.md still describe the code they list. mkdocs build --strict passes.
• CONTRIBUTING.md: the paragraph matches the script: advisory findings exit 0, a malformed depends_on exits 1, an undiffable base exits 2, glob rules, tracked files.
• Workflow: pull_request, contents: read, BASE_REF through env, fetch-depth: 0; the merge-ref checkout diffs against the merge base, so later commits on the base do not leak in.
• Renames and deletes: a renamed or deleted dependency flags the page through both old and new paths.

[antonwolfy]

Nothing pins the three-dot range in changed_files. Changing f"{base}...{head}" to f"{base}..{head}" passes the whole new test file (33 passed, 1 skipped).

The rename test builds a linear history, where .. and ... return the same files. With two dots, files that landed on the base branch after the PR branched count as PR changes, and unrelated pages get warnings. The docstring and the workflow both rely on the merge-base form.

Suggestion: add a tmp_path test with a base that advanced after the branch point (commit x on the base branch after branching, commit y on the branch) and assert changed_files("base", "HEAD") == ["y"].

[napetrov]

Added test_changed_files_ignores_what_the_base_gained_after_branching in 1e258d5 (tmp git repo; main gains landed.py after the PR branched). It fails with .. and passes with ....

[antonwolfy]

I pushed a commit per nit, up to 9b95681, which fix the nit from my review:

• 9b95681: dokimi/scripts/check_docs_review_triggers.py:26: the glob docstring now says the slashes around ** are literal, so a/**/b does not match a/b; list both globs.

make ci-local at 9b95681: 25 passed, 2 failed (host: gitleaks install/scan), 4 skipped.

Not approving yet. Still open: the risk from my review.

[napetrov]

@antonwolfy thanks for the nit commit 9b95681 (glob docstring); kept.

Open item from your review at 965c588:

• Nothing pins the three-dot range in changed_files: fixed in 1e258d5e. changed_files is now tested against a tmp git repo where the base gains a file after the PR branched, so base..head would report it and base...head does not. Test: test_changed_files_ignores_what_the_base_gained_after_branching.

Ready for another look.