[napetrov]

Part of #121, J7 #97.

Summary

Draft of the dokimi-design-evals skill (item 6 of #121), built only on commands that exist on main today.

dokimi/data/skills/dokimi-design-evals/
  SKILL.md                         workflow: sources -> scenario -> 3 layers -> verifier -> self-test -> multi-step -> coverage -> status
  references/scenario-template.md  initial state, request, expected outcome, allowed actions, forbidden consequences, verifier, repro, source, status
  references/verifier-self-test.md validate, null/oracle/isolation, mutation controls (manual), judge check
  references/coverage-report.md    happy path, branches, recovery, ambiguity, constraints, negatives, neighbours
  references/example-scenario.md   fixed-failure Harbor task + verifier; a mutation the first verifier misses
dokimi/tests/test_bundled_skills.py  each bundled skill passes check_skill, name == dir, description <= listing cap, relative links resolve inside the skill
dokimi/tests/test_docs_code_refs.py  now also scans data/skills/**/*.md, so every command/flag the skill names is checked against the parser

Steps the skill marks as manual, because no command exists yet:

Need	Today
Mutation controls (skipped step, false test claim, deleted tests, weakened criterion, wrong tool)	manual: mutate a scratch copy's oracle (solve.sh, or --manifest-file for repo tasks) and expect the self-test to fail
Null check for Harbor tasks	manual docker run from contributing-terminal-bench-task §8; repo tasks check covers it only for repo tasks
Valid inaction	repo tasks check always requires the null to fail; such a task needs a manual null check
Scenario coverage report	written by hand; python -m dokimi.coverage is a product x awareness/work count
draft -> reviewed -> holdout status, scenario source	no field dokimi reads; kept in task.toml [metadata] / extra task.json keys
Multi-step checkpoints	Harbor steps; dokimi does not validate per-step dirs
Drafting tasks from code/issues (AC-J07-006)	presented as hypotheses with source, by hand

Evidence

• --help checked for every command the skill names: repo inspect, repo validate --manifest --fail-on-warning, repo tasks check --manifest --task --manifest-file, tasks list --tasks-root, tasks run --tasks-root --tasks --harnesses docker-oracle|docker-probe --out-json, tasks harnesses, eval judge-validity --answers --sequential, python -m dokimi.coverage --tasks-root.
• pytest tests/test_bundled_skills.py tests/test_docs_code_refs.py tests/test_agent_profiles.py tests/test_audit_test_quality.py: 186 passed. The docs-ref check caught one false path (tests/test_outputs.py in the example) before commit.
• make ci-local: 25 pass, 6 skip, 1 fail: pip-audit on virtualenv 21.14.7 (CVE-2026-113722, fix 21.14.9), already in the lock on main and not touched here. Full test run 6728 passed.
• Not done: no agent run with the skill, no Harbor/docker run of the example task (the example's self-test table is labelled illustrative).

Merge danger

• skill-catalog:data/skills arms now see two skills instead of one; a catalog run from this tree is not comparable with an earlier one.
• The new docs-ref coverage of data/skills/**/*.md fails future skill edits that name a renamed command, by design.
• Draft: wording and the manual steps should be reviewed against User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 items 3-4 (gate, validate) before they land, so the skill can switch to those commands.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.32%. Comparing base (af17696) to head (66800f1).
⚠️ Report is 8 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #162      +/-   ##
==========================================
+ Coverage   92.24%   92.32%   +0.08%     
==========================================
  Files         239      240       +1     
  Lines       29934    30325     +391     
  Branches     4917     5014      +97     
==========================================
+ Hits        27613    27998     +385     
- Misses       1633     1639       +6     
  Partials      688      688              

see 16 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 0389033. The skill is well structured, and every command and flag it names exists in the parser. Two things are worth fixing before merge, plus two nits inline.

What I ran, on a Linux review host at 0389033:

• make ci-local: 25 passed, 1 failed, 6 skipped. The failure is pip-audit on virtualenv 21.14.7 (CVE-2026-113722), which is already in the lock at the PR's base and not touched here (fixed on main by #152). Changed-line coverage passed.
• Reproductions for the findings, with the LLM transport mocked; a mutation run of test_bundled_skills.py and the test_docs_code_refs.py change against broken copies of the new skill files.

This PR adds a second skill under dokimi/data/, which the repo rules reserve for fastcontext. AGENTS.md:94-97 says dokimi/data/ holds only the fixtures the repo's own tests and examples read (data/skills/fastcontext/, ...), and that product skills and tasks belong in their content repository. dokimi/data/README.md:22 calls fastcontext "The one committed skill". The new skill is not a fixture for any treatment arm or example; the only reader is the new lint test. Either update AGENTS.md and data/README.md to say why a second bundled skill is allowed, or move the skill to the content repository. The README line (outside this diff) is wrong as soon as this merges.

Checked, no issues

• Commands: repo inspect, repo validate --fail-on-warning, repo tasks check --manifest/--manifest-file/--task, tasks list|run --tasks-root --tasks --harnesses --out-json, eval judge-validity --answers --sequential and python -m dokimi.coverage all exist in --help. docker-oracle and docker-probe are in the registry. The claim that repo tasks check always requires the null to fail holds.
• Catalog and packaging: discover_skills ignores references/*.md. A skill-catalog treatment over both skills builds with no tool-name collision. The SKILL.md context is under the 12,000-character cap. data/ is outside the package, so no package-data glob changes.
• Changelog: the fragment follows the changelog.d form, and no schema or mkdocs.yml nav change is needed.

[antonwolfy]

I pushed a commit per nit, up to 4a0f048, which fix the two nits from my review:

• e2aae98: dokimi/data/skills/dokimi-design-evals/references/verifier-self-test.md:92: added --strict so an inconclusive PASS exits 1.
• 4a0f048: dokimi/tests/test_bundled_skills.py:24: LINK now accepts an optional title and angle brackets (including spaces inside them), and the dangling-link test covers a titled and an angle-bracket link.

make ci-local at 4a0f048: 25 passed, 1 failed, 6 skipped.

Not approving yet. Still open: the two risks from my review.

[napetrov]

Second bundled skill under data/: kept in place and the rules updated in 294bf2e. AGENTS.md (data/ bullet) and dokimi/data/README.md now name dokimi-design-evals as guidance for using dokimi itself, not product content, and say tests/test_bundled_skills.py checks it; the README no longer calls fastcontext "the one committed skill". If you would rather move it to the content repository, say so and I will; that needs your call.

[napetrov]

Owner decision: dokimi-design-evals stays in dokimi/data/skills/. Skills about using dokimi itself live with the CLI and are versioned with it, like fastcontext, so it is not moving to the content repository. AGENTS.md (data/ bullet) and dokimi/data/README.md already say this (294bf2e).

[xaleryb]

• Ran on a Linux host: the full suite on the head merged into main, the null and mutation recipes on scratch Harbor and repo tasks in docker, and judge-validity through the CLI parser.
• Repo mutation recipe (§3): an oracle of <oracle> && <mutation> gives repo tasks check OK with rc=0 and leaves the workspace unmutated. Expected: rc=1. Entry points run without a shell, so the oracle needs sh -c '...', and the doc should say so.
• Multi-step (§6, template row): a task.toml steps checkpoint verifier that always fails still gives docker-oracle 1/1 pass. Without a top-level instruction.md the task fails to load. Expected: the doc says the per-step self-test is manual.
• Judged verifiers: eval judge-validity on a task-rubric answers file stops with missing question_id (exit 2). It calibrates only dokimi's own answer judge.
• Self-test §1: docker-oracle passes a verifier/ + oracle/ layout that the Harbor-backed arms refuse.
• Pushed: d45d4b2 (the Harbor null command never ran a test.sh written for /tests, so it confirmed any verifier; it now runs a no-op solve.sh through docker-oracle), e1a836c (--strict in SKILL.md).
• For the author: the four points above.

[antonwolfy]

Review at e1a836c. The tasks list fix is right, and the skill still names only commands that exist. The commits since 4a0f048 mostly merged main. Not ready: @xaleryb's four doc points are still open, and the link test checks no harness names.

I agree with @xaleryb's points on the && mutation recipe, the unverified per-step checkpoints, judge-validity on task rubrics and the verifier/+oracle/ layout. New evidence: render_entry (repos/grading.py:111) splits with shlex, no shell, so && rm -f x reaches the oracle as argv.

What I ran, on a Linux review host at e1a836c:

• make ci-local: 27 passed, 1 failed, 7 skipped. The failure is pip-audit on gitpython 3.2.0 (CVE-2026-114145), in the lock files, which this PR does not touch. Changed-line coverage passed.
• test_bundled_skills.py and test_docs_code_refs.py against broken copies of the skill files: 58 passed on the clean copy; dangling links, bad flags, a long description and name != dir are caught. A misspelled harness name is not (inline).

Checked, no issues

• Commands: tasks run --tasks-root --tasks --harnesses --out-json, repo tasks check --manifest --task, tasks list and judge-validity --sequential --strict all exist in --help.
• Earlier nits: --strict and the link-regex fix are in the tree and covered by the tests.
• Changelog: the fragment follows the changelog.d form.

[napetrov]

The four doc points from xaleryb's comment and antonwolfy's review at e1a836c, in c4dd526 (branch also merged with main):

1. Repo mutation recipe: references/verifier-self-test.md §3 now says the oracle command is split into argv and run without a shell, so && never runs the mutation, and to write sh -c '<oracle> && <mutation>'.
2. Multi-step: new §5 of the same file, SKILL.md §6 and the template row say docker-oracle does not run the per-step checkpoint verifiers, the per-step self-test is manual, and a top-level instruction.md is required.
3. Judged verifiers: §4 and SKILL.md §4 say eval judge-validity calibrates only dokimi's own answer judge (rows need question_id), not a task rubric judge.
4. Layout: §1 says to use Harbor's tests/ + solution/ layout; verifier/ + oracle/ passes docker-oracle but the Harbor-backed arms refuse it.
   The per-step procedure in §5 is my wording of the manual check; please check it reads right.

[antonwolfy]

LGTM. The four doc points from the last round are fixed, and the harness-name check from my thread is in. Three nits inline.

What I ran, on a Linux review host at c4dd526:

• make ci-local: 28 passed, 1 failed, 7 skipped; 7,759 tests; changed-line coverage passed. The failure is pip-audit on filelock and setuptools in the lock files, which this PR does not touch.
• Mutants of the skill files against test_bundled_skills.py and test_docs_code_refs.py: a misspelled harness (plain, in a comma list, quoted, with a :variant), a wrong flag, a dangling link, a long description and name != dir are all caught. --harnesses=docker-probee is not (inline).

Checked, no issues

• Oracle recipe: render_entry (repos/grading.py:111) uses shlex.split with no shell, so && reaches the program as an argument and sh -c '<oracle> && <mutation>' runs both.
• Layout and null check: by reading harnesses/tasks.py, docker_solver.py and harnesses/registry.py, docker-oracle accepts verifier/ + oracle/ while harbor_task_problem refuses it for the Harbor-backed arms. A no-op solve.sh leaves the workspace untouched before the verifier runs.
• Judge validity: a task-rubric answers file stops with missing question_id and exit 2, as the text says (read from records.py and evaluate.py).
• Repo rules: the changelog fragment follows the changelog.d form, no generated artifacts, no schema change.

[antonwolfy]

LGTM. The three nits from my last review are fixed: the harness-name extractor now reads the = form, trailing punctuation and backslash continuations, the example's untouched row goes through docker-oracle, and §5 states the top-level tests/test.sh + solution/solve.sh requirement and gives a manual way to run a step's verifier. Two nits inline.

What I ran, on a Linux review host at 6e8f1e2:

• make ci-local: 29 passed, 0 failed, 7 skipped; 8,220 tests. The diff has no changed Python lines for the coverage gate.
• Reproductions of the §5 claims on scratch Harbor tasks, without Docker: a steps task with only steps/<name>/tests passes harbor_task_problem and is refused by docker-oracle with no tests/ directory to copy to /tests; a task with both top-level files is accepted; a missing instruction.md fails to load.
• Probes of the new HARNESSES_FLAG extractor on edge forms (inline).

Checked, no issues

• Manual step recipe: mkdir -p /logs/verifier && chmod +x /tests/test.sh && /tests/test.sh matches the verifier command the harness builds (harnesses/tasks.py:153-154).
• Commands: every command and flag the skill names exists in the parser; the docs-ref scan covers data/skills/**/*.md.
• Repo rules: the changelog fragment follows the changelog.d form, the new files need no license header, no mkdocs.yml nav or schema change is needed, the tests are offline, and AGENTS.md and data/README.md now say why a second bundled skill is allowed.

[antonwolfy]

LGTM. I pushed a commit per nit, up to 66800f1, which fix the two nits from my review:

• bd44dc3: dokimi/tests/test_bundled_skills.py:98: the extractor drops empty names and matches only a value that starts with a letter, with two new cases for comma-plus-space and a wrapped list.
• 66800f1: dokimi/data/skills/dokimi-design-evals/references/verifier-self-test.md:125: names both refusal messages, puts tests/ at /verifier too, and says to read /logs/verifier/reward.txt with the exit code only as a fallback.

make ci-local at 66800f1: 29 passed, 0 failed, 7 skipped.