[napetrov]

Summary

dokimi's skill arm can only answer "is the content useful once the skill is loaded": its prompt opens with TREATMENT_NOTICE, which tells the agent to load the skill. Skill authors also ask whether the agent finds the skill without being told. This PR adds a discovery arm for that question. It installs the same plugin and leaves the notice out. The existing arm and its default are unchanged.

Arm	Skill installed	Notice	Pass rate measures
docker-claude	no	no	baseline
docker-claude-skill:<p>	yes	yes	the skill's content, given that it is loaded
docker-claude-skill-discover:<p> (new)	yes	no	content × whether the agent finds the skill from its description

 registry._create
   docker-claude[-host]-skill:<p>           -> DockerSolverHarness(skill_notice=True)
+  docker-claude[-host]-skill-discover:<p>  -> DockerSolverHarness(skill_notice=False)

 DockerSolverHarness._claude_args
-  prompt = treatment_notice(t) + "Solve this coding task..."
+  prompt = (treatment_notice(t) if self.skill_notice else "") + "Solve this coding task..."

 _record_treatment                      # both skill arms, container and host
   metrics.treatment                    # unchanged
+  metrics.skill_notice   true|false
+  metrics.skill_listing  {description_chars, max_desc_chars: 1536, budget_chars: 8000, truncated}
+  run.log WARNING if discovery arm and truncated

 summarize_task_results
+  summary.discovery_gaps[<discover>] = {with_skill_harness, with_skill_pass_rate,
+      discover_pass_rate, gap = with − discover (withheld if arms lost different cells),
+      with_skill_invocation_rate, discover_invocation_rate}

 task_runs_report
+  "## Skill discovery" table (only when a pair ran)

• Same launch. Same plugin dir, --tools, --settings and env. A test compares the two arms' claude argv with the prompt taken out, and they match.
• Listing visibility. The solver did not pin skillListingBudgetFraction, and this PR still doesn't. Claude Code 2.1.296 cuts each listed description at skillListingMaxDescChars (1536). It also shortens the whole listing to fit 0.01 × context × 4 characters, which is 8000 on a 200k window. These arms list one plugin skill plus doctor (and design on the host route), so the budget can't be reached. Only the per-skill cut can apply, and metrics.skill_listing.truncated records it. A missing plugin was already a solver_error in the init event, and that check also covers the new arm. The hidden-description problem seen in the external tools came from 33-skill catalogs, which these arms don't run. Pinning the fraction would add a second difference between the arms, so I left it out.
• Skill calls: n_skill_invocations and the treatment-loaded check come from stream_stats, the same as for the skill arm. The headline already shows each harness as its own row with "Skill invoked".
• Scope: treatments/specs.py (the Q&A --arms grammar) and eval/arm_summary.py (Q&A rollups) don't see docker-* harness ids, so both are untouched. harnesses/command.py is untouched too.
• Schema: task_runs.v1 is widened in place: optional metrics.skill_notice and optional summary.discovery_gaps. schema_compat against main passes.

Evidence

Why. Measured 2026-10-06 with haiku-4.5 on the onetbb-quickstart skill, in real Claude Code, with the skill installed and no notice:

runner	Skill calls	pass
Harbor	0/6	1/6
Inspect	0/10	4/10 (same as no-skill)
Promptfoo	0/6	—
with notice (any runner)	10/10	10/10

After (live smoke, this branch). onetbb-transpose, us.anthropic.claude-haiku-4-5-20251001-v1:0, 3 reps, in-container agent. Total about $0.26.

arm	pass	Skill invoked	cost
docker-claude	0/3	—	$0.082
docker-claude-skill:onetbb-quickstart	2/2 + 1 errored, rerun 1/1, so 3/3	3/3	$0.096
docker-claude-skill-discover:onetbb-quickstart	0/3	0/3	$0.082

• In every discovery row the plugin was offered: offered_plugins: ["onetbb-quickstart"], with onetbb-quickstart:onetbb-quickstart in offered_skills.
• skill_notice: false in every discovery row. skill_listing.description_chars: 454, truncated: false.
• So the description was in the listing in full, and haiku never loaded the skill. The discovery gap on this task is +100 pp.
• The errored skill cell was FileNotFoundError on the sandbox's /tmp/dokimi-env-*/env-* file. It happened in a shared /tmp before the solve and has nothing to do with this change.
• In the first run, the report's gap reads "— (arms lost different cells)", because the arms lost different cells. That is the intended withholding.

Checks (local).

• ruff check .: clean.
• mypy: no issues in 212 files.
• pytest: 5186 passed, 2 skipped.
• pytest --outcomes -m outcome: pass.
• mkdocs build --strict: OK.
• schema_compat --base intel/main: OK.
• pre-commit on the changed files: pass.
• diff-cover vs intel/main: 100% of 65 lines.
• pre-commit run --all-files reports 50 ruff findings, all in .agent-eval/. Those files are on main and this PR doesn't touch them.

New tests (fake runner, no LLM):

discover arm (container): prompt starts "Solve this coding task.", skill not named;
  claude argv minus prompt == skill arm's; skill_notice False; same treatment;
  n_skill_invocations counted; plugin missing -> solver_error
discover arm (host): parametrized with the existing sealed-env host test
listing: description + when_to_use > 1536 -> truncated, run.log warning
registry: both aliases, with_skill_counterpart, empty path refused, --no-skill-notice in command
summary: discovery_gaps value, withheld when cells differ, absent without a pair
report: three arms in headline, Skill discovery row, truncation note, artifact validates

Merge Danger

Door: two-way

Adds new harness ids and optional artifact keys. The existing arms' prompt, argv and summary are unchanged. Reverting removes the aliases, and artifacts already written still validate.

Blast Radius: additive

Skill-arm rows gain skill_notice and skill_listing. Summaries gain discovery_gaps only when a discovery arm and its pair run together. A discovery arm also enters the McNemar Holm family as one more arm against the baseline, which slightly changes the adjusted p of other arms in runs that include it.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.34%. Comparing base (6e695ce) to head (eba03bd).
⚠️ Report is 17 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
+ Coverage   90.24%   90.34%   +0.09%     
==========================================
  Files         206      206              
  Lines       24289    24502     +213     
  Branches     3906     3958      +52     
==========================================
+ Hits        21920    22136     +216     
+ Misses       1703     1699       -4     
- Partials      666      667       +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/docker_solver.py	94.80% <100.00%> (+0.13%)	⬆️
dokimi/dokimi/harnesses/registry.py	93.54% <100.00%> (+2.11%)	⬆️
dokimi/dokimi/harnesses/runner.py	99.13% <100.00%> (-0.87%)	⬇️
dokimi/dokimi/harnesses/skill_treatment.py	96.47% <100.00%> (+0.19%)	⬆️
dokimi/dokimi/report/task_runs_report.py	93.95% <100.00%> (+0.86%)	⬆️

... and 20 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

The changes preserve existing defaults and have focused regression coverage; only a non-blocking documentation clarification remains.

1 open issue

• Avoid overstating skill discovery from zero invocation counts · New

What changed in this PR

Adds a discovery arm to measure whether agents find an installed skill without a treatment notice, while preserving existing defaults.

Changes:

• Adds container and host discovery harnesses.
• Records notice and listing metrics, plus paired discovery gaps.
• Updates reports, documentation, schema, and regression tests.

File	Description
dokimi/​tests/​test_task_runs_report.py	Tests discovery reporting and artifact validation.
dokimi/​tests/​test_skill_treatment.py	Tests listing-length metrics.
dokimi/​tests/​test_docker_solver_harness.py	Tests registration and discovery gaps.
dokimi/​tests/​test_container_agent.py	Tests discovery launch and warnings.
dokimi/​tests/​test_agent_sandbox.py	Extends host isolation coverage.
dokimi/​dokimi/​schemas/​task_runs.v1.json	Adds optional discovery fields.
dokimi/​dokimi/​report/​task_runs_report.py	Renders discovery comparisons.
dokimi/​dokimi/​harnesses/​skill_treatment.py	Calculates listing metrics.
dokimi/​dokimi/​harnesses/​runner.py	Summarizes paired discovery gaps.
dokimi/​dokimi/​harnesses/​registry.py	Registers discovery harnesses and counterparts.
dokimi/​dokimi/​harnesses/​docker_solver.py	Makes the notice optional and records metrics.
dokimi/​docs/​coding-harnesses.md	Documents discovery usage and interpretation.
dokimi/​docs/​architecture.md	Describes discovery-arm integration.
dokimi/​changelog.d/​20261018_120000_napetrov_skill-discover-arm.md	Announces discovery support.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The additive changes preserve defaults and have targeted regression coverage; the remaining report-wording issue is non-blocking.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Handle no-scored-cells case separately from differing lost cells

dokimi/​dokimi/​report/​task_runs_report.py:164

Non-blocking: If both arms error on every cell, their errored-cell sets match, but the gap is null because neither has a pass rate. The report incorrectly labels this as “arms lost different cells.” Use a separate explanation when either arm has no scored cells.

1 resolved since last review

• Avoid overstating skill discovery from zero invocation counts

_{Review effort: Balanced}

[antonwolfy]

The arm itself is wired correctly: the notice is dropped on both routes, the argv matches the skill arm's, and the registry pairing is right. Two things are worth fixing before merge, both in _discovery_gaps, plus one nit.

What I ran, on a Linux review host at 058646e:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped (semgrep, test-latest, docs deploy). 5,187 tests, changed-line coverage 100% of 65 lines, schema_compat and mkdocs build pass.
• Reproductions for the _discovery_gaps findings, with no LLM involved.
• A mutation run over the new code: 45 mutants, 9 of which pass the five changed test files. One equivalent mutant aside, the survivors are the None guard (inline) and the four gaps listed below.

Test gaps found by mutation (all four mutants pass the changed test files):

1. The truncation boundary is not pinned. skill_treatment.py:238 uses >. Changing it to >= passes. No test uses a description of exactly 1536 characters, so a description that fits would be flagged as cut.
2. The not self.skill_notice condition on the warning is not pinned. docker_solver.py:520: dropping it passes. The only warning test runs the discovery arm, so the skill arm (notice on, long description) could start writing the "description cut" warning to run.log unnoticed.
3. The report's harness in gaps filter is not pinned. task_runs_report.py:169: replacing it with True passes. _record_treatment writes skill_listing on the with-skill rows too, so with the filter gone those rows would be counted as discovery rows. The test's only truncated row is the discovery row.
4. The skill_notice default is not pinned. docker_solver.py:126: flipping it to False passes, because the registry always passes the flag explicitly. A hand-built DockerSolverHarness(skill_path=...) would silently lose the notice.

I agree with @copilot-pull-request-reviewer on the null-gap label (task_runs_report.py:164). New evidence: the schema description of gap already says null also covers "one has none scored", so the report text and the schema disagree.

Checked, no issues

• Registry: docker-claude-skill-discover: cannot match docker-claude-skill:, -host- is tried first, and with_skill_counterpart maps docker-claude-host-skill-discover:/a:b to docker-claude-host-skill:/a:b and a with-skill id to None.
• skill_listing: folded and literal scalars, numeric and list descriptions, and CRLF files parse. 1536 characters is not truncated and 1537 is. check_skill already rejects a missing SKILL.md before it runs.
• Docs: 1536, 8000 and the harness ids match the code. The command example's flags exist in commands/tasks.py. No new page, so mkdocs.yml is unaffected.
• Schema: the change only widens task_runs.v1 (optional skill_notice, optional discovery_gaps), which the in-place rule in AGENTS.md allows. schema_compat passes.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

• bool is a subclass of int in Python, so _fmt_rate(True) would render as 100% rather than… · New
• When gap is None, the report always prints “— (arms lost different cells)”, but gap is also… · New
• description_chars appears to represent the combined listing text (description plus optional… · New

_{Review effort: Lite}

[napetrov]

Fixed in 5904e51. Updated the schema description for description_chars to clarify it counts the listing entry text shown to the model before truncation (description plus optional when_to_use with ' - ' separator), not just description alone. Also updated the docstring in skill_treatment.skill_listing() to match.

[Copilot]

🟢 Approval recommended

The additive implementation preserves existing defaults, includes focused regression coverage, and has no unresolved blocking findings.

3 open issues

• description_chars appears to represent the combined listing text (description plus optional…
• When gap is None, the report always prints “— (arms lost different cells)”, but gap is also…
• bool is a subclass of int in Python, so _fmt_rate(True) would render as 100% rather than…

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The additive changes preserve existing defaults, include focused regression coverage, and have no unresolved blocking findings.

1 open issue

• description_chars appears to represent the combined listing text (description plus optional…

2 resolved since last review

• When gap is None, the report always prints “— (arms lost different cells)”, but gap is also…
• bool is a subclass of int in Python, so _fmt_rate(True) would render as 100% rather than…

_{Review effort: Balanced}

[antonwolfy]

Review at 5904e51. The three threads from my earlier review are fixed: the attempted-cells check, the None guard and the skill_listing schema entry are all in, with tests that fail without them. Two things remain: the four test gaps I listed last time are untouched, and the new null-gap label is still imprecise.

What I ran, on a Linux review host at 5904e51:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped. 5,191 tests passed, changed-line coverage 100%. Python 3.13 was used here, so the project's own coverage gate was not applied.
• Mutation run over the changes since 058646e (attempted-cells check, None guard, the new report branch, _fmt_rate), plus the four gaps from my last review. Reproductions for the gap/label behaviour, with no LLM involved.

Test gaps from my earlier review, still not covered (the five changed test files still pass with each mutant):

1. The truncation boundary is not pinned. skill_treatment.py:242: changing > to >= passes. Add a test with a description of exactly 1536 characters and assert truncated is false, and one with 1537 that asserts true.
2. The not self.skill_notice condition on the warning is not pinned. docker_solver.py:520: reducing it to if listing["truncated"]: passes. Add a skill-arm (notice on) test with a long description that asserts no [skill] WARNING line in run.log.
3. The report's harness in gaps filter is not pinned. task_runs_report.py:174: dropping it passes (18 passed in test_task_runs_report.py), because the only truncated row is on the discovery arm. Add a truncated skill_listing on a -skill row and assert the footnote count excludes it.
4. The skill_notice default is not pinned. docker_solver.py:126: flipping True to False passes, because the registry always passes the flag. Add a test that builds DockerSolverHarness(skill_path=...) directly and asserts the prompt opens with the notice.

Checked, no issues

• Threads: the attempted-cells mutant (True and ...) fails test_discovery_gap_withheld_when_arms_attempted_different_cells. Removing the None guard raises TypeError and fails test_discovery_gap_both_arms_fully_errored.
• Pairing: docker-claude-host-skill-discover:X pairs with docker-claude-host-skill:X. A discover arm with no pair, or beside a host/non-host mismatch, produces no discovery_gaps and no section.
• Gap values: 2/3 against 0 gives 0.6667 and +66.67 pp. A zero-scored discovery arm gives gap: null and "— (no scored cells)". _fmt_rate(True) now renders "—".
• Schema: validate_artifact accepts real summarize_task_results output for a normal gap, a null gap from differing cells and a null gap from fully errored arms. The change only adds optional keys.
• Docs and repo rules: changelog fragment format, no new page for mkdocs.yml, no generated artifacts, no with_docs/without_docs keys touched.

[xaleryb]

• Run: full suite on the head and on the head merged into current main, ruff, mypy, bandit, a mutation pass over the new guards, and a CLI dry run on intel/skills' onetbb-transpose.
• docker-claude-skill:$S beside docker-claude-skill-discover:$S/, $S/SKILL.md or a relative spelling of the same directory → no discovery_gaps, no Skill discovery table, no warning → the arms pair on launch plus skill directory.
• A truncated -skill-discover row run against the baseline only → run.log warns, but the report says nothing → the report note appears whether or not the pair ran, as the docs say.
• Pushed:
  • 9f5e27d: pair by resolved skill directory; docs and changelog fragment updated.
  • 0820840: count truncated rows by skill_notice: false, outside the table.
  • 331c66c: tests for the four mutation gaps raised in review (1536 boundary, warning guard, notice default; the report filter is pinned in 0820840).
• Left for the author: resolve the outdated description_chars thread (fixed in 5904e51).

[xaleryb]

Approved at 331c66c. No must-fix or mid item remains after our commits, each red without its fix; full suite, ruff, mypy and bandit clean on Linux; CI green.

[napetrov]

Re-checked each point against the fork head 331c66c:

• Verification: the touched test files pass on 331c66c (test_container_agent, test_skill_treatment, test_task_runs_report, test_docker_solver_harness: 213 passed, 1 skipped), and ruff is clean. I did not re-run the full suite, mypy, bandit, the mutation pass or the CLI dry run.
• Pushed commits: 9f5e27d (pair by resolved skill directory), 0820840 (count truncated rows by skill_notice false) and 331c66c (tests for the 1536 boundary, warning guard and notice default) are all on feat/skill-discover-arm. Thanks for verifying them.
• description_chars thread: already answered in 5904e51 and left unresolved for you to close.

No further code changes from my side.

🤖 Generated with Claude Code

[antonwolfy]

Review at 331c66c. The pairing by skill directory, the ran/lost/ran-and-lost labels and the report note for unpaired rows all work, and the earlier test gaps are now pinned. Three points remain, all in the tests of the new guards.

What I ran, on a Linux review host at 331c66c:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped (semgrep, test-latest, docs deploy). 5,203 tests passed, changed-line coverage 100% of 93 lines.
• Reproductions with the LLM transport mocked, and a mutation run over the new code: 58 mutants, 45 killed, 13 survived. Three are equivalent and one is near-equivalent. Three of the survivors are listed below; the rest are rounding, warning text and a mode guard on a dry-run label.

Checked, no issues

• Pairing: s, s/, ./s, s/SKILL.md and an absolute path pair with the same discovery arm. A host arm never pairs with a container arm. Two discovery arms with different spellings each get their own gap entry.
• Schema: real summarize_task_results output, including a null gap with errored_cells_match: false, validates against task_runs.v1.json. The change only widens the schema.
• Docs: 1536, 8000, the harness ids and the three withheld-gap labels match the code. mkdocs build --strict passes.
• Earlier test gaps: the 1536 boundary, the skill_notice default and the not self.skill_notice guard now fail when mutated.

[antonwolfy]

LGTM. The three test gaps from my last review are closed, and each new test fails when its guard is mutated. The last commit changes tests only.

What I ran, on a Linux review host at eba03bd:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped. 5,207 tests passed, 1 skipped, changed-line coverage 100% of 93 lines.
• Mutation run over the three guards from my last review: if not self.skill_notice: at docker_solver.py:520, .get("truncated") is not None at task_runs_report.py:150, and or changed to and at task_runs_report.py:171. The new tests kill all three. Reducing the warning guard to if listing["truncated"]: is killed by the existing announced-arm test.

Checked, no issues

• Warning guard: a test exists in each direction (truncated with notice off, truncated with notice on, untruncated with notice off).
• No-scored-cells label: the new test is parametrized over each arm being the unscored one, and both cases assert — (no scored cells).
• Docs and schema: 1536, 8000, the harness ids and the withheld-gap labels match the code. The schema change only widens task_runs.v1.