[napetrov]

Part of #48.

Summary

The judge's view of each arm's treatment material lived in memory for the answering run only. Anything reading the artifact later gave its judge 200-char report snippets instead.

 arms.v1
+  judge_materials: {sha256:… : text}           # one copy per distinct text
   answers[].arms.<arm>
+    judge_context_sha256: sha256:…              # which text this arm's judge saw
   evaluations[].scores.<arm>
     judge_context_source: stored | snippet_fallback
+    judge_context_sha256: sha256:…

 JudgeContextStore
+  digest_of(qid, arm) / materials()
+  from_artifact(data)        # rebuild from the file; text must hash to its digest
 judge_context.cite(records, store)   # stamps records, returns the texts cited
 cross_judge.score_measure(…, store=)
   eval cross-judge --judges  ->  store = JudgeContextStore.from_artifact(data)

cross-judge --judges no longer refuses an arm whose material the file keeps. --pairwise / --checklist still refuse, because those judges see only the answers.

Evidence

• Before: a judge re-reading an arms artifact resolved every non-baseline arm to snippet_fallback. In one review that was 28 characters of context against 11,079, and the same judge on the same answers read +24.2 (p 0.0006) one way and +1.4 (p 0.65) the other.
  After: test_a_judge_reading_the_artifact_sees_what_the_arms_run_judge_saw takes a judged run through JSON into a fresh ArmRunner with JudgeContextStore.from_artifact, re-judges it, and gets byte-identical contexts, every score stored, and the run stamped treatment_material.v1.
• test_a_skill_body_shared_by_every_question_is_stored_once: one digest across questions, cited by the records and by the scores.
• test_an_unjudged_run_still_keeps_the_material, test_material_that_does_not_match_its_digest_is_not_used (tampered text falls back to snippets, with a warning), test_an_artifact_without_materials_falls_back_as_before.
• test_an_arm_whose_material_the_file_keeps_is_reproducible_by_a_rescored_judge, test_score_measure_hands_the_stored_material_to_the_rescored_judge.
• Size: perf skill arm material 4,551 chars, stored once: +0.3% on an existing 81-question, two-arm artifact (1.71 MB), plus about 90 bytes of digest per arm record.
• pytest -q 4299 passed, 2 skipped; ruff check ., mypy, schema_check, schema_compat --base intel/main clean.

Merge Danger

Door: two-way. Additive fields; old readers ignore them, and old artifacts have none and fall back as before.

Blast Radius: artifact size

• Docs arms now carry their retrieved chunks once per question (up to the answerer's 15,000-char cut each), so a docs-heavy run grows by roughly that × questions. Skill arms grow by one copy.
• cross-judge --judges now runs, where it used to exit 2, on new artifacts whose stored judge saw material. Its re-scored judge sees more context than before, so those scores move toward the stored ones; that is the point.
• The artifact now contains the treatment text itself. Anything that ships arms artifacts publicly ships the skill or doc text too. For the public skills measured here that is fine; for private material it is new exposure.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — evaluating-treatments.md, benchmarking-and-comparison.md
• No ad-hoc run artifacts committed

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 87.12121% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.24%. Comparing base (8193ce0) to head (097f872).
⚠️ Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/eval/cross_judge.py	60.86%	4 Missing and 5 partials ⚠️
dokimi/dokimi/eval/judge_context.py	92.53%	3 Missing and 2 partials ⚠️
dokimi/dokimi/commands/judge_reliability.py	91.17%	1 Missing and 2 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #69      +/-   ##
==========================================
+ Coverage   87.94%   89.24%   +1.30%     
==========================================
  Files         196      204       +8     
  Lines       22288    23747    +1459     
  Branches     3546     3837     +291     
==========================================
+ Hits        19601    21193    +1592     
+ Misses       2005     1848     -157     
- Partials      682      706      +24     

Files with missing lines	Coverage Δ
dokimi/dokimi/eval/arm_runner.py	92.42% <100.00%> (+0.26%)	⬆️
dokimi/dokimi/utils.py	88.00% <100.00%> (+0.39%)	⬆️
dokimi/dokimi/commands/judge_reliability.py	86.75% <91.17%> (+0.48%)	⬆️
dokimi/dokimi/eval/judge_context.py	96.03% <92.53%> (-3.97%)	⬇️
dokimi/dokimi/eval/cross_judge.py	94.47% <60.86%> (-5.53%)	⬇️

... and 58 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

Reviewed together with #73 (see the comment there for the full run): the artifact round trips, digest determinism (also when the same skill is loaded from another path), dedupe and tamper handling (mismatching text, missing digest or material are skipped with a warning, NUL and unicode round-trip) hold, and schema_compat shows only the new optional judge_materials property.

One defect on this head: digest_text raises on a lone surrogate in a material (JSON \ud83d), which aborts the judge phase before the artifact is written. Fixed with surrogatepass in 497a489, pushed on #73's branch (it touches utils.py only; digests of valid text are unchanged), so it reaches this PR by cherry-pick or when #73 rebases after this lands. Still open here: a list or dict judge_context_sha256, a non-dict record or an unhashable question_id raise in from_artifact (hand-damaged files only), and the cross_judge.py docstring still says the material is kept "in memory only".

[antonwolfy]

The persistence design is sound: digests on the records and scores, one copy per distinct text, tamper check on read, and --judges no longer refused when the file keeps the material. The gate that decides when --judges may run is weaker than the reader it protects, and the new CLI path has no test. The first two inline comments are the ones I'd fix before merge; the other three risks can follow, plus three nits.

What I ran, on a Linux review host at 656cf2c:

• make ci-local: 18 passed, 1 failed, 4 skipped. 4300 tests passed, 1 skipped; changed-line coverage passes (about 96%, 55 of 57 lines; the two uncovered lines are the continues in from_artifact and cite). The one failure is schema_compat: the branch is 4 commits behind main, so the check reports tool_preference.v1.json and tool_preference_groups.v1.json as removed. The PR's own line is a WARNING (new judge_materials property, no base fixture writes it). A rebase clears it.
• Reproductions for the code findings, with the LLM transport mocked (eval cross-judge --judges driven through the real parser with a fake Judge).
• A mutation run: 31 mutants of the new code against test_arm_runner.py and test_judge_reliability.py; 20 killed, 11 pass the suite. Five of the 11 are equivalent; six are real gaps (see the inline comments).

Open from the earlier review, still unaddressed at this head: the lone-surrogate crash, the hand-damaged-file crashes in from_artifact, and the stale cross_judge.py docstring. Each has an inline comment with the reproduction.

Checked, no issues

• Artifact round trip: a judged run through JSON into a fresh ArmRunner with from_artifact yields identical contexts and treatment_material.v1. Int ids, including 0, round-trip. The real writer sets only question_id, so utils.question_id(rec) and rec["question_id"] agree.
• Ordering: cite runs after judge(), on the same record objects that get saved, so the record digest and the score digest come from the same store. It is idempotent.
• Schema: a real build_output validates against arms.v1, a bad judge_materials key is rejected, and schema_compat flags only the new optional property. No version bump is needed.
• Writers: ArmRunner.save is the only writer of arms.v1; plugin_delta, model_compare, matrix_rollup and arms_report only read it.
• Repo rules: no generated artifacts, tests offline, with_docs / without_docs untouched, no new doc page needing a nav: entry.

[antonwolfy]

Review at 4ba3c6d. The surrogate fix, the malformed-record guards in from_artifact and the end-to-end --judges tests are real progress. Two points from my earlier review are still not closed, plus two smaller ones listed below: the --judges gate can still say "reproducible" when the re-scored judge reads a different text, and the CLI tests still miss the pairwise/checklist-with-materials case. I'd fix those two before merge.

What I ran, on a Linux review host at 4ba3c6d:

• make ci-local: 19 passed, 0 failed, 4 skipped. 4,313 tests passed, 1 skipped; changed-line coverage 91% (121 changed lines measured). schema_compat passes, with the one expected WARNING for the new judge_materials property. The run used Python 3.13.
• Reproductions for the code findings, with the LLM transport mocked (eval cross-judge --judges driven through the real parser with a fake Judge).
• A mutation run: 65 mutants of the new code against test_arm_runner.py + test_judge_reliability.py (157 tests). The ones that pass the full suite are in the guards and gate paths named below.

Still open from my earlier review:

• cross_judge.py / judge_reliability.py:241-252: the gate checks that a stored score's digest is a key in judge_materials and that gate_store.get(qid, arm) is non-None. It never compares the digest on the score with the digest from_artifact serves for that record. A file with scores citing digest D1 ("FULL SKILL BODY") and all three skill records citing digest D2 ("OTHER SKILL BODY"), both held in judge_materials, exits 0 and the re-scored judge reads OTHER SKILL BODY for Q0, Q1 and Q2; the stored judge saw FULL SKILL BODY. The reply says records that cite a different digest than the scores exit 2; that holds only when the scores' digest is missing from judge_materials. The same loop skips a stored score whose question_id is null (qid is not None), while the reader falls back to snippets for that record: contexts seen were {s-None: SNIP, s-Q1: full body, s-Q2: full body}, exit 0. Suggested fix: require s.get("judge_context_sha256") == gate_store.digest_of(qid, arm) and treat a null qid as unresolvable. Also, an evaluation whose question_id has no answer record (a hand-trimmed answers list) makes the gate exit 2 with "a file written before them does not have" although the file has judge_materials; score_measure iterates the answer records and never scores it.
• test_judge_reliability.py:763, :792: test_cross_judge_pairwise_with_materials_refuses_comparison and the checklist twin build their file with _stamped_run(), which has no judge_materials, so they repeat the parametrized test at :840 and do not test "materials present". Mutation check: changing the pairwise/checklist call at judge_reliability.py:233 to unreproducible_arms(data, with_material=True) passes the full suite. Two more --judges mutants pass: building materials only when stored_measure(data) is truthy, and only when --no-stored is not set. Neither has a CLI test (an answers-only file, and --no-stored), and the gate does not run on those paths, so a silent fall back to snippets would go unnoticed.
• judge_reliability.py:255-259 and judge_context.py:261: the stale-wording fix is incomplete. For a file whose material text was altered, stderr says "this run did not keep its material" and then "a file written before them does not have". Both are wrong: the file has judge_materials, and one text fails its digest. The message never says that a digest did not match, and it does not name the questions.
• Malformed files (judge_reliability.py:250-252): the records are now guarded, but the gate loop is not. An evaluation with question_id: ["a"] and a stored score that has an error and no aggregate raises TypeError: unhashable type: 'list' at gate_store.get(qid, arm); origin/main exits 2 on that input. answers: 5 raises TypeError: 'int' object is not iterable in from_artifact (judge_context.py:218). Fix: isinstance(qid, (str, int)) in the loop and isinstance(answers, list) in the reader. The from_artifact(data) warning ("N arm records cite judge material that is missing…") is also printed twice on the --judges path, once from gate_store and once from materials, and now prints on --checklist-only runs, where origin/main printed nothing. Building the store once, only when --judges is set, avoids both.

Checked, no issues

• Lone surrogate: a material containing "\ud83d" runs through ArmRunner judge, build_output, save (schema validation passes, JSON is ASCII-escaped) and a reload; from_artifact serves the text, the re-judge sees identical contexts, all scores are stored, and the stamp is treatment_material.v1. The unjudged build_output also works.
• from_artifact on damaged input: list/dict/int/empty digest, non-dict record or arm, arms as a list, list/dict/float question_id, judge_materials as a list or null, and a non-string material all fall back without a traceback, and the count in the warning is right.
• Flag combinations: --judges with --pairwise or --checklist, --pairwise alone and --checklist alone exit 2 on a file with materials; --no-stored lets them run. For pre-PR stamped and unstamped files, the exit codes and contexts match origin/main. Int ids, including 0, resolve identically in the gate and the reader.
• Schema and repo rules: arms.v1 widened in place with an optional property, key pattern ^sha256:[0-9a-f]{64}$ matches digest_text output; no with_docs/without_docs change, no new doc page, no generated artifacts, not stacked, git diff --check clean.

[antonwolfy]

This assertion cannot see a partial fall back to snippets, and the gate loop it covers is only tested when every question is damaged the same way.

any("full skill body" in ...) passes when one question gets the full body and the other seven get snippets, which is the stored-versus-re-scored mix the gate exists to stop (the same pattern is at :615). The three refusal tests (:654, :689, :727) damage all 8 questions identically.

Mutation check, each of these passes test_arm_runner.py + test_judge_reliability.py (157 passed):

• the gate loop at judge_reliability.py:244 runs over evaluations[:1] only
• if qid is not None and becomes if qid and at :251
• cite() skips question 0 (not qid); QUESTIONS in test_arm_runner.py uses q1/q2, and the int-id CLI test at :577 writes its digests by hand
• the isinstance checks on the answer record, arms, body, the evaluation, scores and s in unreproducible_arms, and the isinstance(qid, (str, int)) check in from_artifact
• errors="surrogatepass" becomes errors="ignore": the surrogate test only checks the sha256: prefix and length, so "a\ud83db" and "ab" would hash alike and the tamper check could not tell them apart

Suggestion: assert all(...) over the contexts seen, add a case where one of eight questions has altered material and expect exit 2, run one ArmRunner round trip with question id 0 through cite, and give each hand-damaged shape a one-line test that expects a fall back and no exception.

[napetrov]

The any() asserts are now all(... for c in contexts_seen if c != NO_CONTEXT) plus an exact count (8 skill contexts) in the end-to-end and int-id tests; an earlier reply claimed this before it was true. test_cross_judge_judges_refuses_one_damaged_question_among_eight damages only question 5 four ways (score digest, record digest, record text failing its digest, null id) and expects exit 2, so evaluations[:1] and qid is not None mutants now fail. Question id 0 goes through a real ArmRunner run, cite and from_artifact in test_question_id_zero_is_cited_and_served_after_a_round_trip (test_arm_runner.py); a not qid mutant in cite fails it. test_digest_text_tells_a_lone_surrogate_from_its_removal fails with errors="ignore", and test_from_artifact_skips_a_malformed_answer_list_and_non_scalar_ids covers the answers guard. Commit 0c9d32e (on top of a382bc4, de4e514).

[antonwolfy]

Nit: the --judges block repeats the pairwise/checklist call, and the comments read like review-round notes.

With --judges and --pairwise together, unrepro.update(unreproducible_arms(data)) runs at :233 and again at :239. The inner branch is dead: removing it passes the suite (157 passed). "Check pairwise/checklist first", "Also check pairwise/checklist refuse --judges combination" and "Additional check" say what the code does, not why. The loop at :244-252 also partly repeats unreproducible_arms(..., with_material=True) and defines the baseline by name (arm != baseline), where that function uses arm_kind. "stored" at :249 is SOURCE_STORED in cross_judge.py.

Suggestion: if args.pairwise or args.checklist: unrepro.update(unreproducible_arms(data)), elif args.judges: ..., with the store check inside one function.

[napetrov]

The dead inner pairwise call was removed in a382bc4 (if args.pairwise or args.checklist ... elif), and SOURCE_STORED replaces the literal. In 0c9d32e the gate loop is the single place that compares the score's digest with what from_artifact serves, with comments that say why. The store is now built once, only when --judges is set, and is reused as the judge's store, so the warning prints once and --checklist-only runs no longer build it (test_cross_judge_builds_the_material_store_once_and_only_for_judges).

[napetrov]

On the review-body points:

• antonwolfy point 1 (gate vs served digest, null qid, no answer record): The --judges gate now compares each stored score's judge_context_sha256 with JudgeContextStore.digest_of(qid, arm), the digest the reader serves, so scores citing D1 and records citing D2 (both in judge_materials) exit 2 (test_cross_judge_judges_refuses_a_score_citing_other_material_than_the_records_serve). A null or non-str/int question_id is unresolvable and exits 2, with no TypeError for ["a"] (..._null_qid case, test_cross_judge_judges_exits_2_for_an_unhashable_question_id). An evaluation whose id has no answer record is skipped, since score_measure never scores it (test_cross_judge_judges_skips_an_evaluation_with_no_answer_record). Commit 0c9d32e.

• antonwolfy point 2 (pairwise/checklist with materials, --no-stored, answers-only): The two pairwise/checklist tests now build their file with _material_run() (judge_materials plus digests), so they test "materials present". New CLI tests cover an answers-only file and --no-stored with --judges, and assert the stored text reaches the judge 8 times out of 8; a with_material=True pairwise mutant, a "build only if stored_measure" mutant and a "build only without --no-stored" mutant each fail a test (0c9d32e).

• antonwolfy point 3 (stale wording): A digest mismatch and an unreadable record now have their own messages that name the arm and the questions (for example 'Q5'), and the generic message no longer says "a file written before them". The get() warning in judge_context.py now says the run did not keep the text or it failed its digest. Tested for the question names and for the absence of the old phrase (0c9d32e).

• antonwolfy point 4 (malformed files, duplicate warning): The gate loop checks isinstance(qid, (str, int)) and from_artifact checks isinstance(answers, list) (the latter from a382bc4/de4e5148), with a test for each. from_artifact is built once and only for --judges, so the "cite judge material that is missing" warning prints once and not for --checklist-only runs.

[antonwolfy]

Review at 0c9d32e. The round trip holds, and most of the 4ba3c6d findings are closed: the --judges gate now compares the score's digest with the one the reader serves, the store is built once and only for --judges, and the pairwise/checklist tests now use a file that has materials. Two risks are left, one in the gate and one in the tests, plus two points from my last review that are still open. I'd fix those before merge.

What I ran, on a Linux review host at 0c9d32e:

• make ci-local: 19 passed, 0 failed, 4 skipped. 4,327 tests passed, 1 skipped; changed-line coverage 94% (137 changed lines measured). schema_compat passes, with the one expected WARNING for the new judge_materials property. The run used Python 3.13; the other matrix legs were not run.
• Reproductions for the code findings, with the LLM transport mocked (eval cross-judge --judges driven through the real parser with a fake Judge).
• A mutation run on the gate, the store and unreproducible_arms against test_arm_runner.py + test_judge_reliability.py (171 tests). The mutants that pass the full suite are named below.

Still open from my earlier review:

• judge_reliability.py:256: the gate still defines the baseline by name (arm != baseline), where unreproducible_arms uses arm_kind. With --baseline-arm skill, the skill arm is never checked. Repro: 4 questions, the skill record for Q2 cites a digest whose text fails its hash. Default baseline: exit 2, "no usable material can be read for question(s) 'Q2'". With --baseline-arm skill: exit 0, s2 is judged on (No documentation retrieved) and s0, s1, s3 on the full body, and the only trace is a log warning. A score that cites another held text (Q1) behaves the same way: exit 2 by default, exit 0 with the flag. Suggestion: skip by arm_kind == "baseline" as unreproducible_arms does, or check every arm.
• judge_context.py:231 and cross_judge.py:168-184: the guards from my last review are still untested. Deleting the isinstance checks on the answer record, arms, the evaluation and scores in unreproducible_arms passes all 171 tests. Deleting isinstance(arm, dict) in from_artifact passes too, so arms: {"a": 5} would raise AttributeError with a green suite. One-line tests that expect a fall back and no exception for each shape would close these.

Checked, no issues

• Gate vs reader: altered material text, records without a digest, records and scores citing different held digests, a null or list question_id on an evaluation, and one damaged question among eight all exit 2 and name the arm and the questions. Damage on an evaluation with no answer record is skipped, as score_measure never scores it (but see the first inline comment).
• Round trip: a judged run through ArmRunner judge, build_output, save (schema validation passes) and reload gives identical contexts, all scores stored, stamp treatment_material.v1. Plain text, NUL plus unicode, and a lone high or lone low surrogate all verify against their digest.
• Flag combinations: --judges with --pairwise or --checklist exits 2. --no-stored skips the gate and the judge still gets the material. An answers-only file runs. A --checklist-only run no longer builds the store, and on the --judges path the "cite judge material" warning prints once.
• Repo rules: arms.v1 widened in place with an optional property, key pattern matches digest_text output; no with_docs/without_docs change, no new doc page needing a nav: entry, no generated artifacts, not stacked, git diff --check clean. The new tests run offline.

[antonwolfy]

The with_material=True call refuses a file the loop below deliberately accepts. The loop skips an evaluation that has no answer record (score_measure never scores it, :247), but unreproducible_arms checks every stored score without looking at the answers per question.

Repro: 4-question skill run with intact materials. Delete answers[3] and point evaluations[3].scores.skill.judge_context_sha256 at a digest that is not in judge_materials.

• eval cross-judge --judges ...: exit 2, "the stored judge saw treatment material for skill ... which these arms do not have".
• The same file with the unreproducible_arms(data, with_material=True) call removed: exit 0, the 3 answered questions are scored (6 judge calls).
• test_cross_judge_judges_skips_an_evaluation_with_no_answer_record (:981) passes only because the digest it leaves on the orphan score is held.

Mutation check: removing this call passes the full suite (171 passed), so no test needs it. The loop already covers a score digest that the reader does not serve.

Suggestion: drop the call, or count only evaluations whose question has an answer record, so the two checks cannot disagree.

[antonwolfy]

Nit: this text says the arms do not have judge_materials, which is false when --pairwise or --checklist refuses a complete file.

Repro: a file with every digest and judge_materials intact, eval cross-judge --answers run.json --pairwise pw.json. Exit 2, with "a re-scored judge sees only the text in the file's judge_materials, which these arms do not have". The arms do have it; the pairwise judge refuses because it sees the answers alone. A user reading it regenerates a good file. For a file with no judge_materials at all, the same message is followed by a second error that lists every question.

Suggestion: build the sentence from the cause. For pairwise/checklist: "a pairwise or checklist judge sees only the answers, not the arm's treatment material". For --judges: "the file's judge_materials does not hold the text the stored judge saw".

[antonwolfy]

Three gate mutants still pass the suite, and each brings back the stored-versus-re-scored mix. Mutation check on judge_reliability.py, each run against test_arm_runner.py + test_judge_reliability.py (171 passed):

• answered built with isinstance(question_id(r), str) instead of (str, int) (:249). The gate then skips every int-id question, so damage on one goes unchecked. test_cross_judge_judges_handles_falsy_question_ids (:1034) only runs an intact file.
• the gate loop without s.get("judge_context_source") == SOURCE_STORED (:255). No --judges test has a snippet_fallback score, which is the case the treatment_material.v1+snippet_fallback stamp exists for.
• a second JudgeContextStore.from_artifact(data) inside the elif materials is not None branch. test_cross_judge_builds_the_material_store_once_and_only_for_judges passes --no-stored (:1024), so the gate never runs in it and the duplicate warning goes unnoticed.

Tests to add: the one-damaged-question-among-eight file with int ids; a --judges run where one skill score is snippet_fallback (no digest) and expect exit 0; and the "once" assertion on a run without --no-stored.

[antonwolfy]

Nit: dokimi eval arm-runner is not a command, and the material is kept for unjudged runs too.

The arms run is dokimi arms run (commands/arms.py:206, --judge at :241). ArmRunner.build_output writes judge_materials "whether or not the run was judged", and test_an_unjudged_run_still_keeps_the_material pins it. As written, a user who runs arms run without --judge on a docs arm expects no growth and gets up to 15,000 characters per question. The same gap is in evaluating-treatments.md.

Suggestion: "dokimi arms run now keeps the exact text the judge reads (or would read) for each arm and question, with or without --judge".

[antonwolfy]

Nit: the paragraph has two unwrapped lines, 132 characters at :43 and 103 at :45, between lines wrapped at about 80. My last review asked for the long line to be wrapped; the rewrite moved it.

The "+0.3%" figure also counts only the one copy of the skill body. Each arm record and each stored score carries a 108-byte judge_context_sha256 line, and the baseline arm gets one too (the digest of the empty string). That is about 430 bytes per question for a baseline plus skill run, roughly 2% of the 1.71 MB artifact at 81 questions. Say "+0.3% for the text itself".

[xaleryb]

Re-ran at ad37e05: the lone-surrogate digest, the damaged from_artifact shapes and the "in memory only" docstring are fixed, the orphan-evaluation refusal is gone, and the pairwise message names the cause. Full suite, ruff, mypy, bandit and schema_compat are clean on the merge with main.

One gap remained: the --judges gate exempted the arm named by --baseline-arm. With --baseline-arm skill, a skill record whose material fails its digest (or a score citing another held text) exited 0 and that question was re-scored on snippets, while the default baseline exits 2 on the same file. Pushed 097f872: the gate exempts by arm_kind == "baseline", as unreproducible_arms does, and falls back to the name only when no kind is recorded; the two new tests fail without it.

Stack note: #73 is based on 656cf2c, not this head, and conflicts with it in 5 files (judge_reliability.py, cross_judge.py, judge_context.py, utils.py, test_judge_reliability.py). The isinstance(arm, dict) guard is still untested (minor).

[xaleryb]

Approving at 097f872: our earlier items and the review points reproduce as fixed (lone-surrogate digest, damaged from_artifact shapes, docstring, orphan-evaluation refusal, pairwise message), the --judges gate now exempts the baseline by kind, and all checks pass. Only the untested isinstance(arm, dict) guard remains (minor). #73 must rebase onto this head (see the stack note).