[napetrov]

Part of #48.

Summary

The gate judged every answer with no context; an arms run judges skill and docs arms with their treatment material. So the setting the arm deltas come from was never validated.

 eval judge-validity
+  --context none | stored            # default none: unchanged
 run_validity / run_sequential (…, store=JudgeContextStore.from_artifact(data))
   _build(pairs)
+    attach_stored_context(pairs, store)   # pair["context"] = arm's stored text
+                                          # bases without material: dropped, counted
   _score_pairs
-    scorer(lib, q, answer)
+    scorer(lib, q, answer, context=pair["context"])   # only when the pair has one
 ValidityReport
+  context: "none" | "stored",  n_without_context

Intact and degraded copies of a pair get the same material. A baseline arm's empty material becomes the no-context marker, as in the arms run. The typed and pairwise scorers have no context slot and are refused before any model loads. A file without materials exits 2. A run left with no usable pair now exits 2 with the reason, instead of a traceback.

Evidence

Perf golden set, Sonnet answers, both arms, Haiku 4.5 judge, 30 bases, 4 variants, --context none vs --context stored. The skill arm's material was added to the stored run from the skill as it is today. That rebuild is an approximation (the skill may have changed since that run), stored judge_materials make it exact going forward.

bases	intact score	contradiction	truncate_20	wrong_numbers	wrong_tools
skill arm (material differs), n 14–18	80.3 → 89.3	0.94 → 1.00	0.89 → 1.00	0.86 → 0.71	0.92 → 0.83
baseline arm (identical input, control), n 7–12	87.5 → 84.8	1.00 → 0.91	1.00 → 1.00	0.79 → 0.86	0.94 → 0.61

• The level shift is the robust result: +9.0 on the skill arm against −2.5 on the control.
• Per-variant accuracy differences are not resolved at this size. The control, which got the same input twice, moved as much as the skill arm (wrong_tools 0.94 → 0.61). Paired McNemar on the skill arm: all p ≥ 0.5. Reading discrimination under context needs a much larger run, which this flag now makes possible.
• Tests:
  • every member of a pair receives its arm's material;
  • the baseline gets the no-context marker;
  • bases without material are dropped and counted;
  • no material at all raises;
  • without a store the scorer is called exactly as before;
  • judge_scorer forwards context only when given;
  • sequential mode takes the store;
  • the CLI refuses typed / pairwise / no-materials with exit 2 before loading anything;
  • end to end through cli.py with a fake judge, the skill text reaches the judge and the report says context: stored.
• pytest -q 4309 passed, 2 skipped; ruff check ., mypy clean.

Merge Danger

Door: two-way. New flag, default off.

Blast Radius: opt-in

• Without --context stored, nothing changes: same pairs, same scorer calls, same report plus two fields (context: "none", n_without_context: 0).
• With it, the pair set can be smaller (bases without material are dropped). The report says how many, and the gate decision is made on what remains.
• Small behaviour change for everyone: "no usable pairs" is now exit 2 with a message, not a ValueError traceback (exit 1).

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — evaluating-treatments.md, benchmarking-and-comparison.md
• No ad-hoc run artifacts committed

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.09302% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.46%. Comparing base (61e0355) to head (3ea012a).
⚠️ Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/eval/degrade.py	90.00%	0 Missing and 2 partials ⚠️
dokimi/dokimi/eval/judge_validity.py	96.92%	2 Missing ⚠️
dokimi/dokimi/eval/cross_judge.py	96.96%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #73      +/-   ##
==========================================
+ Coverage   91.29%   91.46%   +0.16%     
==========================================
  Files         216      216              
  Lines       26565    26877     +312     
  Branches     4337     4411      +74     
==========================================
+ Hits        24252    24582     +330     
+ Misses       1645     1631      -14     
+ Partials      668      664       -4     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/evaluate.py	83.80% <100.00%> (+0.90%)	⬆️
dokimi/dokimi/commands/judge_reliability.py	89.06% <100.00%> (+0.96%)	⬆️
dokimi/dokimi/eval/judge_context.py	98.41% <ø> (ø)
dokimi/dokimi/eval/judge_sequential.py	100.00% <100.00%> (ø)
dokimi/dokimi/eval/validity_scoring.py	94.25% <100.00%> (+0.27%)	⬆️
dokimi/dokimi/report/model_compare.py	89.66% <ø> (ø)
dokimi/dokimi/eval/cross_judge.py	99.47% <96.96%> (-0.53%)	⬇️
dokimi/dokimi/eval/degrade.py	98.20% <90.00%> (-0.66%)	⬇️
dokimi/dokimi/eval/judge_validity.py	98.33% <96.92%> (+0.49%)	⬆️

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

Ran the stack (#73 6c88dba over #69 656cf2c; #73~1 is #69's tree) with main-vs-PR artifact round trips, tamper cases, stub-judge gate runs and three small live haiku runs. --context none is identical to main apart from the two announced fields; a file written by main loads under #73 and --context stored on it exits 2 with a clear message; a #73-written file runs under main with identical numbers; schema_compat shows one warning (new optional judge_materials); a skill body is stored once; the live runs show the stored material (a canary string) in the judge prompt for stored context and not for none.

Two defects, fixed in commits pushed here (red before):

1. 41316ba: the with_material guard in cross_judge.unreproducible_arms accepted any digest that is merely a key of judge_materials. A material edited after the run passed, and the re-scored judge silently read 200-char snippets. It now requires the text to hash to its key, as from_artifact does.
2. 497a489 (belongs to feat(arms): keep the judge's treatment material in the artifact #69's digest_text): a lone surrogate in material (JSON \ud83d) raised and aborted the judge phase before the artifact was written. It is now hashed with surrogatepass; digests of valid text are unchanged.

Open, your call:

• _build() and run_sequential sample --bases N before dropping bases without material: 3 arms x 40 questions, docs digests removed, --bases 30 gives 20/20/18 bases while 80 eligible existed; with a whole arm dropped the stub-judge gate flips FAIL to PASS on the remaining bases; the report names no arms and says "none stored" even for digest failures, and both warnings say "falls back to snippets", which is false on this path. Sample after filtering and print per-arm kept and dropped counts; refuse when a whole arm drops.
• The label suffix " [stored context]" means a stored report never matches in apply_validity: cross-judge --validity <stored report> --require-validated says "matches no measure", and a context=none PASS is applied to measures scored with material. Match on (label, context).
• feat(arms): keep the judge's treatment material in the artifact #69: a list or dict judge_context_sha256, a non-dict record or an unhashable question_id still raise in from_artifact (hand-damaged files only), and the cross_judge.py docstring still says the material is kept "in memory only".
• Artifact growth is linear, about 5-10 KB per docs question (449 KB to 844 KB on 40 questions).

Landing order: #69 first, then #73 rebases.

[antonwolfy]

Review at 497a489. The core is sound: intact and degraded copies get the same material, a baseline arm gets the no-context marker, the typed and pairwise refusals happen before a model loads, and --context none is unchanged. Two things are worth fixing before merge: the [stored context] label means a stored-context report can never gate a cross-judge run, and --bases N is sampled before the bases without material are dropped.

xaleryb's two open points (sampling before filtering, and the label suffix not matching in apply_validity) are still unaddressed at this head. Both are reproduced below.

What I ran, on a Linux review host at 497a489:

• make ci-local: 18 passed, 1 failed, 4 skipped. 4311 tests passed, 1 skipped. Changed-line coverage 95% (118 lines). ruff, mypy, bandit, pip-audit, mkdocs build all pass. The one failure is schema_compat: it reports tool_preference*.v1.json as removed because origin/main here has moved past this branch's base (8193ce0). The only line that concerns this PR is a warning for the new optional arms.v1.json judge_materials, which is a widening and needs no version bump.
• Reproductions for the code findings, with the LLM transport mocked (fake judge scorers, no model calls).
• A mutation run over the new code: 85 mutants, 28 survive the three test files and the same 28 survive the other test files that import these modules; about 19 are real gaps. The ones I re-ran are called out inline.
• The PR's new tests against the base modules: 17 of 18 new test items fail there, as they should. The one that passes is the "called exactly as before" guard.

Findings that cannot be anchored inline:

1. The PR is stacked on #69. git log origin/main..HEAD holds 656cf2c, and AGENTS.md says PRs are not stacked on each other's branches. The description already says to rebase after #69 lands; the diff reads as the sum of both until then. judge_context.py also ends with a blank line at EOF (\n\n).
2. Nit: some statements are now stale. cross_judge.py:160 still says the arms run "keeps it in memory only" and that a re-scored judge sees the 200-char snippets (true only for files without judge_materials). The get() warning in judge_context.py says the same, and report/model_compare.py:310 says "re-judging an answers file only sees the 200-char report snippets". docs/evaluating-treatments.md:64 says bases "whose arm has none" are dropped; the code drops bases with no stored entry, and a baseline arm (empty material) is kept and judged with the no-context marker.
3. Nit: n_noop counts no-ops from bases that were then dropped. With 30 no-ops and 10 bases dropped for lack of material, the summary printed 30 no-op for 20 scored bases (build_pairs counts before attach_stored_context).

Checked, no issues

• Schema: judge_materials is an optional addition to arms.v1.json, the same in-place widening judge_context_sources got. A generated artifact validates and saves. No SCHEMA_VERSIONS change, no with_docs/without_docs change, no mkdocs.yml change needed.
• Pairs: every member of a pair, mismatched_answer included, is scored with its base's arm material; the intact cache is keyed by (base_id, arm).
• Refusals: --scorer typed:... and --mode pairwise exit 2 with Judge never constructed. A list-shaped answers file and a file without judge_materials exit 2. --sequential --context stored runs and reports context: stored.
• Digests: from_artifact skips text that does not hash to its digest; digest_text with surrogatepass leaves valid-text digests unchanged.
• Fixes 41316ba and 497a489: the digest check in unreproducible_arms and the surrogate encoding are both killed by the mutants I tried.

[napetrov]

On the review-body points:

• antonwolfy point 1 (stacked on feat(arms): keep the judge's treatment material in the artifact #69): Yes, this PR is stacked on feat(arms): keep the judge's treatment material in the artifact #69 and I will rebase once feat(arms): keep the judge's treatment material in the artifact #69 lands. The review cannot be treated as final until then.

• antonwolfy point 2 (stale statements): The stale "in memory only" / 200-char snippet wording at cross_judge.py, the get() warning in judge_context.py and docs/evaluating-treatments.md is corrected (409b5a7, f33b74c). The docs now say that bases with no stored entry are dropped and that baseline is kept with the no-context marker. The trailing blank line at EOF of judge_context.py is removed.

• antonwolfy point 3 (n_noop): The dropped count now comes from build_pairs with the real min_chars and only covers the filter. All-no-op runs report the no-op error rather than "no material" (12dcd71, test_all_noop_with_a_store_reports_noop_not_missing_material).

[antonwolfy]

Review at 81725fa. Seven of the eight earlier threads are fixed, and the fixes are tested: the label is provider/model again, --bases samples after the filter, a dropped arm is refused, NoUsablePairs keeps real ValueErrors loud. What is left sits in the new (label, context) matching and in one report field that is wrong in sequential mode. One earlier thread (unreproducible_arms) is still open. I'd fix the first three inline comments before merge.

What I ran, on a Linux review host at 81725fa:

• make ci-local (Python 3.13): 19 passed, 0 failed, 4 skipped. 4,325 tests passed, 1 skipped. Changed-line coverage 98% (156 lines). ruff, mypy, bandit, pip-audit, schema_compat and the mkdocs build pass.
• Reproductions for the code findings, with the LLM transport mocked (fake judge scorers, no model calls).
• A mutation run over the new code: 44 mutants, 43 killed by the touched test files. The survivor is the n_without_context assignment in run_sequential (see the inline comment).
• The new tests against the base modules: they fail there, as they should.

Still open from my earlier review:

• cross_judge.py:184: unreproducible_arms(with_material=True) decides from each score's judge_context_sha256, while JudgeContextStore.from_artifact reads the digest from the answer's arm record. With the arm digests removed and the scores and judge_materials intact, the guard returns [], cross-judge --judges exits 0, and the re-scored judge sees 'snip' where the intact file gives 'FULL SKILL BODY...'. You said you would move this into #69; it is not in this branch yet. Non-blocking.

Findings that cannot be anchored inline:

1. The PR is still stacked on #69. git log origin/main..HEAD starts with 656cf2c. You said you will rebase once #69 lands; the diff reads as the sum of both until then.
2. Nit: the comment in commands/judge_reliability.py:223 is stale. It says every other measure "sees at most the report snippets", but a re-scored judge now gets judge_materials, as the lines below say.
3. Nit: the --context help text, the changelog fragment and both docs pages list fewer exit-2 cases than the code. --context stored with --mode pairwise or a typed scorer also exits 2 (evaluate.py:284). The docs say eval pairwise / eval checklist pass no context by default, but those commands have no --context flag.

Checked, no issues

• Label and matching: a --context stored PASS or FAIL report gates both the stored measure and the --judges measure that saw material. A --context none report gates only the no-material measure. A report with a (stored) suffix matches nothing, as on main.
• Sampling: with 27 of 120 eligible answers holding material, --bases 10 and --bases 20 score exactly 10 and 20 bases, fixed and sequential. --bases 30 scores all 27. n_without_context is not double counted, and by_arm reconciles with it, duplicate rows included.
• Refusals: a whole arm without verified material, a baseline-only artifact, typed and pairwise scorers, and --judges with edited material or with --pairwise all exit 2 before any scorer call. A bug ValueError inside a degradation is not caught.
• Schema and repo rules: judge_materials is an optional addition to arms.v1.json, so schema_compat reports a widening, not a break. No generated artifacts, no mkdocs.yml change needed, all commits signed off.
• Tests: dropping store=materials in the cross-judge wiring, never skipping on context, and dropping with_material=True are each caught now.

[antonwolfy]

Review at 1210bd3. Nothing in the code changed since my last review at 81725fa; the only new commit adjusts how two tests assert the exit code, and it does not weaken them. All nine earlier threads still apply, so this is not ready to merge yet: the --sequential report blames the cap when the pool ran out, and the (label, context) matching in cross-judge has two gaps.

What I ran, on a Linux review host at 1210bd3:

• make ci-local (Python 3.13): 19 passed, 0 failed, 4 skipped. 4,325 tests passed, 1 skipped. Changed-line coverage 98% (156 lines).
• Reproductions for every open thread below, with the LLM transport mocked (fake judge scorers, no model calls).
• The mutation from my earlier review, report.n_without_context = dropped[0] -> = 0 in run_sequential: all 384 tests in test_judge_validity.py and test_judge_reliability.py still pass.

Still open from my earlier review:

• cross_judge.py:184: unreproducible_arms(with_material=True) reads digests from the scores, from_artifact from the arm records. With the arm digests stripped it returns [] and the re-scored judge sees snip (4 chars) where the intact file gives the 320-char material. Non-blocking: hand-edited or merged files only. (risk)
• judge_sequential.py:177: stopped_by is cap when the pool with material ran out. 100 long skill answers, 10 with material, --max-bases 50 reports "cap reached" after 10 pairs; with all 10 holding material it reports pool. (bug)
• cross_judge.py:189: stored_measure is always context="stored". A snippet_fallback or pre-judge_context file can only be gated by a none report, and that report now matches nothing (gate: None, "matches no measure"). (risk)
• cross_judge.py:301: a FAIL does not cross contexts. A --context stored FAIL leaves the coverage and pairwise measures voting; a --context none FAIL leaves the grounded measure voting. The docstring and benchmarking-and-comparison.md:399-400 say otherwise. (risk)
• judge_sequential.py:145: nothing pins n_without_context in sequential mode (mutant above survives). (risk)
• judge_validity.py:361: "N bases dropped" and by_arm count answers before --bases sampling, not bases. (nit)
• degrade.py:655: row in has_material is a list scan; build_pairs with a store takes 0.21 s at 5k rows, 0.84 s at 10k, 3.49 s at 20k (no store: 0.02 s at 20k). (nit)
• benchmarking-and-comparison.md:402: the stored measure is provider/model (stored), not provider/model. (nit)
• evaluate.py:344: the "this file has none" error is printed for files that do hold judge_materials but no usable text (only the empty baseline text, edited text, or no record citing a matching digest). (nit)

Findings that cannot be anchored inline:

1. Nit: the exit-2 cases are still listed incompletely. The --context help (evaluate.py:500-503), the changelog fragment (line 7) and both docs pages name only the lost-arm and no-usable-pair cases. --context stored with --mode pairwise or a typed scorer (evaluate.py:284) and with a file that has no usable material (evaluate.py:344) also exit 2. benchmarking-and-comparison.md:410 and evaluating-treatments.md:62 say eval pairwise / eval checklist pass no context "by default", but those commands have no --context flag. I raised this at 81725fa; it is unaddressed.
2. Nit: report/model_compare.py:310 (outside this diff) still says re-judging an answers file "only sees the 200-char report snippets". That is false for a file with judge_materials.
3. The PR is still stacked on #69. git log origin/main..HEAD starts with 656cf2c. You said you will rebase once #69 lands.

Checked, no issues

• Test change in 1210bd3: code stays None unless a SystemExit is raised, and the command exits only 1 or 2, so assert code in (None, 1) accepts what exc.code == 1 accepted. The exit-2 refusal is pinned by test_cli_refuses_stored_context_it_cannot_honour.
• n_noop: counts only kept bases now (20 long answers, 10 with material, 5 no-op for wrong_numbers -> n_noop {'wrong_numbers': 5}).
• Exit 2: only NoUsablePairs is caught; a bug ValueError in a degradation stays loud.
• Schema and repo rules: judge_materials is an optional addition to arms.v1.json; no SCHEMA_VERSIONS, with_docs / without_docs or mkdocs.yml change needed; no generated artifacts; tests are offline.
• digest_text with surrogatepass: artifacts are written with ensure_ascii=True, so a lone surrogate round-trips; digests of valid text are unchanged.

[xaleryb]

Re-ran at 40ffb49: sampling after the filter (30 of 30 bases on every seed, was 19-23), the whole-arm refusal, the warning text and (label, context) matching are fixed, and so are stopped_by, the stored_measure context, FAIL across contexts (order-independent) and the build_pairs cost (20k rows in 0.14 s). Our two commits are intact and still red without their fixes.

Remaining for the rebase: this branch sits on #69's first commit 656cf2c, not on #69's head, and conflicts with it in 5 files. On this branch alone --judges with the arm digests stripped exits 0 and re-scores on snippets; #69's gate refuses the same file. When rebasing, take #69's gate (now also exempting the baseline by kind, 097f872 on #69), drop our 41316ba (superseded: #69 removed with_material, which its test calls), and keep only one of our 497a489 and #69's 0c8dfeb (same surrogate fix).

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

9 open issues

• Measure.context is documented as “stored when every score saw the stored material, none when none… · New
• score_measure() sets Measure.context based on whether any non-empty material exists in the…
• The stored-context plumbing relies on two “hidden channel” mutations: (1) dropped as a… · New
• The stored-context plumbing relies on two “hidden channel” mutations: (1) dropped as a… · New
• The stored-context plumbing relies on two “hidden channel” mutations: (1) dropped as a… · New
• Relying on a dict-key check plus # type: ignore[call-arg] makes it easy for signature mismatches… · New
• When pairs carry context, _call() unconditionally invokes the scorer with a context= kwarg.…
• Using a sentinel key like \"_kept_count\" inside the public-ish by_arm structure makes the data…
• This error is raised from run_validity() (fixed-n mode), but the guidance mentions --max-bases,…

_{Review effort: Lite}

[antonwolfy]

Review at ccf04a1. The --scorer cleanup is right, and the fixed-mode refusal now catches an arm whose bases were all no-ops. Two of the three threads from my last review are not closed, and one earlier point (a re-scored measure labelled stored) is still in the code. I'd fix those before merge.

What I ran, on a Linux review host at ccf04a1:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,219 tests passed, 1 skipped. Changed-line coverage 100%. ruff, mypy, bandit, pip-audit, schema_compat and the mkdocs build pass.
• Reproductions for the code findings and for each open thread, with the LLM transport mocked (fake scorers, no model calls).
• A mutation run over the new tests: sum(1 for _, a in scored_bases) without if a == arm passes all 430 tests in test_judge_validity.py and test_judge_reliability.py.

Findings that cannot be anchored inline:

1. A re-scored measure is still labelled stored when some answers were judged on snippets, so a --context stored PASS validates it. I raised this in my last review body; cross_judge.py did not change. score_measure sets context="stored" as soon as the store holds any non-empty material (cross_judge.py:282-283) and ignores the per-answer source that resolve_with_source returns. Two answers, one with intact material and one whose material was edited after the run:

seen [('a', 'TREATMENT MATERIAL'), ('b', 'snip2')]
context stored
apply_validity(stored PASS) -> gate PASS
stored_measure on the same file -> mixed

Through the CLI, cross-judge --no-stored with a stored PASS report shows ✓ in the Gate column for the arm judged on a snippet. Suggested fix: derive the context from the sources of all scored jobs, with the rule stored_measure uses (stored if all, none if none, mixed otherwise). test_judge_reliability.py covers all-material, empty-store and no-store only; add a store with one arm edited.

Still open from my earlier review:

• judge_validity.py / judge_sequential.py:190-192: the starved-arm refusal exists only in run_validity. run_sequential has none, and the old sampled check was removed from _build. 100 baseline answers and 3 skill answers with material, max_bases=50, variants=["truncate_20"], fake length scorer: seeds 0, 1, 3 and 5 of 8 finish PASS with skill scored: 0; the summary shows per arm : skill: 3 kept, 0 dropped, 0 bases scored and RESULT: PASS. The help text, changelog and docs promise exit 2 for an arm that "gets none sampled". Suggestion: run the check in _build on the arms present in pairs (see the inline comment), and in sequential mode on scored. (bug)
• judge_sequential.py:191: the per-arm filter in scored is still not pinned. test_sequential_per_arm_scored_counts_differ uses one arm, so it cannot tell the filtered count from the total. Mutation check: dropping if a == arm passes all 430 tests. The test needs two arms where one is scored on fewer bases than the other, asserting the two scored values differ. (nit)

Checked, no issues

• --scorer check: --scorer bogus, with and without --context stored and with --mode pairwise, exits 2 with unknown --scorer 'bogus' and never builds a Judge. --scorer typed:... and --mode pairwise with --context stored still exit 2 with their own messages. The changelog no longer claims the check as new.
• _kept_count: it does not reach by_arm in the returned report, the summary or to_dict() in fixed or sequential mode.
• Fixed-mode no-op case: variants=["strip_code"], 10 baseline answers with a fence and 3 skill answers without one, raises NoUsablePairs naming ['skill'].
• Repo rules: no generated artifacts or uv.lock tracked, no foreign commits on the branch, one changelog fragment, all commits signed off, no schema or mkdocs.yml change needed, tests offline.

[xaleryb]

Re-ran fbb7017 and its merge with main: full suite, ruff, mypy, plus reproductions with a fake scorer (no model calls).

• Open: under --sequential --context stored an arm can be in the pool and still get no base scored, because the SPRT stops early. Run: 100 baseline answers plus 3 skill answers with material, --max-bases 50, truncate_20. Seeds 0, 1 and 5 give skill: sampled 1, scored 0 and RESULT: PASS. Expected: exit 2, as the help, changelog and docs promise. The options are to refuse after the loop, to keep scoring until every arm has a base, or to order the pool round-robin by arm. This choice is yours.
• Pushed a3ba73b: sequential mode now runs the same unpaired-arm refusal as fixed mode, before any scorer call. Before it, --max-bases could leave the only skill base out of the pool and the run still passed. The new test is red without the fix.

[antonwolfy]

Review at a3ba73b. The changes since ccf04a1 are right: score_measure now derives stored / none / mixed from the per-answer source, the starved-arm refusal runs before any judge call, and _build returns Built. One bug remains in --sequential, and two nits.

What I ran, on a Linux review host at a3ba73b:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,223 tests passed, 1 skipped. Changed-line coverage 98% (160 lines). ruff, mypy, bandit, pip-audit, schema_compat and the mkdocs build pass.
• Reproductions for the code findings, with the LLM transport mocked (fake scorers, no model calls).
• Mutation runs over refuse_unpaired_arms and score_measure: kept > 1, dropping a not in paired_arms, swapping the {True} / {False} branches, mapping mixed to stored or none are all killed. The empty-flags mutant survives (inline).

I agree with @xaleryb's point on the sequential early stop (comment of 2026-10-20). New evidence: seeds 0, 1 and 5 of 8 finish PASS with skill: scored 0 at this head, a3ba73b included.

Checked, no issues

• mixed context: a store with an edited or missing digest for one arm gives mixed in score_measure; a stored PASS leaves gate=None and a stored FAIL still gates. Real arms files cite the empty baseline digest (arm_runner.py:288), so baselines resolve as stored.
• require_context_scorer: accepts judge_scorer(...), a MagicMock and a partial over a function that takes context; refuses a partial or callable object without it.
• Fixed mode: the refusal runs on the built pairs before any scorer call; a strip_code no-op on the skill arm raises NoUsablePairs naming ['skill'] with zero calls.
• Repo rules: no schema change, no mkdocs.yml change needed, one changelog fragment, no generated artifacts, tests offline.

[antonwolfy]

Review at 3aacf7c. The two nits from my last review are fixed and tested. The sequential-mode bug is still there, so this is not ready to merge.

What I ran, on a Linux review host at 3aacf7c:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,224 tests passed, 1 skipped. ruff, mypy, bandit, pip-audit, schema_compat and the mkdocs build pass.
• A reproduction of the sequential thread with a fake length scorer (no model calls).
• A mutation run over the score_measure context rule: empty-flags -> stored / mixed, mixed -> stored / none, swapped branches and constant flags are all killed by the touched test files.

Still open from my earlier review:

• judge_sequential.py:143: --sequential --context stored still finishes PASS with an arm that has verified material and no scored base. a3ba73b checks the built pool only. 100 baseline answers, 3 skill answers with material, max_bases=50, variants=["truncate_20"], seeds 0, 1 and 5 give skill: kept 3, sampled 1, scored 0 and a PASS on 20 baseline bases; seeds 2, 4, 6 and 7 score 1-2 skill bases, seed 3 is refused. The help, changelog and both docs pages promise exit 2 for an arm that "gets none sampled". Suggestion: after the loop, refuse (or warn on stderr) when an arm with kept > 0 has scored == 0, or keep scoring until every pooled arm has one base, and add a sequential test over a few seeds.

Checked, no issues

• Empty-measure label: test_score_measure_with_no_scored_answer_is_context_none_even_with_a_store fails when the empty-flags branch is changed to stored or mixed.
• Refusal wording: both the starved-arm and lost-arm messages now end "raise , or remove those arms from the answers file"; no other "drop those arms" is left in the code, docs or changelog.

[vlad-perevezentsev]

This branch no longer merges with main: #80 replaced the dict reads here with records.arm_scores() and Score objects, so stored_measure and score_measure conflict. Please rebase and read judge_context_source from Score.raw (and the answer from Answer.raw in score_measure).

[napetrov]

Fixed in 50174e3. Merged main instead of rebasing, so the push needs no force. The conflict is in cross_judge.py: stored_measure now reads evaluations through arm_scores and takes judge_context_source from Score.raw, and score_measure takes the answer from Answer.text and resolves context from dict(Answer.raw). The branch merges cleanly with intel/main now. Tests: full suite green apart from one unrelated socket-guard failure in test_pipeline_question_set.py in this sandbox.

[vlad-perevezentsev]

When every kept base of the arm was sampled and only its degradations were no-ops, a higher --bases cannot help, but the message still says to raise it. Suggest the flag only when by_arm[a]["sampled"] < by_arm[a]["kept"], and name the no-op variants otherwise.

[napetrov]

Fixed in 50174e3. refuse_unpaired_arms now says "raise " only when an arm has a kept base that was not drawn (sampled < kept). When every kept base was sampled, the message names the requested variants that left the text unchanged and says a higher flag cannot help. Regression: test_refusal_does_not_say_raise_the_sample_when_every_kept_base_was_sampled, red on the old message.

[vlad-perevezentsev]

This also collects the context of grounded and coverage measures, which a PASS never validates.
With only a grounded "stored" measure the note says the measures are "stored", but a --context stored PASS still leaves gate=None. Only add m.context here when m.kind is "aggregate" or "pairwise".

[napetrov]

Fixed in 50174e3. apply_validity skips grounded and coverage measures for a PASS, so their context is never collected or named in the note. Regression: test_a_pass_in_one_context_never_names_the_context_of_a_grounded_measure, red on the old code.

[xaleryb]

Re-ran every earlier point's original input at d441c7a with fake judges (no model calls), plus the full suite, ruff, mypy and bandit.

• Fixed at this head: sampling happens after the filter (30 of 30 bases on 8 seeds); a whole arm without material is refused; (label, context) matching works; a re-scored measure with edited material is mixed and no stored PASS gates it; an unpooled arm is refused in sequential mode; the early stop no longer passes.
• --sequential --context stored with a judge that always gives the same score, on 100 baseline + 3 skill answers -> exit 2, no report written -> expected FAIL (exit 1) with the report written.
• The same run with a working judge at --max-bases 50, 103 or 400 -> the same refusal on the same seeds, and it says "raise --max-bases" -> expected a remedy that works.

Pushed:

• 1c4d0c6: refuse only when the run would pass; a FAIL stands. Docs and fragment updated.
• b5b420b: point the refusal to fixed mode or to removing the arm.

Left for the author: nothing in this PR. The red License header check also fails on main.

[antonwolfy]

Review at b5b420b. The sequential-mode thread is fixed: a stored --sequential run now refuses after the loop when it would pass with an arm unscored, and a FAIL still stands. One gap in how "scored" is counted is worth fixing before merge, plus two nits.

What I ran, on a Linux review host at b5b420b:

• make ci-local: 26 passed, 0 failed, 4 skipped. 6,405 tests passed, 1 skipped. Changed-line coverage 98% (171 lines). ruff, mypy, bandit, pip-audit, schema_compat and the mkdocs build pass.
• Reproductions for the findings below, with the LLM transport mocked (fake length scorers, no model calls).
• A mutation run over the new block in run_sequential: 18 mutants. 14 are killed (including dropping or inverting the passed test, and hard-coding scored), 4 survive: kept > 1, dropping sorted(...), rewording the second remedy, and the redundant store guard.

Still open from my earlier review: none.

Checked, no issues

• FAIL path: strip_code PASS with truncate_20 FAIL, and a constant-score FAIL, return the report with passed=False and no refusal. The CLI exits 1.
• Refusal path: --context stored --sequential --max-bases 50 --seed 0 --variants truncate_20 --output out.json exits 2, prints the arm on stderr and writes no output file.
• --max-bases claim: with 100 baseline and 3 skill answers, --max-bases 50, 103 and 400 give the same outcome on seeds 0-7 (refused on 0, 1, 2, 4, 5; pass on 3, 6, 7).
• Other arms and concurrency: with three arms the message names only the starved one, and concurrency=4 behaves like concurrency=1.
• Repo rules: no schema, mkdocs.yml or generated-artifact change; tests are offline.

[antonwolfy]

A base scored only by a neutral variant counts as "scored", so the gate can still pass with no graded pair from an arm.

scored_bases is filled from every pair in todo (judge_sequential.py:169), including padding, which no gate decision uses. Fixed mode refuses this case because it looks at rows. 100 baseline answers with a code fence, 3 skill answers without one, max_bases=120, variants=["strip_code", "padding"], scorer len(a):

seed 3  OK  skill: {'kept': 3, 'sampled': 3, 'scored': 1}  skill rows: strip_code 0, padding 1  passed True
seed 8  OK  skill: {'kept': 3, 'sampled': 3, 'scored': 1}  skill rows: strip_code 0, padding 1  passed True
seeds 0, 1, 2, 4, 5, 6, 7, 9 -> refused

The skill arm has no graded row, yet the run reports PASS with context: stored. Whether it refuses depends on the seed.

Suggestion: count an arm as scored only when a graded (non-neutral) pair of one of its bases was scored, e.g. build the set from p["variant"] not in NEUTRAL_VARIANTS. Tests to add: this strip_code + padding case, one skill answer instead of three (mutation check: c.get("kept", 0) > 1 passes both test files), and two unscored arms asserting the sorted list in the message.

[napetrov]

Fixed in be965f2. scored_bases is now built only from pairs whose variant is not in NEUTRAL_VARIANTS, so a base scored only by padding no longer covers its arm. Your repro (100 fenced baseline / 3 skill answers, strip_code + padding) now refuses for seeds 0-9, including 3 and 8. Tests: test_sequential_padding_alone_does_not_cover_an_arm (3 skill answers and 1 skill answer, seeds 0-9) and test_sequential_unscored_arm_refusal_names_every_arm_sorted (asserts "['alpha', 'zeta']"). All three fail on the old code.

[antonwolfy]

I pushed a commit per nit, up to 02d7397, which fix the two nits from my review:

• 2344f45: dokimi/dokimi/commands/evaluate.py:526: the --context help and the first changelog fragment now list the --sequential run that would pass with none of an arm's bases scored.
• 02d7397: dokimi/dokimi/eval/judge_sequential.py:209: the refusal now says to run without --sequential and with --bases at least the number of answers, or remove those arms.

make ci-local at 02d7397: 26 passed, 0 failed, 4 skipped.

Not approving yet. Still open: the risk from my review.

[napetrov]

@antonwolfy thanks for 2344f45 and 02d7397, both kept. The open risk from your b5b420b review (a base scored only by a neutral variant counted as scored) is fixed in be965f2, on top of your commits: scored_bases in judge_sequential.py is built only from pairs whose variant is not in NEUTRAL_VARIANTS. Your repro (100 baseline / 3 skill answers, strip_code + padding) now refuses on seeds 0-9; before the fix seeds 3, 6, 7 and 8 passed. Three regression tests in tests/test_judge_validity.py (padding-only with 3 skill answers, with 1, and two unscored arms listed sorted) fail on the old code. Details on the inline thread at judge_sequential.py:208. Ready for another look.

[xaleryb]

Re-ran every earlier point's original input at be965f2 with fake judges and no model calls. Also ran the full suite, ruff, mypy and bandit.

• Fixed at this head: sampling after the filter, the whole-arm refusal, (label, context) matching, mixed measures, the unpooled arm, the early stop (exit 2), a constant judge giving FAIL with the report written, and padding-only sequential coverage (seeds 0-9 refuse).
• --context stored --variants strip_code,padding --bases <all answers>, where strip_code does not change the skill answers -> exit 0, PASS, with the skill arm covered only by padding rows -> expected exit 2. The sequential refusal points users to this exact run.

Pushed:

• 3ea012a: only a graded pair counts as covering an arm in refuse_unpaired_arms. Both modes now refuse before any call. A padding-only run still fails as unmeasured (exit 1). Also adds a seeded test that pins be965f2's post-loop count, and updates the docs, help and fragment.

Left for the author: nothing.

[xaleryb]

Approved at 3ea012a. Every earlier point re-run at this head passes and no must-fix or mid item is open after our commit, which is red without its fix; full suite, ruff and mypy clean on Linux; CI green.