feat(judge-validity): --context stored — gate the judge the way arms are scored

dokimi/changelog.d/20261016_000000_judge_validity_stored_context.md

@@ -0,0 +1,28 @@
+### Added
+
+- **`eval judge-validity --context stored`** — judges each base with the arms
+  artifact's verified `judge_materials`, the same material the arms were scored
+  with. Bases without verified material are dropped before `--bases` /
+  `--max-bases` sampling; the summary reports `n_without_context` and per-arm
+  kept/dropped counts in `by_arm` (with `scored` bases in `--sequential` mode). The run exits 2 with a typed scorer or
+  `--mode pairwise`, on a file with no non-empty verified material, if an arm
+  loses all its bases, has only `padding` pairs or gets none sampled, if a `--sequential` run would pass
+  with none of an arm's bases scored, or if no pair is usable.
+
+### Changed
+
+- **`eval judge-validity` exits 2 when no pair is usable, in every mode** — a
+  file with no answer longer than the minimum (or where every degradation is a
+  no-op) now prints an `Error:` line and exits 2 instead of a `ValueError`
+  traceback (exit 1), with or without `--context`.
+
+- **`cross-judge --validity` matches a PASS by judge and context** — a passing
+  report validates a measure only when both the judge label and the context
+  (`stored` or `none`) agree, so a no-context PASS no longer validates a judge
+  that read the stored material. A FAIL still silences every measure of that
+  judge in any context, and a PASS never lifts it, whatever the report
+  order. Reports without a `context` key (written before this
+  change) count as `none`, and a stored measure whose scores fell back to
+  snippets is `none` too; one mixing stored and snippet scores is `mixed`,
+  which no PASS validates. `cross-judge --judges` re-scores with the
+  stored material.

dokimi/changelog.d/20261021_000000_napetrov_sequential-unscored-arm.md

@@ -0,0 +1,7 @@
+### Changed
+
+- **`eval judge-validity --context stored --sequential` exits 2 when an arm with
+  verified material got no scored base** — the SPRT can decide on the other arms
+  and stop first; the run now refuses, as fixed mode does, instead of passing
+  without covering that arm. A FAIL on the bases that were scored is still
+  reported as a FAIL (exit 1).

dokimi/docs/benchmarking-and-comparison.md

@@ -447,19 +447,30 @@ under several judges (absolute, and with `--grounded` against each question's
 measures, and labels each arm `CONSISTENT ±`, `LEANING ±`, `JUDGE-DEPENDENT` or
 `NO DETECTABLE EFFECT`, on Holm-adjusted p over the measures that vote. With
 `--validity` reports, a judge that failed its gate is shown but does not vote
-(its grounded and checklist measures included); `--require-validated` leaves
-only judges that passed. The stored scores are labelled from the file's
-`judge_provider`/`judge_model` (`provider/model (stored)`), so a `--validity`
-report for that judge reaches them and a verdict resting on it alone reads
-`(one judge)`. Re-scoring resolves each arm's context the way the arms run does,
-from the file's `judge_materials`, so a `--judges` judge sees what the stored
-judge saw. A file written before `judge_materials` has only the report snippets;
-with stored scores on such material, `--judges` refuses (exit 2), and
-`--pairwise` or `--checklist` always do, since those judges see only the answers.
-`--no-stored` compares those judges with each other. The gate itself (and `eval pairwise` / `eval checklist`) passes no
-context, so it validates a judge only in the no-context setting. A label whose significant measures all rest on one judge says `(one judge)`; an escalated pairwise measure counts as its cheap judge's. Gate a judge on
-answers from the same answer model it will judge: `cross-judge` warns when the
-validity report was measured on another model's answers.
+(its grounded and checklist measures included, in any context);
+`--require-validated` leaves only judges that passed. The stored measure is
+named `provider/model (stored)` from the file's `judge_provider`/`judge_model`,
+and a validity report names `provider/model`. A PASS validates only measures of
+the same context: a `--context stored` report for measures scored with material,
+a `--context none` report (or one without `context`) for the rest. A stored file
+whose scores all fell back to snippets counts as `none`; one that mixes stored and
+snippet scores counts as `mixed`, which no PASS validates (a FAIL still reaches it). Re-scoring resolves each arm's
+context the way the arms run does, from the file's `judge_materials`, so a `--judges`
+judge sees what the stored judge saw. A file written before `judge_materials` has
+only the report snippets; with stored scores on such material, `--judges` refuses
+(exit 2), and `--pairwise` or `--checklist` always do, since those judges see only
+the answers. `--no-stored` compares those judges with each other. The gate itself
+passes no context by default (`eval pairwise` and `eval checklist` never pass
+one), so it validates a judge in the no-context setting; `judge-validity --context
+stored` on an arms artifact gates it with each arm's stored material instead. That
+run exits 2 with a typed scorer or `--mode pairwise`, on a file with no non-empty
+verified material, when an arm loses all its bases, has only `padding` pairs or gets none sampled by
+`--bases`, when a `--sequential` run would pass with none of an arm's bases
+scored, or when no pair is usable. A label whose
+significant measures all rest on one judge says `(one judge)`; an escalated
+pairwise measure counts as its cheap judge's. Gate a judge on answers from the
+same answer model it will judge: `cross-judge` warns when the validity report was
+measured on another model's answers.
 
 ```bash
 python cli.py eval cross-judge --answers run.json --questions golden.json --grounded --both \

dokimi/docs/evaluating-treatments.md

@@ -72,8 +72,14 @@ judge, 24 bases, one run), the full material raised the mean intact score from
 0.674, not significant at this size), unchanged on wrong numbers. A skill arm's
 delta measured with the material is therefore on a higher level than one
 measured without it, and the difference between the two is not evidence that
-the skill helps. `eval judge-validity` passes no context, so it does not
-measure this setting.
+the skill helps. `eval judge-validity` passes no context by default; run it on an
+arms artifact with `--context stored` to gate the judge the way it scored the
+arms — each answer with its arm's stored material (bases with no stored entry are
+dropped and counted, except baseline with the no-context marker). It exits 2
+with a typed scorer or `--mode pairwise`, on a file with no non-empty verified
+material, when an arm loses all its bases, has only `padding` pairs or gets none sampled by `--bases`,
+when a `--sequential` run would pass with none of an arm's bases scored, or
+when no pair is usable.
 
 Runtime plugins are a separate run dimension, not arms. They modify every arm
 inside the same `(model, harness, plugin_set)` cell, and the JSON/Markdown

dokimi/dokimi/commands/evaluate.py

@@ -19,7 +19,7 @@
     JUDGE_MODEL_HELP,
 )
 from dokimi.eval.judge import JUDGE_PROVIDERS
-from dokimi.eval.judge_validity import DEFAULT_MIN_ACCURACY
+from dokimi.eval.judge_validity import DEFAULT_MIN_ACCURACY, NoUsablePairs
 from dokimi.utils import normalize_model_ref
 
 
@@ -298,14 +298,20 @@ def cmd_eval_judge_validity(args: argparse.Namespace) -> None:
     # the length-only one, not a judge replacement.
     # Built once and passed to both modes, so a new scorer kind is a change to
     # this block, not to each run path.
+    if args.scorer not in ("judge", "typed") and not args.scorer.startswith("typed:"):
+        print(f"Error: unknown --scorer {args.scorer!r}; use 'judge' or 'typed:<backend>'", file=sys.stderr)
+        sys.exit(2)
+    if args.context == "stored" and (args.scorer != "judge" or args.mode == "pairwise"):
+        # Refused before any model is loaded: neither scorer has a context slot.
+        what = "the typed scorer" if args.scorer != "judge" else "--mode pairwise"
+        print(f"Error: --context stored needs a judge that takes context; {what} does not",
+              file=sys.stderr)
+        sys.exit(2)
     typed = None
     if args.scorer == "judge":
         judge = Judge(model=args.judge_model, provider=args.judge_provider, temperature=args.judge_temperature)
         scorer = judge_scorer(judge)
         label = f"{args.judge_provider}/{args.judge_model}"
-    elif args.scorer != "typed" and not args.scorer.startswith("typed:"):
-        print(f"Error: unknown --scorer {args.scorer!r}; use 'judge' or 'typed:<backend>'", file=sys.stderr)
-        sys.exit(2)
     elif args.mode == "pairwise":
         # _score_pairs gives pair_scorer precedence, so the pairwise judge would grade
         # the pairs and the report would carry the typed arm's label.
@@ -346,6 +352,20 @@ def cmd_eval_judge_validity(args: argparse.Namespace) -> None:
         # The budget is part of what was measured: one checkpoint at 512 and at
         # 2048 tokens is two arms.
         label = typed.name + (f" max_len={typed.effective_max_len}" if typed.effective_max_len else "")
+    store = None
+    if args.context == "stored":
+        # The gate in the setting arms runs score in: each answer with its arm's
+        # treatment material, from the artifact's judge_materials.
+        from dokimi.eval.judge_context import JudgeContextStore
+
+        # from_artifact reads the file's judge_materials, which the Run records do not keep.
+        data = json.loads(Path(args.answers).read_text(encoding="utf-8"))
+        store = JudgeContextStore.from_artifact(data) if isinstance(data, dict) else None
+        if store is None or not any(store.materials().values()):
+            print("Error: --context stored needs an arms artifact with non-empty stored judge "
+                  "material; this file has none (judge_materials missing, or no record cites a "
+                  "digest that matches)", file=sys.stderr)
+            sys.exit(2)
     pair_scorer = None
     if args.mode == "pairwise":
         from dokimi.commands.judge_reliability import pairwise_judge
@@ -369,13 +389,18 @@ def progress(done: int, total: int) -> None:
         # overstates the cost per score by the pool size.
         concurrency=1 if typed is not None else args.concurrency,
         pair_scorer=pair_scorer,
+        store=store,
     )
-    if args.sequential:
-        from dokimi.eval.judge_sequential import run_sequential
+    try:
+        if args.sequential:
+            from dokimi.eval.judge_sequential import run_sequential
 
-        report = run_sequential(answers, scorer, max_bases=args.max_bases, **common)
-    else:
-        report = run_validity(answers, scorer, n_bases=args.bases, **common)
+            report = run_sequential(answers, scorer, max_bases=args.max_bases, **common)
+        else:
+            report = run_validity(answers, scorer, n_bases=args.bases, **common)
+    except NoUsablePairs as exc:  # none long enough, all no-ops, none with material
+        print(f"Error: {exc}", file=sys.stderr)
+        sys.exit(2)
 
     # Written before anything is printed: a console that cannot encode the summary must not lose the run.
     if args.output:
@@ -493,6 +518,13 @@ def register(sub, positive_int) -> None:
                                  "it PASS or FAIL (ignores --bases; see --max-bases)")
     validity_p.add_argument("--max-bases", type=positive_int, default=120, dest="max_bases",
                             help="Cap on base answers in --sequential mode (default: 120)")
+    validity_p.add_argument("--context", choices=["none", "stored"], default="none",
+                            help="What the judge sees with each answer: none (default), or 'stored' — "
+                                 "the arm's treatment material from the artifact's judge_materials, the "
+                                 "setting an arms run scores skill and docs arms in. Exits 2 with a typed "
+                                 "scorer or --mode pairwise, with no verified material, when an arm "
+                                 "loses all its bases, has only padding pairs or gets none sampled, when a --sequential run "
+                                 "would pass with none of an arm's bases scored, or when no pair is usable")
     validity_p.add_argument("--mode", choices=["absolute", "pairwise"], default="absolute",
                             help="absolute: score each member 0-100 (the shipped judge); pairwise: show "
                                  "the judge both members, in both orders, and ask which is better")

dokimi/dokimi/commands/judge_reliability.py

@@ -189,7 +189,7 @@ def cmd_eval_judge_wording(args: argparse.Namespace) -> None:
     if unknown:
         print(f"Error: unknown wording(s): {unknown}; known: {sorted(WORDINGS)}", file=sys.stderr)
         sys.exit(2)
-    pairs = _build(answers, {}, n_bases=args.bases, seed=args.seed, variants=variants, library=library)
+    pairs = _build(answers, {}, n_bases=args.bases, seed=args.seed, variants=variants, library=library).pairs
     label = f"{args.judge_provider}/{args.judge_model}"
     n_cond = 1 + args.redraws + len(wordings)
     print(f"Judge wording sensitivity: {label}, {len(pairs)} pairs x {n_cond} conditions "
@@ -366,7 +366,7 @@ def score(library, question, answer, ground_truth, context):
             print(f"⚠️  --validity for `{v.get('judge')}` was measured on {v['answers_model']} answers, "
                   f"these are {answers_model}'s — the gate may not transfer", file=sys.stderr)
     for missed in report.apply_validity(validity):
-        print(f"⚠️  --validity for `{missed}` matches no measure (compare the judge labels)", file=sys.stderr)
+        print(f"⚠️  --validity for `{missed}` matches no measure (compare the judge labels and contexts)", file=sys.stderr)
     if not report.measures:
         print("Error: nothing to compare — give --judges, --checklist, or an answers file with evaluations",
               file=sys.stderr)