feat(arms): keep the judge's treatment material in the artifact

dokimi/changelog.d/20261016_190100_napetrov_judge_materials.md

@@ -0,0 +1,19 @@
+### Added
+
+- **Judge's treatment material stored in arms artifacts** — `dokimi arms run` now keeps
+  the exact text the judge reads (or would read) for each arm and question, with or
+  without `--judge`, indexed by SHA256 digest. Later readers of the artifact (e.g.,
+  `dokimi eval cross-judge --judges` re-scoring the same answers) see the stored material
+  instead of a 200-char snippet fallback, so the re-scored judge reads what the original
+  judge saw. Artifact sizes grow by one copy per
+  distinct text per arm (perf skill +0.3% for the text itself, docs runs grow by the
+  answerer's 15,000-char context cut per question).
+
+### Changed
+
+- **`eval cross-judge --judges` accepts stored scores when the file keeps the material** —
+  it used to exit 2 for any arm judged on treatment material. It now re-scores with the
+  stored text and exits 2 only when a stored score cites different material than the arm
+  record serves, or a question's material cannot be read; the error names the questions.
+  `--no-stored` still compares the measures anyway, and `--pairwise` and `--checklist`
+  still refuse stored scores because those judges see only answers.

dokimi/docs/benchmarking-and-comparison.md

@@ -401,11 +401,11 @@ measures, and labels each arm `CONSISTENT ±`, `LEANING ±`, `JUDGE-DEPENDENT` o
 only judges that passed. The stored scores are labelled from the file's
 `judge_provider`/`judge_model` (`provider/model (stored)`), so a `--validity`
 report for that judge reaches them and a verdict resting on it alone reads
-`(one judge)`. Re-scoring resolves each arm's context the way the arms run does;
-from a file that is the report snippets. Since `judge_context` the stored judge
-saw each treatment's full material, which the file does not keep, so with
-`--judges`, `--pairwise` or `--checklist` and stored scores for such an arm the
-command refuses (exit 2): the other judges see only snippets or answers.
+`(one judge)`. Re-scoring resolves each arm's context the way the arms run does,
+from the file's `judge_materials`, so a `--judges` judge sees what the stored
+judge saw. A file written before `judge_materials` has only the report snippets;
+with stored scores on such material, `--judges` refuses (exit 2), and
+`--pairwise` or `--checklist` always do, since those judges see only the answers.
 `--no-stored` compares those judges with each other. The gate itself (and `eval pairwise` / `eval checklist`) passes no
 context, so it validates a judge only in the no-context setting. A label whose significant measures all rest on one judge says `(one judge)`; an escalated pairwise measure counts as its cheap judge's. Gate a judge on
 answers from the same answer model it will judge: `cross-judge` warns when the

dokimi/docs/evaluating-treatments.md

@@ -36,16 +36,26 @@ loaded plus every catalog file list (see [Skill selection](#skill-selection-does
 `baseline`, `mcp_agent:` and `profile:` arms give it none.
 The output records this as `judge_context: treatment_material.v1` when every
 score saw the stored material, and `judge_context_sources` counts the scores by
-source. The material is kept in memory for the answering run only, so it reaches
-the judge only when answering and judging happen in the same `arms` run with
-unique question ids (a question with no id is given a positional one; duplicated
-ids are refused). Scores that fell back to the 200-character report snippets
-leave the run unstamped if all of them did — the protocol runs used before the
-stamp existed — and stamp it `treatment_material.v1+snippet_fallback` if only some
-did, which pools with nothing. A `baseline` arm's empty material always counts as
-stored, so with a baseline arm a fallback gives the mixed stamp. To compare with
-an older run, run answering and judging again; re-judging an answers file sees
-only the snippets.
+source. The artifact keeps the material: each arm record names its text by digest
+(`judge_context_sha256`) and `judge_materials` holds one copy of each distinct
+text — a skill arm's, identical for every question, is stored once (about 4,500
+characters for the perf skill, +0.3% for the text itself on an 81-question run;
+each arm record and stored score also carries a 108-byte digest line, about 2% more
+for a baseline plus skill run). The material is kept whether or not `arms run` was
+given `--judge`. For docs arms, the material is the retrieved chunks, rarely
+identical across questions, so each question adds its own copy (up to the
+15,000-char cut); on a synthetic 20-question docs run the artifact grew from
+93,209 to 392,852 bytes (4.2x). Sharing an arms artifact shares the skill or doc
+text it contains. With `cross-judge --judges`, the judge is given the same text through `JudgeContextStore.from_artifact`, which
+skips any text that does not match its digest. Question ids must be unique for this
+(a question with no id is given a positional one; duplicated ids are refused).
+Scores that fell back to the 200-character report snippets leave the run unstamped
+if all of them did — the protocol runs used before the stamp existed — and stamp it
+`treatment_material.v1+snippet_fallback` if only some did, which pools with nothing.
+A `baseline` arm's empty material always counts as stored, so with a baseline arm a
+fallback gives the mixed stamp. To compare with an older run, written before
+`judge_materials`, run answering and judging again: re-judging that file sees only
+the snippets.
 
 What this changes is the evidence the judge has, not how well it separates good
 answers from bad ones. On 81 skill-arm answers to the perf golden set (Haiku 4.5

dokimi/dokimi/commands/judge_reliability.py

@@ -9,6 +9,7 @@
 import json
 import sys
 from pathlib import Path
+from typing import Optional
 
 from dokimi.defaults import DEFAULT_CONCURRENCY, DEFAULT_JUDGE_MODEL, DEFAULT_JUDGE_PROVIDER
 from dokimi.eval.judge import JUDGE_PROVIDERS
@@ -219,17 +220,78 @@ def score(library, question, answer, ground_truth, context):
     baseline = args.baseline_arm or (data.get("baseline_arm") if isinstance(data, dict) else None) or "baseline"
     library = args.product or (data.get("library_name") if isinstance(data, dict) else None) or "the library"
     report = CrossJudgeReport(baseline_arm=baseline, require_validated=args.require_validated)
+    from dokimi.eval.judge_context import SOURCE_STORED, JudgeContextStore
+    from dokimi.utils import question_id
+
+    # What the arms run's judge saw, when the file keeps it (judge_materials). Built
+    # once and only for --judges: it is the store the gate checks and the judge reads.
+    materials: Optional[JudgeContextStore] = (
+        JudgeContextStore.from_artifact(data) if args.judges and isinstance(data, dict) else None
+    )
+
     if not args.no_stored and isinstance(data, dict) and (m := stored_measure(data)):
-        # Every other measure (re-scored, pairwise, checklist) sees at most the report snippets.
-        unrepro = unreproducible_arms(data) if (args.judges or args.pairwise or args.checklist) else []
+        # Re-scored judges get the file's judge_materials; pairwise and checklist
+        # judges see the answers alone. Each path sees different input, so we refuse
+        # the comparison to prevent mixing a change of judge with a change of input.
+        unrepro: set[str] = set()
+        # Per arm, the questions where the re-scored judge would read a different
+        # text than the stored judge (differs), or none at all (unserved).
+        differs: dict[str, list] = {}
+        unserved: dict[str, list] = {}
+
+        # Pairwise and checklist judges see only answers, so stored material makes them incompatible
+        if args.pairwise or args.checklist:
+            unrepro.update(unreproducible_arms(data))
+        elif materials is not None:
+            # The loop below is the whole --judges gate: it compares each stored score's
+            # digest with what the reader serves, for the questions score_measure scores.
+            # score_measure only scores questions that have an answer record.
+            answered = {question_id(r) for r in data.get("answers") or []
+                        if isinstance(r, dict) and isinstance(question_id(r), (str, int))}
+            # The arm exempt from the check is the one whose recorded kind is baseline, as in
+            # unreproducible_arms: --baseline-arm only picks the comparison's reference arm.
+            kinds = {arm: (body.get("metadata") or {}).get("arm_kind")
+                     for r in data.get("answers") or []
+                     if isinstance(r, dict) and isinstance(r.get("arms"), dict)
+                     for arm, body in r["arms"].items() if isinstance(body, dict)}
+            for ev in data.get("evaluations") or []:
+                if not isinstance(ev, dict):
+                    continue
+                qid = ev.get("question_id")
+                for arm, s in (ev.get("scores") or {}).items():
+                    kind = kinds.get(arm) or ("baseline" if arm == baseline else None)
+                    if not (isinstance(s, dict) and s.get("judge_context_source") == SOURCE_STORED
+                            and kind != "baseline"):
+                        continue
+                    if not isinstance(qid, (str, int)):
+                        unserved.setdefault(arm, []).append(qid)  # no usable id: reader uses snippets
+                    elif qid not in answered:
+                        continue
+                    elif (served := materials.digest_of(qid, arm)) is None:
+                        unserved.setdefault(arm, []).append(qid)
+                    elif s.get("judge_context_sha256") != served:
+                        differs.setdefault(arm, []).append(qid)
+
         if unrepro:
-            print(f"Error: the stored judge saw treatment material for {', '.join(unrepro)} that the "
-                  "answers file does not keep; the other measures (a re-scored judge sees only the report "
-                  "snippets, a pairwise or checklist judge only the answers), so the comparison would mix "
-                  "a change of judge with a change of input. Pass --no-stored to compare them with each "
-                  "other.", file=sys.stderr)
+            print(f"Error: the stored judge saw treatment material for {', '.join(sorted(unrepro))}, but a "
+                  "pairwise or checklist judge sees only the answers, not the arm's treatment material, "
+                  "so the comparison would mix a change of judge with a change of input. Pass "
+                  "--no-stored to compare them with each other.", file=sys.stderr)
+        for arm, qids in sorted(differs.items()):
+            print(f"Error: for {arm}, the stored score cites different material than the arm record "
+                  f"serves a re-scored judge, for question(s) {', '.join(map(repr, qids))}: the "
+                  "re-scored judge would read a different text. Pass --no-stored to compare "
+                  "them with each other.", file=sys.stderr)
+        for arm, qids in sorted(unserved.items()):
+            print(f"Error: for {arm}, no usable material can be read for question(s) "
+                  f"{', '.join(map(repr, qids))} (the record cites no digest, the text fails its "
+                  "digest, or the question has no usable id), so a re-scored judge would read "
+                  "snippets where the stored judge read the material. Pass --no-stored to "
+                  "compare them with each other.", file=sys.stderr)
+        if unrepro or differs or unserved:
             sys.exit(2)
         report.measures.append(m)
+
     specs = [j.strip() for j in (args.judges or "").split(",") if j.strip()]
     _check_judge_temperature([(m, p) for p, _, m in (s.partition("/") for s in specs) if m and p in JUDGE_PROVIDERS],
                              args.judge_temperature)
@@ -245,6 +307,7 @@ def score(library, question, answer, ground_truth, context):
             report.measures.append(score_measure(
                 records, scorer_of(judge), name=f"{provider}/{model}", library=library,
                 concurrency=args.concurrency, progress=_progress(), ground_truth=grounding,
+                store=materials,
             ))
     for path in args.pairwise or []:
         pw = json.loads(Path(path).read_text(encoding="utf-8"))

dokimi/dokimi/eval/arm_runner.py

@@ -27,9 +27,11 @@
 )
 from dokimi.eval.cells import Cell
 from dokimi.eval.judge_context import PROTOCOL as JUDGE_CONTEXT_PROTOCOL
+from dokimi.eval.judge_context import SOURCE_STORED
 from dokimi.eval.judge_context import stamp_for as judge_context_stamp
 from dokimi.eval.judge_context import JudgeContextStore
 from dokimi.eval.judge_context import build as build_judge_context
+from dokimi.eval.judge_context import cite as cite_judge_materials
 from dokimi.eval.judge_context import join_injected
 from dokimi.metrics.usage import to_record
 from dokimi.treatments.base import Treatment
@@ -461,6 +463,11 @@ def _score(rec: Dict[str, Any]) -> Dict[str, Any]:
                     score = {"error": str(exc)}
                 if isinstance(score, dict):
                     score = {**score, "judge_context_source": source}
+                    digest = (self.judge_context.digest_of(rec["question_id"], arm_name)
+                              if source == SOURCE_STORED else None)
+                    if digest:
+                        # Which of `judge_materials` this score saw.
+                        score["judge_context_sha256"] = digest
                 scores[arm_name] = score
 
             deltas = {}
@@ -522,6 +529,11 @@ def build_output(
             "agentic_usage_summary": self._summarize_agentic_usage(records),
             "skill_selection_summary": summarize_skill_selection(records),
         }
+        # One copy of each judge context the arm records cite, written whether or
+        # not the run was judged, so an answers-only artifact can be judged later.
+        materials = cite_judge_materials(records, self.judge_context)
+        if materials:
+            out["judge_materials"] = materials
         if self.matrix_cell:
             out["matrix_cell"] = self.matrix_cell
         if evaluations is not None:

dokimi/dokimi/eval/cross_judge.py

@@ -147,25 +147,44 @@ def stored_label(data: Dict[str, Any]) -> str:
 
 
 def unreproducible_arms(data: Dict[str, Any]) -> List[str]:
-    """Arms whose stored scores saw treatment material the answers file does not hold.
-
-    Since ``judge_context`` the arms run shows its judge the arm's full material
-    (:mod:`dokimi.eval.judge_context`) and keeps it in memory only, so
-    a re-scored judge reading the file sees the 200-char report snippets. In
-    review that was 28 characters of context against 11,079, and the same judge
-    on the same answers read +24.2 (p 0.0006) one way and +1.4 (p 0.65) the
-    other. A baseline arm has no material, so both sides see the same thing.
+    """Arms whose stored scores saw treatment material the other measures will not.
+
+    This is the gate for pairwise and checklist judges, which are shown the answers
+    alone, so every arm the stored judge saw material for is unreproducible. A
+    re-scored judge (``cross-judge --judges``) is gated per question instead, on
+    what :meth:`JudgeContextStore.from_artifact` serves it.
+
+    The arms run stores its judge's view of treatment material in ``judge_materials``
+    (:mod:`dokimi.eval.judge_context`). ``cross-judge --judges`` gives its judge
+    that same material by using the artifact's ``judge_materials``. Pairwise and
+    checklist judges see the answers alone. In review, the same judge on the same
+    answers read +24.2 (p 0.0006) with the full material (11,079 chars) and +1.4
+    (p 0.65) with the 28-char snippet. A baseline arm has no material, so all judges
+    see the same thing.
     """
-    kinds = {
-        arm: (body.get("metadata") or {}).get("arm_kind")
-        for r in data.get("answers") or [] for arm, body in (r.get("arms") or {}).items()
-        if isinstance(body, dict)
-    }
-    return sorted({
-        arm for ev in data.get("evaluations") or [] for arm, s in (ev.get("scores") or {}).items()
-        if isinstance(s, dict) and s.get("judge_context_source") == SOURCE_STORED
-        and kinds.get(arm) != "baseline"
-    })
+    kinds = {}
+    for r in data.get("answers") or []:
+        if not isinstance(r, dict):
+            continue
+        arms = r.get("arms")
+        if not isinstance(arms, dict):
+            continue
+        for arm, body in arms.items():
+            if isinstance(body, dict):
+                kinds[arm] = (body.get("metadata") or {}).get("arm_kind")
+    unreproducible = set()
+    for ev in data.get("evaluations") or []:
+        if not isinstance(ev, dict):
+            continue
+        scores = ev.get("scores")
+        if not isinstance(scores, dict):
+            continue
+        for arm, s in scores.items():
+            if not (isinstance(s, dict) and s.get("judge_context_source") == SOURCE_STORED
+                    and kinds.get(arm) != "baseline"):
+                continue
+            unreproducible.add(arm)
+    return sorted(unreproducible)
 
 
 def stored_measure(data: Dict[str, Any], name: Optional[str] = None) -> Optional[Measure]:
@@ -201,6 +220,7 @@ def score_measure(
     arms: Optional[Sequence[str]] = None, concurrency: int = 1,
     progress: Optional[Callable[[int, int], None]] = None,
     ground_truth: Optional[Dict[str, str]] = None,
+    store: Optional[JudgeContextStore] = None,
 ) -> Measure:
     """Score every answer; with ``ground_truth`` (question_id -> text), grounded.
 
@@ -210,14 +230,13 @@ def score_measure(
     and +0.3..+2.3 grounded (none against it).
 
     Each answer's context comes from the arms run's own resolver
-    (:meth:`~dokimi.eval.judge_context.JudgeContextStore.resolve_with_source`).
-    Read from a file it has no stored treatment material, so it falls back to
-    the arm's report snippets, which is what a run before ``judge_context``
-    showed its judge; without them a stored-vs-re-scored gap mixes a change of
-    judge with a change of what the judge was shown. Material the stored judge
-    saw that the answers file does not persist is not reproduced.
+    (:meth:`~dokimi.eval.judge_context.JudgeContextStore.resolve_with_source`)
+    over *store* — pass ``JudgeContextStore.from_artifact(data)`` and the judge
+    sees what the arms run's judge saw. Answers whose material the file does not
+    hold (or with no store) fall back to the arm's report snippets, which is
+    what a run before ``judge_context`` showed its judge.
     """
-    store = JudgeContextStore()
+    store = store if store is not None else JudgeContextStore()
     jobs = [
         (r["question_id"], arm, r.get("question_text", ""), body["answer"],
          store.resolve_with_source(r["question_id"], arm, body)[0])