feat(guardian): an OpenRouter provider, so #246 has a third vendor to compare

benchmarks/guardian/experiments/246-cross-vendor/README.md

@@ -0,0 +1,71 @@
+# Cross-vendor skeptic: still blocked, and now on measured limits (#246)
+
+Two attempts at the third arm, both stopped by the validity gate before any
+number was reported. The gate is the deliverable here; the comparison is not.
+
+## Why a gate at all
+
+`judge_finding` contains provider errors per finding and returns None, so a
+spent quota, an HTTP 429, a 402 and unparseable JSON all arrive as **the same
+row shape as a skeptic that refused to refute anything** — which is exactly what
+#246 predicts for a same-vendor skeptic. A broken arm would confirm the
+hypothesis. `MAX_UNRULED_RATE` refuses to print a comparison above 5% unruled,
+keeps the rows, and exits 2.
+
+It fired on both attempts. Neither produced a number that could be read.
+
+## Attempt 1 — free tier. The quota is 50 requests per day.
+
+`nvidia/nemotron-3-super-120b-a12b:free`, the whole corpus: **96 of 135 findings
+unruled**, and the pattern is a cutoff in time rather than a property of the
+input.
+
+| PR | evidence | unruled |
+|---|---|---:|
+| 122 (first) | no | **0 / 29** |
+| 140 | yes | 29 / 39 |
+| 141, 142, 143, 144 | — | **100%** |
+
+39 answered, then silence. With the day's earlier probes (4 candidate calls, 6
+in a smoke run) that is 49 successful requests before the wall — the documented
+free allowance is 50/day, and this is that number arrived at from the data.
+
+Rows: `free-nemotron-quota-cutoff.jsonl`.
+
+## Attempt 2 — paid. Two separate faults, one after the other.
+
+`qwen/qwen3.7-plus`, chosen because capability has to match: comparing
+`gemini-2.5-flash` against a small free model measures strong-versus-weak and
+calls it same-versus-cross.
+
+**First fault — thinking.** 118 of 135 unruled. `qwen3.7-plus` puts its chain of
+thought in a separate `reasoning` field and fills `content` only at the end, so
+on the larger prompts it spent the entire 8,000-token budget thinking and
+returned `content: null`. Raising the budget would have fixed the symptom and
+broken the experiment: the arm it is compared against is not doing extended
+thinking, so the two arms would differ in a dimension nobody chose. The provider
+now sends `reasoning: {"enabled": …}` explicitly in both directions, off by
+default, so a run can state what it did.
+
+**Second fault — credits.** The re-run failed on HTTP **402**: the account has
+`total_credits: $0` against `total_usage: $0.159`. The trial allowance is spent;
+paid models are unavailable until it is topped up.
+
+The second fault took half an hour to identify because the warning read only
+"Skeptic judgement failed; finding stays unruled" — 118 identical lines, no
+type, no message. It now carries both. A quota, a truncated reasoning model and
+a 402 have three different remedies and all three land in that one `except`.
+
+## To unblock
+
+1. **Top up OpenRouter.** The measured cost of the run with reasoning off is
+   ~$0.15: 135 findings × ~2,700 prompt tokens at \$0.32/M, plus short
+   completions at \$1.28/M. A dollar covers it several times over.
+2. **Free tier across three days.** The input is a frozen recording, so the run
+   is deterministic and can be split — 50 requests a day, 135 needed. Fragile,
+   and it needs resume support the tool does not have.
+3. A local ollama, which this issue already names as the no-cost option.
+
+Everything else is in place: the corpus with both finder orientations, the
+recordings, per-arm models, the scoring, and the gate that stopped two wrong
+answers from being published.

scripts/skeptic_arms.py

@@ -66,10 +66,17 @@
 REPO_ROOT = Path(__file__).resolve().parent.parent
 BENCH_DIR = REPO_ROOT / "benchmarks" / "guardian"
 
-#: The two arms. Named rather than derived, because "the opposite of the primary"
-#: is exactly the rule under test — deriving the arm from the finder would build
-#: the hypothesis into the instrument.
-ARMS = ("gemini", "mistral")
+#: The default pair. Named rather than derived, because "the opposite of the
+#: primary" is exactly the rule under test — deriving an arm from the finder
+#: would build the hypothesis into the instrument.
+#:
+#: `--arms` overrides it, and each arm may name its own model
+#: (`openrouter=nvidia/nemotron-3-super-120b-a12b:free`). Per arm rather than
+#: from the environment: `GUARDIAN_SKEPTIC_MODEL` applies to whichever provider
+#: is built, so a single value would hand one arm the other's model — one arm
+#: running something that does not exist, reported as a difference between
+#: vendors.
+DEFAULT_ARMS = "gemini,mistral"
 
 
 class NoArmError(RuntimeError):
@@ -80,17 +87,36 @@ class NoArmError(RuntimeError):
     """
 
 
-def arm_provider(name: str, env: Mapping[str, str]) -> tuple[BaseProvider, str]:
+def parse_arms(spec: str) -> dict[str, str | None]:
+    """`gemini,openrouter=vendor/model:free` → {arm: model or None}.
+
+    A model per arm, never one shared. The environment's
+    `GUARDIAN_SKEPTIC_MODEL` is dropped in `arm_provider` for the same reason.
+    """
+    arms: dict[str, str | None] = {}
+    for item in spec.split(","):
+        name, _, model = item.strip().partition("=")
+        if name:
+            arms[name] = model or None
+    return arms
+
+
+def arm_provider(
+    name: str, env: Mapping[str, str], model: str | None = None
+) -> tuple[BaseProvider, str]:
     """The skeptic for one arm, or a refusal naming the missing key.
 
-    `GUARDIAN_SKEPTIC_MODEL` is dropped rather than passed through.
-    `build_skeptic_provider` applies it to whichever provider it builds, so an
-    environment holding the production value (`gemini-2.5-flash`) would hand
-    that model name to the mistral arm — one arm running a model that does not
-    exist, while the other ran the intended one. Each arm takes its provider's
-    own default, and the models used are recorded on every row.
+    The environment's `GUARDIAN_SKEPTIC_MODEL` is dropped and `model` used
+    instead. `build_skeptic_provider` applies that variable to whichever
+    provider it builds, so an environment holding the production value
+    (`gemini-2.5-flash`) would hand that name to a mistral or openrouter arm —
+    one arm running a model that does not exist while the other ran the intended
+    one, and the difference reported as a comparison between vendors. The models
+    actually used are recorded on every row.
     """
     per_arm = {k: v for k, v in env.items() if k != "GUARDIAN_SKEPTIC_MODEL"}
+    if model:
+        per_arm["GUARDIAN_SKEPTIC_MODEL"] = model
     built = build_skeptic_provider({**per_arm, "GUARDIAN_SKEPTIC": name}, primary="none")
     if built is None:
         _msg = (
@@ -162,15 +188,17 @@ async def judge_one(
     }
 
 
-async def collect_rows(repo_root: Path, limit: int | None) -> list[dict[str, Any]]:
+async def collect_rows(
+    repo_root: Path, limit: int | None, arm_spec: dict[str, str | None]
+) -> list[dict[str, Any]]:
     """Both arms over every frozen pass; one row per (pass, arm).
 
     Returns the rows rather than writing them. The write is the caller's, and
     synchronous: a blocking file write inside the event loop is the same defect
     the worktree collection above avoids with `to_thread`, and here there is
     nothing to gain by being in the loop at all.
     """
-    arms = {name: arm_provider(name, os.environ) for name in ARMS}
+    arms = {name: arm_provider(name, os.environ, model) for name, model in arm_spec.items()}
     models = finder_models(BENCH_DIR / "results.jsonl")
 
     with tempfile.TemporaryDirectory(prefix="arms-") as tmp:
@@ -241,6 +269,43 @@ async def collect_rows(repo_root: Path, limit: int | None) -> list[dict[str, Any
 UNKNOWN_VENDOR = "unknown"
 
 
+#: The share of findings an arm may leave unruled before its numbers are refused.
+#:
+#: Not a tolerance — a validity gate, and it is the only thing standing between
+#: this experiment and a wrong conclusion. `judge_finding` contains provider
+#: errors per finding and returns None, so a spent quota, an upstream 429 and a
+#: model that cannot produce JSON all arrive as "unruled". An arm that answered
+#: nothing and an arm that refuted nothing are the same row shape, and "refutes
+#: nothing" is precisely what #246 predicts for a same-vendor skeptic. Reported
+#: without this check, a broken arm would confirm the hypothesis.
+MAX_UNRULED_RATE = 0.05
+
+
+class ArmTooQuietError(RuntimeError):
+    """Raised when an arm left too many findings unruled to be scored.
+
+    Refused rather than footnoted. A caveat under a table does not stop the
+    table being read, and this table's whole content is how much each skeptic
+    refuted.
+    """
+
+
+def validity_problems(rows: list[dict[str, Any]]) -> list[str]:
+    """Arms whose unruled share is too high to interpret, with the numbers."""
+    totals: dict[str, list[int]] = defaultdict(lambda: [0, 0])
+    for row in rows:
+        seen = totals[str(row["arm"])]
+        seen[0] += int(row.get("unruled") or 0)
+        seen[1] += int(row["findings"])
+    return [
+        f"arm {arm!r} left {unruled} of {findings} findings unruled "
+        f"({unruled / findings:.0%} > {MAX_UNRULED_RATE:.0%}); its numbers cannot be read as "
+        f"leniency because they may be silence."
+        for arm, (unruled, findings) in sorted(totals.items())
+        if findings and unruled / findings > MAX_UNRULED_RATE
+    ]
+
+
 def _vendor(model: str) -> str:
     """The vendor behind a model name, or `UNKNOWN_VENDOR`.
 
@@ -276,7 +341,7 @@ def report(rows: list[dict[str, Any]]) -> None:
     """The comparison the issue asks for, split by which vendor found the findings."""
     print(f"\n{len(rows)} (pass, arm) results\n")
     print(
-        f"{'finder':9} {'skeptic':9} {'kind':7} {'passes':>6} {'refuted':>8} "
+        f"{'finder':10} {'skeptic':11} {'kind':8} {'passes':>6} {'refuted':>8} "
         f"{'of':>5} {'killed GT':>10}"
     )
     groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
@@ -287,7 +352,7 @@ def report(rows: list[dict[str, Any]]) -> None:
         refuted = sum(int(r["refuted"]) for r in group)
         total = sum(int(r["findings"]) for r in group)
         killed = sum(len(r["killed_gt"]) for r in group)
-        print(f"{finder:9} {arm:9} {kind:7} {len(group):6} {refuted:8} {total:5} {killed:10}")
+        print(f"{finder:10} {arm:11} {kind:8} {len(group):6} {refuted:8} {total:5} {killed:10}")
     unattributed = sorted(
         {str(r["finder_model"]) for r in rows if _vendor(str(r["finder_model"])) == UNKNOWN_VENDOR}
     )
@@ -307,13 +372,26 @@ def main() -> int:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--out", type=Path, default=REPO_ROOT / ".guardian-arms.jsonl")
     parser.add_argument("--repo-root", type=Path, default=REPO_ROOT)
+    parser.add_argument(
+        "--arms",
+        default=DEFAULT_ARMS,
+        help="comma-separated arms, each optionally name=model (see DEFAULT_ARMS)",
+    )
     parser.add_argument(
         "--limit", type=int, default=None, help="judge only the first N passes (a smoke run)"
     )
     args = parser.parse_args()
-    rows = asyncio.run(collect_rows(args.repo_root, args.limit))
+    rows = asyncio.run(collect_rows(args.repo_root, args.limit, parse_arms(args.arms)))
     args.out.parent.mkdir(parents=True, exist_ok=True)
+    # Written before the gate: the rows are evidence either way, and a refused
+    # run whose data was discarded cannot be diagnosed.
     args.out.write_text("".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8")
+    problems = validity_problems(rows)
+    if problems:
+        for problem in problems:
+            print(problem, file=sys.stderr)
+        print(f"Rows kept at {args.out}; no comparison printed.", file=sys.stderr)
+        return 2
     report(rows)
     return 0
 

src/cgis/guardian/providers/base.py

@@ -86,7 +86,7 @@ def __init_subclass__(cls, **kwargs: object) -> None:
             return
         _msg = (
             f"{cls.__name__} must declare `name: ClassVar[str]` — one of the "
-            'GUARDIAN_PROVIDER values: "gemini", "mistral", "ollama". Add '
+            'GUARDIAN_PROVIDER values: "gemini", "mistral", "ollama", "openrouter". Add '
             f'`name: ClassVar[str] = "..."` as the first line of '
             f"{cls.__name__}'s class body."
         )