guardian: per-criterion finder passes + relevance triage — a structural fix for attention theft (#247)

[zaebee]

Idea sourced from ecom.tech's production AI reviewer (GitLab CI, 700+ projects, 400–500 reviews/day). Their Triage Matrix is the one mechanism in that write-up we don't have and that maps onto two open problems here.

The mechanism they use

They never run one "find everything" prompt over the whole diff. Review is decomposed by criterion, and each criterion goes through two cheap gates before the expensive pass:

1. Relevance: is this criterion applicable to this MR at all? → third verdict NOT_APPLICABLE. Reported to cut processing ~40%.
2. Triage Matrix: which of the changed files does this criterion apply to? Only those pairs get reviewed.

They name this as their fix for false positives / comments landing in files unrelated to the diff — i.e. an architectural fix for noise, not a prompt one.

Why it fits us

Our finder names ~7 focus areas in a single prompt and gets one pass over the whole diff. Two measured consequences:

• Attention theft is real and burned us (guardian: recall blind classes — pr-142 cosine masking + float-equality-in-tests #247). A targeted few-shot for float == did not crack that class and dropped pr-144 median recall 3/5 → 1/5 by stealing attention from the yaml-mapping class it was reliably catching. Reverted on no-evidence-no-ship. Per-criterion passes remove the mechanism: each class gets its own call, so there is nothing to steal attention from.
• pr-142 cosine/component-masking is structurally blind (0/2 across runs) — a class with no focus area. Under decomposition, adding a class is adding a pass, not re-weighting a shared prompt.

It also attacks noise from the side the skeptic can't (#246): a same-model skeptic is binary (refute-all or refute-nothing), and cross-family gemini cost −33% noise for −10% recall. Narrowing scope before the finder is the untried lever.

Proposed experiment

Arms, all on the same fixtures, control run in the same session (a recorded baseline row is not a control when a sampled model produced it):

• A (control): current single-pass finder, unchanged.
• B: per-criterion decomposition — split the finder focus areas into K criteria, one call per criterion over the same diff, union the findings, same skeptic downstream.
• C: B + relevance gate — a cheap pre-pass per criterion returning APPLICABLE / NOT_APPLICABLE against the changed-file list, skipping the criterion's expensive pass when not applicable. C only runs if B shows signal; its job is cost, not recall.

Fixtures: pr-140, pr-143, pr-144 (rich GT), plus pr-142 — it is the whole point of the blind-class half and must be in the set.

Measure per arm: recall per PR, noise, findings count, prompt+completion tokens, wall time.

Pre-registered gates (decided before running)

Ship B only if all of:

• mean recall across pr-140/143/144 ≥ control, and strictly up on ≥ 2 of the 3 — a one-finding delta on a single PR is not evidence, this system has swung on that before;
• no regression on the classes control already catches — specifically the yaml-mapping class on pr-144 must not fall (that is the exact failure mode guardian: recall blind classes — pr-142 cosine masking + float-equality-in-tests #247 recorded);
• noise per surfaced finding no worse than control after the skeptic.

Additionally, count as a blind-class win (the #247 half) only if at least one of float-equality-in-tests (pr-144) or the pr-142 cosine class is caught in ≥ 3 of 5 runs — single-window hits are what the reverted few-shot already produced.

Kill B if: token spend > 2.5× control with no recall gain, or recall improves solely by emitting more findings with proportionally more noise (check findings count alongside recall).

Notes / non-goals

• Do not import their agentic loop (up to 100 iterations, tool calling, LSP + pgvector/HyDE retrieval). Their own anti-hallucination guards — abort after 5 identical errors, nudge after 5 identical actions, "wrap up" reminder at 85% of iterations — are the cost of that design, not a feature. This issue is decomposition only, no new retrieval layer.
• Do not import their criterion text: the article reports no precision/recall figures at all, so their taxonomy has no measured standing here. This is about the control flow.
• Their negative result is worth honouring: per-project criterion customisation "turned out to be redundant", only a scope parameter (diff | repository) survived. Argues against adding knobs on top of our universal-core + project-overlay design.

Related: #247 (blind classes, over-anchoring lesson), #258 (focus-area taxonomy — decomposition changes how a new class is added), #246 (cross-model skeptic), #155 (run-ensemble union — B's union step shares the dedup machinery).