guardian: does impact_score earn its cost? (collinear with verdict, +69% tokens)

[zaebee]

Decision point deferred from #246, to be taken once there is more evidence than the 3-PR run in #269.

What was measured

The two-axis contract (verdict = is it true, impact_score 0-10 = does it matter) was designed so the axes are orthogonal. Measured across pr-143/144/140 with a gemini-2.5-flash skeptic over frozen finder sets:

• Pooled separation median(score|GT) − median(score|non-GT) = 3.0 against a 3.0 gate. It passes, with zero margin.
• Per-PR it swings 1.5 / 7.0 / 2.0 — the axis is not stable across PRs.
• On pr-143 the separation among survivors is 0.0: the low scores sit exactly on findings gemini had already refuted, and every surviving noise finding scores the same as the ground-truth ones (medians 7.0 vs 7.0).

In other words, for both models tested the score is largely a restatement of the verdict rather than an independent signal.

Cost side: per-finding judgement ran 221k skeptic prompt tokens across the three PRs versus ~131k for the batch arm — +69%. (Note that cost belongs to per-finding isolation, which is justified independently by recall protection; the question here is only whether the score adds anything on top.)

Options

1. Keep as observe-only — record the score, never threshold on it, revisit when a third skeptic family is benchmarked. GUARDIAN_IMPACT_THRESHOLD already defaults to 0, so this is the status quo.
2. Try to make it orthogonal — the rubric currently invites the model to score low for the same reasons it refutes. A prompt that asks for importance assuming the finding is true might decouple them. Cheap to test with replay (no finder cost).
3. Remove it — drop the field, the threshold knob and the separation gate; keep per-finding isolation for its recall protection alone.

Option 2 is the cheapest informative next step and should be tried before 3.

Do not decide from one more run

Per-PR variance (1.5 / 7.0 / 2.0 on n=4-6 ground-truth findings each) means a single additional PR cannot settle this. Needs the pooled figure over more PRs, or a deliberate prompt A/B via --replay-finder.

Refs #246, #269.

[zaebee]

Option 2 tested and refuted: the rubric is not what couples the axes

Ran the decoupled-rubric A/B on the pr-143 frozen set (same model, gemini-2.5-flash, same findings, ~98k tokens). Rubric B asks for importance assuming the claim is true — "even if you refuted it, score it as if it were real" — removing every invitation to express claim-weakness through the score.

	rubric A (shipped)	rubric B (decoupled)
score histogram	{0:5, 1:2, 4:2, 5:2, 6:3, 7:14}	{0:1, 1:1, 4:6, 5:2, 7:17, 8:1}
mode (7) covers	50% of findings	60%
GT scores	[7,7,7,7,7,6]	[7,7,7,7,7,7] — zero variance
separation all / survivors	1.5 / 0.0	0.0 / 0.0
recall / noise	1.00 / 15	1.00 / 13

The rewrite made the score MORE modal, not less. Every ground-truth finding scored exactly 7, and most noise scored 7 too. The coupling I hypothesised — a rubric that lets the model say "weak claim" through a low score — is not the mechanism. This model simply does not rank on the importance axis; it emits a mode with a small tail.

Worth noting the low scores under rubric A sat on findings it had already refuted, which is what produced the illusion of signal (separation 1.5). Removing that overlap removed the signal entirely.

Side effect worth keeping

Rubric B moved the verdict axis: 9 refutations instead of 7, recall still 1.00, noise 15 → 13. Telling the model to judge importance independently seems to have made it slightly more willing to refute correctly. n=1, but the direction is interesting and it is free to re-test via replay.

Where this leaves the options

• Option 2 (make it orthogonal by prompt) — tested, refuted on gemini-2.5-flash.
• Option 1 (observe-only) is now the default: GUARDIAN_IMPACT_THRESHOLD stays 0, the field keeps being recorded, and the question reopens only if a different skeptic family shows a non-modal distribution.
• Option 3 (remove) becomes reasonable if no model produces a spread — but removal costs a contract change, so it should wait for at least one more family (local ollama is free to test) before deciding.

The measurement to watch is the mode share, not the separation: a model that puts 50-60% of findings on one score cannot rank regardless of what the medians say.

🤖 Generated with Claude Code

[zaebee]

Rubric B on pr-144 and pr-140: verdict effect is real but marginal; the impact axis is finished

PR	rubric	recall	noise	mode share	verdicts
143	A	1.00	15	7 = 50%	7 ref / 20 conf
143	B	1.00	13	7 = 60%	9 ref / 18 conf
144	A	0.60	5	0 = 45%	9 ref / 10 conf
144	B	0.60	5	7 = 35%	8 ref / 11 conf
140	A	0.20	11	0 = 36%	8 ref / 12 conf
140	B	0.27	11	7 = 40%	7 ref / 13 conf

Means: recall 0.600 → 0.622, noise 10.33 → 9.67. Rubric B is never worse and better on two PRs of three. But each win is a single finding, not a trend — shipping a production prompt on a one-finding delta at n=1 per PR is exactly how this system has swung before.

The impact axis is now definitively dead under B

Separation: 0.0 / 0 / 0.0, and among survivors on pr-140 it is −1.0 — noise scored above ground truth.

The mode-share column explains why rubric A ever looked informative: its mode on pr-144 and pr-140 was 0 (36–45% of findings), and those zeros sat on refuted findings. A was mixing "unimportant" with "untrue". Rubric B removes the mixing — and what remains is a constant at 7.

Recommendation: do not ship B yet

Rubric B trades a marginally better verdict axis for definitively killing the importance axis. That trade is fine if impact_score stays observe-only forever — but it forecloses the one open question first.

Run a local ollama skeptic over the existing frozen sets before deciding. It is free, needs no finder calls, and answers the actual question of this issue: does any model produce a non-modal distribution on this axis? If ollama also lands at 50%+ on one value, then the axis cannot rank for structural reasons, option 3 (remove) is justified, and rubric B comes along as a free bonus.

Frozen sets for pr-143/144/140 are reproducible via --record-finder; the recordings used here were produced with mistral-medium.

🤖 Generated with Claude Code

[zaebee]

The collinearity is quantifiable from data already in the repo — no model calls needed

benchmarks/guardian/results.jsonl stores per-finding impact_score, verdict and matched_gt, so the question this issue asks can be re-measured offline. Four runs carry scores (98 scored findings, mistral-medium, pr-143/144/140). The c4c391b rows reproduce this issue's rubric-A mode-share column exactly — 143: 7 = 50%, 144: 0 = 45%, 140: 0 = 36% — so the dataset is the same one the earlier analysis used.

Among survivors, the axis carries almost nothing

Restricting to findings that actually reach a human (verdict != "refuted"):

run	GT median (n)	noise median (n)	separation
pr-143 a75aef7	7.0 (6)	6.0 (22)	+1.0
pr-143 c4c391b	7.0 (6)	7.0 (15)	0.0
pr-144 c4c391b	7.0 (3)	7.0 (8)	0.0
pr-140 c4c391b	4.0 (3)	7.0 (11)	−3.0
pooled	7.0 (18)	6.5 (56)	+0.5

The distributions sit on top of each other, both peaking at 7:

GT   : {3:1, 4:2, 6:4, 7:8, 8:3}
noise: {2:4, 3:3, 4:4, 5:7, 6:10, 7:25, 8:3}

This extends the earlier observation — separation 0.0 among survivors on pr-143 — to all three PRs and to a pooled figure: +0.5 on a 0–10 scale, with one PR negative.

Why the gate's 3.0 passes anyway

The gated separation is computed over all findings, and refuted findings score a median of 0 (visible per-run: 0 / 0 / 0.0 for the refuted subsets). Refuted findings are mostly noise, so including them makes the "separation" a restatement of the verdict — the noise median is dragged down by the very findings the verdict already removed.

Split the population and the effect disappears: on the subset where a ranking signal would actually be useful, there is none.

That is the issue's title — collinear with verdict — now quantified across the whole committed dataset rather than inferred from one PR.

What this does not settle

One model family. All 98 scored findings come from mistral-medium-latest. The structural question this issue poses — does any model produce a non-modal distribution on this axis — needs a second family, which is exactly what the local-ollama step was for. This measurement strengthens the case for option 3 (remove) but does not substitute for that evidence.

Two notes for whoever picks this up:

• Ollama is not trivially available here (needs frpc forwarding), so "free" means "no API spend", not "no setup".
• cohere is declared in the guardian dependency group but has no provider module — a third family exists on paper only, and writing one plus obtaining a key is more work than standing up the tunnel.

Since #279, production runs record the finder pass (guardian-finder-<pr> artifact) and those recordings retain impact_score, so per-finding score data now accumulates on its own from real reviews rather than only from bench runs.

Reproduce with:

python3 -c "
import json, statistics, collections
rows = [json.loads(l) for l in open('benchmarks/guardian/results.jsonl') if l.strip()]
rows = [r for r in rows if any(f.get('impact_score') is not None for f in r.get('findings', []))]
gt, noise = [], []
for r in rows:
    surv = [f for f in r['findings'] if f.get('impact_score') is not None and f.get('verdict') != 'refuted']
    gt += [f['impact_score'] for f in surv if f.get('matched_gt')]
    noise += [f['impact_score'] for f in surv if not f.get('matched_gt')]
print(statistics.median(gt) - statistics.median(noise), len(gt), len(noise))
"

🤖 Generated with Claude Code

[zaebee]

A downstream data point for option 1 ("keep as observe-only"), from the consuming side.

impact_score does not stay observe-only once it leaves this repository. hivemark averages it per reviewer identity and prints mean impact 6.31 on a public page, with no scale and no attribution: https://zaebee.github.io/hivemark/

Two consequences worth weighing here, because they are not visible from inside Guardian.

The number is published without its provenance. Nothing in the emitted record says who assigned impact_score. Reading this issue's two-axis contract — verdict = is it true, impact_score = does it matter, both from the skeptic — settles it, but only for someone who reads this issue. Downstream it arrives inside the finding object alongside confidence, and a consumer cannot tell them apart.

For a same-model run, the published impact is self-assigned. One of the three identities on that page is mistral · graph, where the skeptic is the same model as the finder — the fallback case #246 is about. Its mean impact 6.31 is therefore a model scoring the importance of its own findings, and it currently sits in the same position and format as the other two identities' numbers.

hivemark already renames the confirmation metric to self-graded rate for that identity rather than merely footnoting it. It does not do the same for impact, because the field's provenance is not in the record. Tracked downstream as zaebee/hivemark#49.

This is not an argument for option 3. The measurement in this issue — separation 0.0 among survivors on pr-143, per-PR swing 1.5/7.0/2.0 — is a much stronger reason to question the field than anything downstream, and option 2 (decouple the rubric by asking for importance assuming the finding is true) is clearly the right next step. The only ask from here is small:

Whichever option wins, put the assigner in the record. A field named impact_score_by or moving the two skeptic-assigned fields under a skeptic object would let a consumer treat a self-graded impact the way it already treats a self-graded verdict. If option 3 wins and the field goes away, this costs nothing.