RFC: give the skeptic executable verbs — 84% of one review's noise was settleable by running a command

[zaebee]

The observation

A guardian review of #399 produced 19 findings, of which 1 had substance. The other 18 were not near-misses — six were the same false claim repeated, and two recommended changes that would have removed deliberate guards.

The single most common failure was a claim about a rule that the repository does not have and a checker that does not look at the file:

"The use of Any is prohibited in strict mypy mode" — ×6

$ uv run mypy --strict /tmp/anytest.py     # def f(x: dict[str, Any]) -> Any
Success: no issues found in 1 source file

pyproject.toml:230  exclude = [..., "scripts"]      # the file under review
Makefile:14         uv run mypy src                 # scripts/ is never checked

One command settles it. The finder did not run it, and the skeptic — gemini-2.5-flash, same vendor as the finder — confirmed all six.

The measurement

Classifying all 19 by "would executing something have settled this?":

tier	what it needs	findings	share
static (mypy, ruff, grep) — does not execute PR code	9	47%
running code (pytest)	7	37%
cumulative settleable by execution		16	84%
beyond any tool	3	16%

Of the 16, execution kills 15 and confirms 1 — the one real finding (git diff with check=True raised CalledProcessError where the module promised MissingFixtureError; fixed in #399).

Where execution does not reach, and why that matters

Three findings survive any tool, and two of them share a shape worth naming: they proposed relaxing a >= 72 ratchet floor and an ambiguous_hits invariant "to reflect the actual number". The code is correct and the tests pass — the model was not wrong about behaviour, it was wrong about intent. It saw a rigid assertion and offered to soften it, not knowing that rigidity was the entire point.

No sandbox fixes that. It is an argument for keeping the ratchets' reasoning in the docstring where a reviewer reads it, not for a better tool.

The runtime question is already answered

The guardian review job checks out refs/pull/N/head at fetch-depth: 0 and runs uv sync --frozen --extra guardian. The repository and the toolchain are already there — the guardian process is running inside them. The model needs no sandbox of its own; the guardian executes an allowlisted verb on its behalf and feeds the output back.

The security crux is not "can it run", it is what running costs:

verb	executes PR code?	risk
mypy --strict <path>	no	low — static
ruff check <path>	no	low — static
grep/read within the repo	no	low
pytest <node-id>	yes	the job holds pull-requests: write and issues: write

The job is gated on a commenter with write access, so the threat model is "a maintainer runs the guardian on a hostile PR" — which is precisely the review scenario, not an edge case.

Proposal, tiered so the cheap half ships first

v1 — static verbs only. A fixed allowlist, not a shell: typecheck(path), lint(path), grep(pattern, path). Arguments constrained to repo-relative paths; output truncated before it re-enters the prompt. Buys 47% of today's noise at essentially no new risk, because none of these executes the code under review.

v2 — pytest behind a boundary. Reaches 84%, and needs its own answer to the token question: a separate job without pull-requests: write, or network-off, or both. Not in v1.

Where it belongs: the skeptic, not the finder. The skeptic already runs once per finding and its job is refutation. "Show the command whose output supports this claim, or withdraw it" is a natural strengthening of a prompt that today asks only for an opinion. A finder with tools would be slower and would still emit whatever it emits.

This is measurable now, and cheaply

#399 rebuilt 72 frozen finder passes from the corpus at zero model cost. A tool-using skeptic is therefore a replay arm: the same finder output, judged by two skeptics, no finder tokens. The comparison is recall and noise against the current skeptic — the same shape #246 registers.

The one existing matched data point already says the skeptic is the right place to intervene: on pr-143 with the finder frozen at 28 findings, a same-vendor skeptic refuted 0 and a cross-vendor one refuted 7 while recall stayed at 1.00.

Cost not yet counted

No provider here supports tool calling. BaseProvider exposes generate_content and generate_structured; there are zero occurrences of tools=, function_declarations or tool_choice across gemini.py, mistral.py and base.py. A tool loop is new surface on every provider, and it lands in providers/, which is inside the review-fingerprint closure — so it re-mints every reviewer identity (by design, #375).

Open questions

1. Does a tool-using skeptic actually refute more, or does it confirm with a citation? Today's run confirmed six false claims without checking any of them; the failure may be disposition rather than capability.
2. Call budget. The skeptic is per-finding; a tool loop multiplies that. On a 28-finding review at 2 calls each, that is 56 round trips.
3. Is grep enough for the 9 static findings, or is mypy doing the work? Six of the nine are the mypy claim alone.

Related: #246 (cross-model skeptic — same intervention point), #155 (ensemble dedup — six of today's 19 were one finding repeated, so dedup alone cuts ~26% with no tools at all), #248 (finder robustness).

[zaebee]

Measured. v1 is real, and it costs something the RFC did not price.

Method and data in #406; this is what it says about the questions asked here.

One review — the very one this RFC was written from, guardian-finder-399, 24 findings of which exactly one substantive. Three arms, paired per finding, finder never called: the recorded verdicts, a re-judging without evidence (control), and a re-judging with evidence collected in a worktree at the reviewed commit.

Baseline 19 confirmed / 5 refuted.

arm	conf→ref	ref→conf	moved	cites a checker
control	2	2	4	6/24
treatment draw 1	8	0	8	7/24
treatment draw 2	7	1	9	8/24

Open question 1 — disposition or capability?

Capability. All six findings claiming a mypy rule this repository does not have were refuted, and all six cite the checker output. Finding 0's rationale is this RFC's hypothesis word for word:

The project's mypy --strict checker, run as part of the review process, reports 'Success: no issues found', directly contradicting the claim that the use of Any is prohibited in strict mypy mode for this code.

Shown the output it withdraws the claim; not shown it, it confirms six copies. The 47% static tier is delivered on the review it was estimated from.

Open question 3 — is grep enough, or is mypy doing the work?

mypy is doing the work, and the six it kills are the six the RFC predicted. The one finding refuted without a checker citation is the one that mattered — see below.

The price

The single substantive finding — git diff failing other than through the promised MissingFixtureError, the one that produced the real fix in #399 — was also withdrawn, citing no checker, on reasoning that reproduces exactly the half of the argument diff_for's own comment rejects ("unlikely" is not the same as "cannot").

	baseline	treatment
false findings confirmed	18	12
true findings confirmed	1	0

But that is not attributable to the evidence, and the only reason we can say so is the control arm: no checker is cited on it, and judging the same findings with no evidence at all produced two flips of precisely that shape. The recorded verdicts are one draw of a sampled skeptic — gemini.py sends no temperature, guardian.yml sets none — so the RFC's proposed comparison (recorded vs. evidence) would have measured the effect plus the resampling noise and reported "evidence destroys recall" without support.

That correction belongs in this RFC's method section: the replay arm as described here is missing its control.

What this does not establish

The recall denominator is one. Rationale-text attribution is a weak proxy — citations barely moved while verdicts moved a lot. Two draws of the same arm differ, so single-draw asymmetries are not results. And the control has no per-finding rows: whether finding 4 dies without evidence too is the most informative cell still empty, at a cost of 24 skeptic calls.

New, unregistered

Six of 24 control rationales cite mypy or ruff although no checker output was supplied. The skeptic states what a checker would say without one having run — the same unverified move this RFC faults the finder for, one level up. Filing separately.

Where that leaves v1 and v2

v1 stays on: it removes a third of the false confirmations and the mechanism is visible in the rationales. v2 (pytest) is now harder to justify on this evidence, not easier — the tier that would have settled finding 4 is exactly the tier that executes PR code, and the skeptic here discarded it by argument rather than by measurement. Running the tests would only help if the skeptic prefers a result to its own reasoning, and this run gives no reason to think it does.

[zaebee]

Correction: the control rows reverse the attribution above

My previous comment said the loss of the one true finding was not attributable to the evidence — its refutation cited no checker, and the control arm had produced two confirmed → refuted flips of its own. That rested on an aggregate. The per-finding rows now exist for both arms, and they say the opposite.

finding	control (no evidence)	treatment (evidence)
0, 2, 3, 5, 12, 15 — claim a mypy rule the repo does not have	all confirmed	all refuted, all citing the checker
4 — git diff fails other than through MissingFixtureError (the real one)	confirmed, correctly reasoned	refuted, citing no checker

The control's rationale on finding 4 is right:

The subprocess.run call for git diff includes check=True, but the CalledProcessError it raises on command failure is not explicitly caught, leading to a different error type than MissingFixtureError …

The same model, with evidence in the prompt, withdrew it. The single control flip was finding 6, not finding 4. So the loss sits on the evidence arm's side of the ledger, and "eight refutations, one of them noise" was the wrong story — told by a number that cannot answer which.

The mechanism on the false claims, now visible from both sides

Finding 0 in the control:

… and if mypy --strict is configured to disallow Any explicitly, it would be flagged.

Finding 0 in the treatment:

The project's mypy --strict checker … reports 'Success: no issues found', directly contradicting the claim.

A conditional it cannot check becomes a reading. That is the RFC's design intent working exactly as written, and it is worth keeping separate from the recall cost, because the two are now known to be separate events rather than one trade.

What still is not established

Only one draw per arm carries rows; control draw 1 had two unidentified confirmed → refuted flips, so "finding 4 survives the control" is one observation and not a rate. The recall denominator is still one. And why the true finding falls is unexplained — its refutation cites nothing, so the same mechanism does not account for it. Unattributed moves run 3 in treatment against 1 in control, which is consistent with the evidence section pushing toward refutation generally, but at these counts that is a hypothesis.

Consequence for the proposal

v1's benefit and v1's cost are now both real and, on this review, both large: six false confirmations removed, the one true finding lost. That is not an argument to switch it off on one review with a recall denominator of one — but it is an argument that this cannot be decided on aggregates, and that any further arm must dump rows before the calls are spent rather than after.

Data and method: #406. Corrected in place there as well; the wrong version was published.