feat(guardian): tell the skeptic when no checker ran, and which verdict that takes (#407)

[zaebee]

Closes #407.

The defect

_evidence_section(None) returned the empty string, and the emptiness was the defect. Measuring #401 needed a control arm judging the same 24 findings with no checker output at all — and in it 6 of 24 rationales cited mypy or ruff anyway, every one to confirm a false claim, on a conditional the model could not check:

The use of Any does indeed bypass strict type checking in mypy … and if mypy --strict is configured to disallow Any explicitly, it would be flagged.

Told nothing about checkers, the skeptic supplied its own — the same unverified move the RFC faults the finder for, one level up.

Two prompts, both measured

The replay control arm is this configuration, so each attempt costs 24 skeptic calls and no finder call. Registered before running: primary — how many of the six checker-appeals stop being confirmed on that basis; recall guard — finding 4, the one substantive finding of the 24, must stay confirmed; noise floor — two draws of the original prompt moved 4 and 1 verdicts, so anything smaller is not a result.

Attempt 1: a prohibition. Failed.

Do not rest a verdict on what a checker would report — not to confirm a finding … and not to refute one …

Citations 6/24 → 5/24, five of six still confirmed. Inside the noise floor, and the text got worse — the conditional became a flat assertion:

This is an issue that would be caught by the project's mandatory mypy --strict gate.

There is no such gate over that file: make type-check runs mypy src and pyproject.toml excludes scripts. Told what not to do, the model complied in form and confirmed anyway, with more confidence than before.

Attempt 2: a directive. Worked.

If a finding rests entirely on what a checker reports … it cannot be settled here. Mark it 'uncertain'. Not 'confirmed': you have not seen a checker agree. Not 'refuted': you have not seen one disagree either.

finding	original control	directive
0, 2, 3, 5, 15 — claim a mypy rule the repo does not have	confirmed	uncertain
12 — sys.path "violates type safety"	confirmed	confirmed
4 — the one true finding	confirmed	confirmed ✅
21, 22 — propose relaxing a ratchet	confirmed	refuted

Five of six moved, all one direction, all on cited-checker rows — outside the noise floor and mechanistically legible. Finding 0 now states its position instead of inventing a gate:

Without access to the project's mypy configuration or output, it is not possible to confirm if this specific usage would be flagged.

The recall guard held with the correct reasoning: CalledProcessError is raised and not caught, which is the defect actually fixed in #399.

Unregistered gain: findings 21 and 22 were refuted. Those are the two the RFC singled out as beyond any tool — wrong about intent, not behaviour. The model got there unaided: "a deliberate guardrail … its failure due to a diminished corpus is intended behavior, not a defect." One draw, not predicted; recorded as an observation.

uncertain rather than refuted because it is the true state: the checker neither agreed nor disagreed, because it never ran. Refuting an unadjudicable claim is the recall regression this component exists to avoid.

What this does not buy

uncertain is kept. apply_judgements multiplies confidence by 0.9 as a ranking signal and nothing more, and render_review_body takes threshold: int = 0. All five findings are still posted; only their order changes. This is an epistemic fix, not a noise fix — suppression is a separate knob (impact_threshold, which #246 §3.5 records as shipping inert) and deserves its own evidence rather than being folded in here.

One draw per arm; recall denominator of one. The primary effect is well outside the noise floor, nothing else here is.

Note

skeptic.py is inside the review-fingerprint closure (43 modules), so this re-mints every reviewer identity — by design (#375).

Data, including the failed first attempt, under benchmarks/guardian/experiments/407-no-checker-appeal/.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a directive to the skeptic prompt when no checker output is available, instructing the model to mark findings that rely entirely on checker output as 'uncertain' rather than 'confirmed' or 'refuted'. This addresses an issue where the model would assume checker configurations when none were provided. Feedback on the pull request includes improving the reproducibility of the command in the README by using a repository-root-relative path, and making the test assertion for the prompt's text more robust by normalizing whitespace.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[zaebee]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements a directive instructing the skeptic model to mark findings that rely entirely on checker reports as 'uncertain' when no checker output is available. This change is documented in a new experiment README, supported by JSONL data files, and verified with updated unit tests. As there are no review comments, I have no feedback to provide.