feat(scorecard): sweep every model config into one committed table (G4)

[henleda]

Roadmap G4, built but not yet run — see the last section.

vpcopilot bench <repo> --all-configs runs the same scan and answer key through every
config/agents*.yaml into out-<tag>, and writes benchmarks/RESULTS.md with one row per config.

Closes both halves of the old BACKLOG item

G4's note had this half-right, and reconciling it against the code corrected it:

• verify precision is genuinely new — real-over-reported (matched key or bonus entries, over
  everything the verifier let through). Recall already existed as bench.discovery_recall and
  answers a different question: a run can find every labelled vuln and still bury them in false
  positives.
• discovery duplicates were logged by _dedup_findings but never reached metrics.json. They
  do now, as discovery.duplicates_dropped, and appear as a scorecard column.

Failure handling is the point, not a detail

A four-way run is expensive. A config whose run fails gets a row carrying its error rather than
vanishing from the table — losing the other three because one provider 404s would be the worst
possible failure mode, and a table that silently omits the provider that crashed reads as if it was
never tried. Pinned by a test.

Two acceptance criteria answered honestly rather than faked

"Re-running with the same seed and model reproduces the score" is not achievable here. Anthropic
accepts no seed at all, and none of the four providers guarantee determinism. Rather than add seed
plumbing that would imply a guarantee it can't make, RESULTS.md states the run date, says scores
vary between runs, and tells the reader to re-run before reading anything into a small difference.

Token cost is absent by the decision of 2026-07-27 — and the table says so, rather than leaving
a reader to wonder whether it was measured and omitted.

MODELS.md now links the table instead of restating numbers, keeping the qualitative findings (the
"gpt-4o trusted the 'intentional' comments" hardening win) that no table captures.

Verification

Against a real scored run — out-claude with --rescore, so no model calls and no quota:

expected 9 · found 4 · discovery_recall 0.44
verified 10 · verify_precision 0.40 · noise 6
triage_accuracy 0.75 · wall_time_s 189.64

(That run scanned a different target than the Nimbus key, so the numbers are meaningless — it
proves the plumbing populates, which is what a rescore can prove.)

12 new tests · 262 pass · coverage 73% · ruff clean.

Why the roadmap box is still unchecked

benchmarks/RESULTS.md does not exist yet, and the acceptance requires it. Producing it means a
real four-way run that spends live Gemini and OpenAI quota and takes a while — a deliberate start
for the maintainer, not something to kick off unasked. The capability is complete and tested; the
table is one command away.

🤖 Generated with Claude Code