Yesterday's audit found 39 checks, across 35 suites, that cannot fail. Each was proved by applying a mutation, running the suite, and watching the named check stay green. Three of them are inside harness_selftest, the suite whose whole purpose is to stop this.
None of them ever went red, and none of them ever could. That is the problem this issue is about: the gate answers "did anything print FAIL", and has never answered "could anything print FAIL".
What the harness reports today, measured
- Per suite:
PASSED / FAILED / SKIPPED, exits 0 / 1 / 66. Three states already.
pgc_skip fails on a missing dependency unless waived with PGC_ALLOW_MISSING_<CAP>=1. That is already the right shape and this issue does not change it.- Per check: two states.
check, check_num and check_text print PASS or FAIL, and nothing else.
- The matrix: 237 registered suites; the last CI legs ran 228 and 231. Nothing reconciles those numbers, and no skip carries a machine-readable reason.
The four gaps, in the order they have cost us
- A check has no third state. When a check's input is absent — a fixture that did not build, a capability the server lacks — it either passes vacuously or fails for a reason unrelated to the property under test.
checks run: N counts it either way.
- Nothing reconciles a run.
registered == passed + failed + skipped is asserted nowhere. The project already demands inputs == sum(buckets) for any list-derived claim; the runner does not apply that rule to itself.
- Results are prose. Proving that a mutation reddened one named check means grepping
PASS/FAIL text. That is how a reverted guard once reported plain green while the check count silently fell from 190 to 186.
- Nothing records whether a check has ever been red. This is the gap that let 39 vacuous checks ship. An audit fixes today; only a ledger keeps it fixed.
Phases
Phase 1 — the third state. check_unrunnable NAME REASON_CODE DETAIL, a closed enum of reasons, check-level reconciliation, and a new PGC_EXIT_INCOMPLETE that is not a pass. Failure still outranks it.
Phase 2 — reconciliation. The runner asserts registered == passed + failed + skipped + incomplete with a reason beside every non-passed suite.
Phase 3 — machine-readable results. One line per check: suite, name, verdict, reason.
Phase 4 — the mutation ledger. Per check, the mutation that reds it and when it last did. This is the only phase that changes how we would notice next time.
Two constraints that came out of review and shape the design
The budget must be a tracked file, not an environment variable. PGC_SKIP_TIMING is the precedent: set in two workflow files, it suppressed whole suites for months and no diff ever showed it. A number in a tracked file is an object a reviewer sees change.
Phase 4 must start as an allowlist for new checks only. A gate that fails on any of 3,762 unledgered check sites is a gate someone disables under deadline, and then we are back at PGC_SKIP_TIMING with extra steps.
One measured limit on phase 2
12 of 238 suites are outside the accounting. bench_guards.sh and docs_style.sh never source lib.sh; smoke, audit, concurrency, phase2 through phase6, unique_conc and update_conc source it but never call pgc_summary and print their own verdicts — smoke.sh simply says SMOKE TEST PASSED. Any reconciliation is a lie for those twelve until they are brought in or exempted with a premise that fails when the list grows.
Yesterday's audit found 39 checks, across 35 suites, that cannot fail. Each was proved by applying a mutation, running the suite, and watching the named check stay green. Three of them are inside
harness_selftest, the suite whose whole purpose is to stop this.None of them ever went red, and none of them ever could. That is the problem this issue is about: the gate answers "did anything print FAIL", and has never answered "could anything print FAIL".
What the harness reports today, measured
PASSED/FAILED/SKIPPED, exits 0 / 1 / 66. Three states already.pgc_skipfails on a missing dependency unless waived withPGC_ALLOW_MISSING_<CAP>=1. That is already the right shape and this issue does not change it.check,check_numandcheck_textprintPASSorFAIL, and nothing else.The four gaps, in the order they have cost us
checks run: Ncounts it either way.registered == passed + failed + skippedis asserted nowhere. The project already demandsinputs == sum(buckets)for any list-derived claim; the runner does not apply that rule to itself.PASS/FAILtext. That is how a reverted guard once reported plain green while the check count silently fell from 190 to 186.Phases
Phase 1 — the third state.
check_unrunnable NAME REASON_CODE DETAIL, a closed enum of reasons, check-level reconciliation, and a newPGC_EXIT_INCOMPLETEthat is not a pass. Failure still outranks it.Phase 2 — reconciliation. The runner asserts
registered == passed + failed + skipped + incompletewith a reason beside every non-passed suite.Phase 3 — machine-readable results. One line per check: suite, name, verdict, reason.
Phase 4 — the mutation ledger. Per check, the mutation that reds it and when it last did. This is the only phase that changes how we would notice next time.
Two constraints that came out of review and shape the design
The budget must be a tracked file, not an environment variable.
PGC_SKIP_TIMINGis the precedent: set in two workflow files, it suppressed whole suites for months and no diff ever showed it. A number in a tracked file is an object a reviewer sees change.Phase 4 must start as an allowlist for new checks only. A gate that fails on any of 3,762 unledgered check sites is a gate someone disables under deadline, and then we are back at
PGC_SKIP_TIMINGwith extra steps.One measured limit on phase 2
12 of 238 suites are outside the accounting.
bench_guards.shanddocs_style.shnever sourcelib.sh;smoke,audit,concurrency,phase2throughphase6,unique_concandupdate_concsource it but never callpgc_summaryand print their own verdicts —smoke.shsimply saysSMOKE TEST PASSED. Any reconciliation is a lie for those twelve until they are brought in or exempted with a premise that fails when the list grows.