Skip to content

Make every gate fail when it examines nothing - #96

Merged
AdamClemens merged 2 commits into
mainfrom
fix/gates-that-pass-on-an-empty-sweep
Sep 14, 2026
Merged

AdamClemens merged 2 commits into
mainfrom
fix/gates-that-pass-on-an-empty-sweep

Conversation

@AdamClemens

@AdamClemens AdamClemens commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Make every gate fail when it examines nothing

Asked what else the repository might not be implementing as believed, and
answered it mechanically rather than by reasoning: every validator run with
its discovery function replaced by one returning an empty set.

Four of the nine validators in make ci returned success. check_docs,
check_duplicate_blocks, check_references and check_scenarios printed
their ordinary success message and exited 0 over nothing at all. In CI that
output is indistinguishable from a clean run, which makes it worse than
having no gate: the pipeline reports a check that did not happen.

docs/practices.md's "A rule that matches nothing reports nothing" has
recorded this shape since 2026-08-30. All three instances there are a check
whose pattern matched nothing. This is the fourth and the first where the
input was empty -- which is why it survived: every prior instance trained
attention on the pattern.

Two of the four are their own lesson.

check_references.py states the principle in a comment and did not
implement it.
The twelve lines above EXTS exist because .feature was
missing from that tuple for two days while every feature path in every
document went unchecked behind a green gate. They end "a rule that matches
nothing reports nothing, which reads exactly like a pass". Two hundred lines
below, main() printed "Every path named in prose resolves" over an empty
file list.

check_scenarios.py guarded one half of itself and not the other -- and
that was not an omission.
Its bindings side has always failed loudly; its
features side returned 0, deliberately, with a test stating why: the gate
was built before Stage 4 wrote its first .feature, so an empty
tests/features/ was then the repository's ordinary state. That reason
expired on 2026-08-27 when TASK-023 landed the first feature file, and
nothing revisited it -- 17 days, under the one gate
adr/ADR-007-executable-acceptance-criteria.md rests on entirely, since
pytest says nothing about a .feature file no module runs. An exemption
states its own expiry condition and then nobody is assigned to watch it.
tests/unit/test_check_scenarios.py's test is inverted here rather than
deleted, with that history in its docstring.

The fix is a sweep, not four guards. Four guards would be four fixes and
a hope. tests/unit/test_validator_guards.py runs every check_*.py with
nothing to find and requires a non-zero exit and a message saying so, with
test_every_validator_is_covered failing if a validator is added without an
entry. check_claims is excluded by name with its reason (advisory by
design, exits 0 even with findings) rather than by absence -- an unexplained
omission is how this gap existed. The sweep failed for exactly those four
before the fix, which is its own mutation evidence.

Three other classes were probed and came back clean, recorded so the
next pass does not redo them: no configuration field is unread by engine
code (TASK-055's sweep generalised to all 14 schema dataclasses); the
ADR-003 registries agree exactly with the schema Literals that select them
(periodic and boussinesq_buoyancy look like mismatches and are both
deliberate, the latter pinned in three directions by
tests/integration/test_boussinesq_buoyancy_registration.py); and all 163
@then step definitions assert something.

The general form, carried into docs/practices.md as the rule rather than
the four instances: when a check reports success, ask what it would have
reported had it examined nothing. If those are the same output, it is not
yet a check.

🤖 Generated with Claude Code


Also in this branch: labelling the self-test window that is supposed to diverge

Unrelated to the gates, and riding along rather than taking its own branch.

While this PR's CI was running, the maintainer watched a lid-driven cavity tear itself apart on screen and reported it as a suspected engine failure. It was test_frame_failure.py's own display-guarded case, which runs the cavity at 2.05× its stability limit in a real glfw window because proving "a diverged run terminates rather than hanging" needs the real backend. It diverges at step 17 of 25, by design.

Checked rather than assumed: the shipped 16×16 cavity runs 1200 frames to a clean steady state (max|u| → 0.7611 against a lid speed of 1.0, kinetic energy asymptoting to 6.37, monotone throughout), and this is the only one of four windowed integration modules whose config diverges.

Behaviour right, presentation wrong. Stage 7 exists on the premise that the render window explains itself; this one said PyFlow while blowing up. It now sets rendering.title, drawn in the scene by hud.build_title_text, verified by reading it back out of the real rendered scene:

'PyFlow self-test -- this run is SUPPOSED to diverge'

No assertion changes — only what a human sees while it runs.

AdamClemens and others added 2 commits September 13, 2026 22:39
Asked what else the repository might not be implementing as believed, and
answered it mechanically rather than by reasoning: every validator run with
its discovery function replaced by one returning an empty set.

**Four of the nine validators in `make ci` returned success.** `check_docs`,
`check_duplicate_blocks`, `check_references` and `check_scenarios` printed
their ordinary success message and exited 0 over nothing at all. In CI that
output is indistinguishable from a clean run, which makes it worse than
having no gate: the pipeline reports a check that did not happen.

`docs/practices.md`'s "A rule that matches nothing reports nothing" has
recorded this shape since 2026-08-30. All three instances there are a check
whose *pattern* matched nothing. This is the fourth and the first where the
*input* was empty -- which is why it survived: every prior instance trained
attention on the pattern.

Two of the four are their own lesson.

**`check_references.py` states the principle in a comment and did not
implement it.** The twelve lines above `EXTS` exist because `.feature` was
missing from that tuple for two days while every feature path in every
document went unchecked behind a green gate. They end "a rule that matches
nothing reports nothing, which reads exactly like a pass". Two hundred lines
below, `main()` printed "Every path named in prose resolves" over an empty
file list.

**`check_scenarios.py` guarded one half of itself and not the other -- and
that was not an omission.** Its bindings side has always failed loudly; its
features side returned 0, deliberately, with a test stating why: the gate
was built before Stage 4 wrote its first `.feature`, so an empty
`tests/features/` was then the repository's ordinary state. That reason
expired on 2026-08-27 when TASK-023 landed the first feature file, and
nothing revisited it -- 17 days, under the one gate
`adr/ADR-007-executable-acceptance-criteria.md` rests on entirely, since
pytest says nothing about a `.feature` file no module runs. An exemption
states its own expiry condition and then nobody is assigned to watch it.
`tests/unit/test_check_scenarios.py`'s test is inverted here rather than
deleted, with that history in its docstring.

**The fix is a sweep, not four guards.** Four guards would be four fixes and
a hope. `tests/unit/test_validator_guards.py` runs every `check_*.py` with
nothing to find and requires a non-zero exit *and* a message saying so, with
`test_every_validator_is_covered` failing if a validator is added without an
entry. `check_claims` is excluded by name with its reason (advisory by
design, exits 0 even with findings) rather than by absence -- an unexplained
omission is how this gap existed. The sweep failed for exactly those four
before the fix, which is its own mutation evidence.

**Three other classes were probed and came back clean**, recorded so the
next pass does not redo them: no configuration field is unread by engine
code (TASK-055's sweep generalised to all 14 schema dataclasses); the
ADR-003 registries agree exactly with the schema `Literal`s that select them
(`periodic` and `boussinesq_buoyancy` look like mismatches and are both
deliberate, the latter pinned in three directions by
`tests/integration/test_boussinesq_buoyancy_registration.py`); and all 163
`@then` step definitions assert something.

The general form, carried into `docs/practices.md` as the rule rather than
the four instances: when a check reports success, ask what it would have
reported had it examined nothing. If those are the same output, it is not
yet a check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The maintainer watched a lid-driven cavity tear itself apart on screen --
arrows exploding out of the frame -- and reported it as a suspected engine
failure. It was not one. It was
`tests/integration/test_frame_failure.py::
test_an_interactive_run_terminates_rather_than_hanging_when_a_frame_raises`,
which runs the cavity refined to 64x64 with the timestep left at 2.05x its
own stability limit, in a **real glfw window**, because proving "a diverged
run terminates instead of hanging" needs the real backend rather than
offscreen. It is display-guarded, so it only does this on a machine with a
display, and it diverges at step 17 of 25 exactly as designed.

Checked before concluding that, rather than assumed: the shipped 16x16
cavity runs 1200 frames to a clean steady state -- max|u| rising to 0.7611
against a lid speed of 1.0, kinetic energy asymptoting to 6.37, monotone
throughout. And of the four integration modules that open real windows,
this is the only one whose config diverges.

So the behaviour is right and the presentation is wrong. Stage 7 exists on
the premise that the render window explains itself without the config file
beside it, and this window said "PyFlow" while a simulation blew up inside
it. Anyone watching reads that as a broken engine, which is precisely what
happened.

The fixture now sets `rendering.title`, which `hud.build_title_text` draws
*in* the scene rather than only in the OS title bar. Verified by reading it
back out of the real rendered scene with the same private-`_text_blocks`
readback `tests/unit/test_bootstrap.py` already uses, not by assuming the
config field took effect:

    'PyFlow self-test -- this run is SUPPOSED to diverge'
    'step 2  t = 0.001 s\ncell: 0.01562 m x 0.01562 m\ndomain: 1 m x 1 m'

No assertion in that module changes -- it still checks the exit code and the
stderr substring. This only changes what a human sees while it runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AdamClemens
AdamClemens merged commit 44ce52d into main Sep 14, 2026
2 checks passed
@AdamClemens
AdamClemens deleted the fix/gates-that-pass-on-an-empty-sweep branch September 14, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant