Make every gate fail when it examines nothing - #96
Merged
Merged
Conversation
Asked what else the repository might not be implementing as believed, and answered it mechanically rather than by reasoning: every validator run with its discovery function replaced by one returning an empty set. **Four of the nine validators in `make ci` returned success.** `check_docs`, `check_duplicate_blocks`, `check_references` and `check_scenarios` printed their ordinary success message and exited 0 over nothing at all. In CI that output is indistinguishable from a clean run, which makes it worse than having no gate: the pipeline reports a check that did not happen. `docs/practices.md`'s "A rule that matches nothing reports nothing" has recorded this shape since 2026-08-30. All three instances there are a check whose *pattern* matched nothing. This is the fourth and the first where the *input* was empty -- which is why it survived: every prior instance trained attention on the pattern. Two of the four are their own lesson. **`check_references.py` states the principle in a comment and did not implement it.** The twelve lines above `EXTS` exist because `.feature` was missing from that tuple for two days while every feature path in every document went unchecked behind a green gate. They end "a rule that matches nothing reports nothing, which reads exactly like a pass". Two hundred lines below, `main()` printed "Every path named in prose resolves" over an empty file list. **`check_scenarios.py` guarded one half of itself and not the other -- and that was not an omission.** Its bindings side has always failed loudly; its features side returned 0, deliberately, with a test stating why: the gate was built before Stage 4 wrote its first `.feature`, so an empty `tests/features/` was then the repository's ordinary state. That reason expired on 2026-08-27 when TASK-023 landed the first feature file, and nothing revisited it -- 17 days, under the one gate `adr/ADR-007-executable-acceptance-criteria.md` rests on entirely, since pytest says nothing about a `.feature` file no module runs. An exemption states its own expiry condition and then nobody is assigned to watch it. `tests/unit/test_check_scenarios.py`'s test is inverted here rather than deleted, with that history in its docstring. **The fix is a sweep, not four guards.** Four guards would be four fixes and a hope. `tests/unit/test_validator_guards.py` runs every `check_*.py` with nothing to find and requires a non-zero exit *and* a message saying so, with `test_every_validator_is_covered` failing if a validator is added without an entry. `check_claims` is excluded by name with its reason (advisory by design, exits 0 even with findings) rather than by absence -- an unexplained omission is how this gap existed. The sweep failed for exactly those four before the fix, which is its own mutation evidence. **Three other classes were probed and came back clean**, recorded so the next pass does not redo them: no configuration field is unread by engine code (TASK-055's sweep generalised to all 14 schema dataclasses); the ADR-003 registries agree exactly with the schema `Literal`s that select them (`periodic` and `boussinesq_buoyancy` look like mismatches and are both deliberate, the latter pinned in three directions by `tests/integration/test_boussinesq_buoyancy_registration.py`); and all 163 `@then` step definitions assert something. The general form, carried into `docs/practices.md` as the rule rather than the four instances: when a check reports success, ask what it would have reported had it examined nothing. If those are the same output, it is not yet a check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The maintainer watched a lid-driven cavity tear itself apart on screen --
arrows exploding out of the frame -- and reported it as a suspected engine
failure. It was not one. It was
`tests/integration/test_frame_failure.py::
test_an_interactive_run_terminates_rather_than_hanging_when_a_frame_raises`,
which runs the cavity refined to 64x64 with the timestep left at 2.05x its
own stability limit, in a **real glfw window**, because proving "a diverged
run terminates instead of hanging" needs the real backend rather than
offscreen. It is display-guarded, so it only does this on a machine with a
display, and it diverges at step 17 of 25 exactly as designed.
Checked before concluding that, rather than assumed: the shipped 16x16
cavity runs 1200 frames to a clean steady state -- max|u| rising to 0.7611
against a lid speed of 1.0, kinetic energy asymptoting to 6.37, monotone
throughout. And of the four integration modules that open real windows,
this is the only one whose config diverges.
So the behaviour is right and the presentation is wrong. Stage 7 exists on
the premise that the render window explains itself without the config file
beside it, and this window said "PyFlow" while a simulation blew up inside
it. Anyone watching reads that as a broken engine, which is precisely what
happened.
The fixture now sets `rendering.title`, which `hud.build_title_text` draws
*in* the scene rather than only in the OS title bar. Verified by reading it
back out of the real rendered scene with the same private-`_text_blocks`
readback `tests/unit/test_bootstrap.py` already uses, not by assuming the
config field took effect:
'PyFlow self-test -- this run is SUPPOSED to diverge'
'step 2 t = 0.001 s\ncell: 0.01562 m x 0.01562 m\ndomain: 1 m x 1 m'
No assertion in that module changes -- it still checks the exit code and the
stderr substring. This only changes what a human sees while it runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make every gate fail when it examines nothing
Asked what else the repository might not be implementing as believed, and
answered it mechanically rather than by reasoning: every validator run with
its discovery function replaced by one returning an empty set.
Four of the nine validators in
make cireturned success.check_docs,check_duplicate_blocks,check_referencesandcheck_scenariosprintedtheir ordinary success message and exited 0 over nothing at all. In CI that
output is indistinguishable from a clean run, which makes it worse than
having no gate: the pipeline reports a check that did not happen.
docs/practices.md's "A rule that matches nothing reports nothing" hasrecorded this shape since 2026-08-30. All three instances there are a check
whose pattern matched nothing. This is the fourth and the first where the
input was empty -- which is why it survived: every prior instance trained
attention on the pattern.
Two of the four are their own lesson.
check_references.pystates the principle in a comment and did notimplement it. The twelve lines above
EXTSexist because.featurewasmissing from that tuple for two days while every feature path in every
document went unchecked behind a green gate. They end "a rule that matches
nothing reports nothing, which reads exactly like a pass". Two hundred lines
below,
main()printed "Every path named in prose resolves" over an emptyfile list.
check_scenarios.pyguarded one half of itself and not the other -- andthat was not an omission. Its bindings side has always failed loudly; its
features side returned 0, deliberately, with a test stating why: the gate
was built before Stage 4 wrote its first
.feature, so an emptytests/features/was then the repository's ordinary state. That reasonexpired on 2026-08-27 when TASK-023 landed the first feature file, and
nothing revisited it -- 17 days, under the one gate
adr/ADR-007-executable-acceptance-criteria.mdrests on entirely, sincepytest says nothing about a
.featurefile no module runs. An exemptionstates its own expiry condition and then nobody is assigned to watch it.
tests/unit/test_check_scenarios.py's test is inverted here rather thandeleted, with that history in its docstring.
The fix is a sweep, not four guards. Four guards would be four fixes and
a hope.
tests/unit/test_validator_guards.pyruns everycheck_*.pywithnothing to find and requires a non-zero exit and a message saying so, with
test_every_validator_is_coveredfailing if a validator is added without anentry.
check_claimsis excluded by name with its reason (advisory bydesign, exits 0 even with findings) rather than by absence -- an unexplained
omission is how this gap existed. The sweep failed for exactly those four
before the fix, which is its own mutation evidence.
Three other classes were probed and came back clean, recorded so the
next pass does not redo them: no configuration field is unread by engine
code (TASK-055's sweep generalised to all 14 schema dataclasses); the
ADR-003 registries agree exactly with the schema
Literals that select them(
periodicandboussinesq_buoyancylook like mismatches and are bothdeliberate, the latter pinned in three directions by
tests/integration/test_boussinesq_buoyancy_registration.py); and all 163@thenstep definitions assert something.The general form, carried into
docs/practices.mdas the rule rather thanthe four instances: when a check reports success, ask what it would have
reported had it examined nothing. If those are the same output, it is not
yet a check.
🤖 Generated with Claude Code
Also in this branch: labelling the self-test window that is supposed to diverge
Unrelated to the gates, and riding along rather than taking its own branch.
While this PR's CI was running, the maintainer watched a lid-driven cavity tear itself apart on screen and reported it as a suspected engine failure. It was
test_frame_failure.py's own display-guarded case, which runs the cavity at 2.05× its stability limit in a real glfw window because proving "a diverged run terminates rather than hanging" needs the real backend. It diverges at step 17 of 25, by design.Checked rather than assumed: the shipped 16×16 cavity runs 1200 frames to a clean steady state (max|u| → 0.7611 against a lid speed of 1.0, kinetic energy asymptoting to 6.37, monotone throughout), and this is the only one of four windowed integration modules whose config diverges.
Behaviour right, presentation wrong. Stage 7 exists on the premise that the render window explains itself; this one said
PyFlowwhile blowing up. It now setsrendering.title, drawn in the scene byhud.build_title_text, verified by reading it back out of the real rendered scene:No assertion changes — only what a human sees while it runs.