feat(ci): a floor under the size of the test suite itself (#405)

[zaebee]

Closes #405.

The incident

On 2026-08-17 a stale-tree push over #339 deleted 62 files — 5 modules under src/cgis (review_fingerprint.py, evidence.py, martian.py, calibrate.py, registry.py) and 24 test files. Python Verification passed. 1922 collected tests became 1264 and nothing said so, because the tests that would have failed were deleted by the same commit.

To CI, "the module and its tests are gone" and "both still pass" are one observation.

Why the existing ratchets could not help

>= 72 in test_recordings_from_corpus.py and >= 16 in test_backfill_calibration_fingerprint.py were written against exactly this shape of silence. They live inside test files, and this failure deletes the file. A guard shipped inside the thing it guards is removed by the event it exists to catch — so this is a step in ci.yml, not a test.

The load-bearing half: two baselines

source	role
git show origin/<base>:.github/test-count-baseline	the floor — a branch cannot edit it
the file in the branch's tree	the copy a branch is allowed to raise

A stale tree carries a stale baseline. The tree in the incident was six days old and held roughly the count it had then, so checking a branch against its own file compares 1264 to 1264 and passes. Against the base branch's 1932 it fails.

fetch-depth: 0 is already on the checkout step (for Sonar), which is what makes origin/<base> readable.

Four rules

1. actual >= floor — the suite may not shrink below the base branch's number. Cannot be evaded by editing the file in the branch.
2. here >= floor — the baseline may only rise, or a branch lowers it and every branch cut afterwards inherits a weaker floor.
3. here <= actual — it may not claim more tests than exist, or the next branch starts red through no fault of its own.
4. actual - here <= 150 — it may not fall far behind, or it stops bounding anything.

Rules 3 and 4 are suppressed while rule 1 is broken: "lower the baseline to 1304" is the wrong advice to print beside "628 tests have gone missing".

Verified end to end, not only by unit test

A worktree of this branch with 24 test files removed:

The suite collects 1304 tests; 1932 were recorded on the base branch. 628 have gone
missing. Deleting a module together with its tests leaves CI green, which is why this
is checked separately from whether the tests pass.
exit=1

The intact tree exits 0.

Parsing, measured rather than assumed

pytest --collect-only -q ends in three shapes:

1957 tests collected in 1.84s
137/1957 tests collected (1820 deselected) in 1.95s     # under -k — the *right* number is 1957
no tests collected in 0.00s                             # collection failed; not a count of zero

The third raises CannotCountError and exits 2. A floor that cannot fail is worse than no floor, because it reads as protection.

Bootstrap

A base ref with no baseline yet returns None rather than a number, and the step prints on stderr that the anti-deletion rule did not apply — a green step where the rule never ran must not look like one where it ran and passed. This covers the introducing commit and branches cut before it.

It cannot be reached by deleting the file later: such a branch fails on the working-tree read first. An unresolvable base ref is a different thing (a broken workflow) and raises — told apart by resolving the ref, not by matching git's error text.

Out of scope, worth confirming separately

The workflow file is itself part of the pushed tree. A stale push that removed ci.yml would run no check at all; branch protection with required status checks is the answer to that, not this PR.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to prevent silent test deletions by establishing a baseline test count floor. It adds a baseline file, a script (check_test_count.py) to verify the current test count against the base branch's baseline, and a comprehensive test suite for this script. The review feedback suggests improving the robustness of the test count parser by selecting the last regex match in the output to avoid false matches from logs, along with adding a corresponding unit test.

[gemini-code-assist]

Using _COLLECTED.search(output) returns the first match in the pytest output. If any test, plugin, or warning prints a message containing a matching pattern (e.g., in stdout or stderr during collection), it could result in an incorrect test count being parsed. \n\nTo make this more robust, we should find all matches and select the last one, as the actual pytest summary line is guaranteed to be at the very end of the output.

    matches = list(_COLLECTED.finditer(output))\n    if not matches:\n        _msg = (\n            \"Could not find a collected-test count in pytest's output. The last line is \"\n            \"normally '<N> tests collected'; 'no tests collected' means collection failed \"\n            \"and is not a count of zero.\\n\"\n            f\"--- output ---\\n{output.strip()[-2000:]}\"\n        )\n        raise CannotCountError(_msg)\n    return int(matches[-1].group(2))

[gemini-code-assist]

Add a unit test to verify that parse_collected correctly handles multiple matches in the output (e.g., when logs or test names contain matching patterns) and always selects the last match (the actual summary line).

    def test_the_refusal_carries_the_output_it_could_not_parse(self) -> None:\n        \"\"\"A 'could not run' with no output is a bug report nobody can action.\"\"\"\n        with pytest.raises(CannotCountError, match=\"ImportError: cannot import name\"):\n            parse_collected(\"ImportError: cannot import name 'x' from 'y'\")\n\n    def test_multiple_matches_picks_the_last_one(self) -> None:\n        \"\"\"If earlier output or log messages contain matching text, the last one (the summary) is chosen.\"\"\"\n        output = \"10 tests collected\\nSome other logs...\\n1932 tests collected in 1.83s\"\n        assert parse_collected(output) == 1932

[zaebee]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new test-count baseline guard script and its corresponding unit tests to prevent silent test suite deletions in CI. The reviewer feedback suggests expanding the git revision validation regex to support relative references (such as HEAD~1 and HEAD^), updating the validation error message and unit tests accordingly, and handling potential FileNotFoundError exceptions when executing uv and git subprocesses to ensure clean exits.

[gemini-code-assist]

The current _REF regular expression is overly restrictive and rejects valid git revision expressions such as HEAD~1, HEAD^, or origin/main~2. This limits the usability of the script in CI workflows where relative references are commonly used. Consider updating the regex to allow common git revision characters like ~, ^, and @.

Suggested change

	_REF = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/-]*$")
	_REF = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/~^@-]*$")

[gemini-code-assist]

Update the validation error message to reflect the newly allowed characters (~, ^, and @) in the git revision validation.

Suggested change

	_msg = (
	f"{ref!r} is not a usable git revision. It must start with a letter or digit and "
	f"hold only letters, digits, and `._/-` — a value beginning with `-` reaches git "
	f"as an option rather than a revision."
	)
	_msg = (
	f"{ref!r} is not a usable git revision. It must start with a letter or digit and "
	f"hold only letters, digits, and `._/-~^@` — a value beginning with `-` reaches git "
	f"as an option rather than a revision."
	)

[gemini-code-assist]

If uv is not installed or not available in the system's PATH, subprocess.run will raise a FileNotFoundError. This unhandled exception will cause the script to crash with a traceback and exit with status 1, rather than exiting cleanly with status 2 as intended for "could not run" scenarios. Wrapping the call in a try-except block to catch FileNotFoundError and raise CannotCountError ensures robust error handling.

Suggested change

	result = subprocess.run(
	["uv", "run", "--frozen", "pytest", "--collect-only", "-q"],
	cwd=repo_root,
	capture_output=True,
	text=True,
	check=False,
	)
	try:
	result = subprocess.run(
	["uv", "run", "--frozen", "pytest", "--collect-only", "-q"],
	cwd=repo_root,
	capture_output=True,
	text=True,
	check=False,
	)
	except FileNotFoundError as exc:
	raise CannotCountError(
	f"Could not run 'uv'. Please ensure 'uv' is installed and available in your PATH. "
	f"Original error: {exc}"
	) from exc

[gemini-code-assist]

Similarly to the uv process execution, if git is not installed or not available in the system's PATH, subprocess.run will raise a FileNotFoundError. Wrapping this call in a try-except block to catch FileNotFoundError and raise CannotCountError ensures a clean exit with status 2 and an actionable error message.

Suggested change

	resolved = subprocess.run(
	["git", "rev-parse", "--verify", "--quiet", f"{ref}^{{commit}}"],
	cwd=repo_root,
	capture_output=True,
	check=False,
	)
	try:
	resolved = subprocess.run(
	["git", "rev-parse", "--verify", "--quiet", f"{ref}^{{commit}}"],
	cwd=repo_root,
	capture_output=True,
	check=False,
	)
	except FileNotFoundError as exc:
	raise CannotCountError(
	f"Could not run 'git'. Please ensure 'git' is installed and available in your PATH. "
	f"Original error: {exc}"
	) from exc

[gemini-code-assist]

Add test cases for relative git references (such as HEAD~1, HEAD^, and origin/main~2) to ensure they are correctly accepted by the updated _validated_ref logic.

Suggested change

	@pytest.mark.parametrize("ref", ["main", "origin/main", "HEAD", "release/1.2", "98380bc"])
	@pytest.mark.parametrize("ref", ["main", "origin/main", "HEAD", "HEAD~1", "HEAD^", "origin/main~2", "release/1.2", "98380bc"])

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud