Repository navigation
Open
Labels
area: test-suiteThe validator's own tests, mutation testing and fuzzingThe validator's own tests, mutation testing and fuzzingenhancementNew feature or requestNew feature or request
Description
Mutation testing: carry results between CI runs, and reset them when the tests that cover a function change
The PR mutation job (.github/workflows/mutation-testing.yml) starts from an empty mutants/ on every push. So each push pays for:
- the full stats pass, which runs the whole suite to map tests to functions (15–20 minutes);
- every mutant of the changed functions, again, even when nothing about them changed since the last push.
A PR with many pushes (such as #310) redoes the same work each time and often stops at the 30-minute budget.
What mutmut 3.8.1 already does
From its source:
- It keeps its results in
mutants/: per-mutant results in*.meta, plusmutmut-stats.jsonwithtests_by_mangled_function_name,duration_by_testand per-function source hashes. - When a function's source hash changes, only that function's mutants are re-tested; the other verdicts are kept.
- A change to the pytest settings resets every result.
- Non-Python files changed since the last full run are found with git. The default (
on_dependency_change = "warn") only warns;"rerun"resets everything. - New tests get stats collected for themselves only.
The two gaps
- Nothing is kept between runs. The CI job never restores
mutants/, so none of the above helps. - Editing a test does not reset anything. Only a change to the function's own source, the config or a dependency resets a verdict. Changing a test file resets nothing:
- strengthen a test, and its survivors stay "survived";
- weaken a test, and its killed mutants stay "killed", which hides a real gap.
Proposal
- Cache
mutants/withactions/cache. Keys are immutable, so save under a unique key (for examplemutmut-<run id>) and restore the newest by prefix. Record the commit each cache was made at. - Reset after restore, before
mutmut run.- List the test files changed between the cached commit and HEAD (
git diff --name-only <cached>..HEAD -- tests/). - Using
tests_by_mangled_function_name, find every function covered by a test in those files. - Reset those functions' verdicts to "not checked" in the
.metafiles. - A change to shared test helpers or fixtures should count as a change to every test that uses them. If that is hard to know, reset everything when anything under
tests/outside the test modules changes.
- List the test files changed between the cached commit and HEAD (
- Set
on_dependency_change = "rerun"in[tool.mutmut], so a change to the requirements or other non-Python inputs resets everything instead of warning. - Give new PRs a warm start. A PR can read caches from its own branch and its base branch only, so the first push of every PR starts cold. A job on pushes to
mainthat runs only mutmut's stats pass (no mutants) and savesmutants/would give each new PR a stats map to start from. - The priming run and
git -C mutants initstep should become unnecessary on a restored cache. Keep them for a cold start.
Done when
- A second push to a PR that changes nothing mutmut sees skips the stats pass and re-tests no mutants.
- Changing a test re-tests exactly the mutants of the functions that test covers.
- Changing
pyproject.tomlor a requirements file re-tests everything. - The reset step has unit tests in the repo.
Related: #307 (the cost of mutation runs), #308.
🤖 Generated with Claude Code
Metadata
Metadata
Assignees
Labels
area: test-suiteThe validator's own tests, mutation testing and fuzzingThe validator's own tests, mutation testing and fuzzingenhancementNew feature or requestNew feature or request
Activity