Skip to content

Mutation testing: carry results between CI runs, and reset them when the covering tests change #311

Open

Description

@leftibot

Mutation testing: carry results between CI runs, and reset them when the tests that cover a function change

The PR mutation job (.github/workflows/mutation-testing.yml) starts from an empty mutants/ on every push. So each push pays for:

  • the full stats pass, which runs the whole suite to map tests to functions (15–20 minutes);
  • every mutant of the changed functions, again, even when nothing about them changed since the last push.

A PR with many pushes (such as #310) redoes the same work each time and often stops at the 30-minute budget.

What mutmut 3.8.1 already does

From its source:

  • It keeps its results in mutants/: per-mutant results in *.meta, plus mutmut-stats.json with tests_by_mangled_function_name, duration_by_test and per-function source hashes.
  • When a function's source hash changes, only that function's mutants are re-tested; the other verdicts are kept.
  • A change to the pytest settings resets every result.
  • Non-Python files changed since the last full run are found with git. The default (on_dependency_change = "warn") only warns; "rerun" resets everything.
  • New tests get stats collected for themselves only.

The two gaps

  1. Nothing is kept between runs. The CI job never restores mutants/, so none of the above helps.
  2. Editing a test does not reset anything. Only a change to the function's own source, the config or a dependency resets a verdict. Changing a test file resets nothing:
    • strengthen a test, and its survivors stay "survived";
    • weaken a test, and its killed mutants stay "killed", which hides a real gap.

Proposal

  1. Cache mutants/ with actions/cache. Keys are immutable, so save under a unique key (for example mutmut-<run id>) and restore the newest by prefix. Record the commit each cache was made at.
  2. Reset after restore, before mutmut run.
    • List the test files changed between the cached commit and HEAD (git diff --name-only <cached>..HEAD -- tests/).
    • Using tests_by_mangled_function_name, find every function covered by a test in those files.
    • Reset those functions' verdicts to "not checked" in the .meta files.
    • A change to shared test helpers or fixtures should count as a change to every test that uses them. If that is hard to know, reset everything when anything under tests/ outside the test modules changes.
  3. Set on_dependency_change = "rerun" in [tool.mutmut], so a change to the requirements or other non-Python inputs resets everything instead of warning.
  4. Give new PRs a warm start. A PR can read caches from its own branch and its base branch only, so the first push of every PR starts cold. A job on pushes to main that runs only mutmut's stats pass (no mutants) and saves mutants/ would give each new PR a stats map to start from.
  5. The priming run and git -C mutants init step should become unnecessary on a restored cache. Keep them for a cold start.

Done when

  • A second push to a PR that changes nothing mutmut sees skips the stats pass and re-tests no mutants.
  • Changing a test re-tests exactly the mutants of the functions that test covers.
  • Changing pyproject.toml or a requirements file re-tests everything.
  • The reset step has unit tests in the repo.

Related: #307 (the cost of mutation runs), #308.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Activity

  1. added
    enhancementNew feature or request
    area: test-suiteThe validator's own tests, mutation testing and fuzzing
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: test-suiteThe validator's own tests, mutation testing and fuzzingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions