Skip to content

Most message and hint text is never asserted (1,501 surviving mutants) #305

Open

Description

@leftibot

Most message and hint text is never asserted

From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).

1,501 surviving mutants in 444 functions change only a string, and no test fails:

  • an XX…XX wrapper;
  • a case change;
  • a hint replaced by None.

Users read these messages, and their wording is part of the product (the hints say what to do). Most tests assert a result's status and subject, but not its message or hint.

Functions with the most survivors

Function Survivors Example mutant
cli._parser 23 -prog="project_validator", description="Validate a project's build, CI and tooling against best practices." |
cli._add_check_options 19 -parser.add_argument("--build", action="store_true", help="configure and build the project to list its tests")
scaffold.main 17 -parser = argparse.ArgumentParser(prog="python -m project_validator.scaffold", description=__doc__.split("\n")
ecosystems.dotnet.tools._read_msbuild_properties 17 -root = xml_root("MSBuild", path, text) || +root = xml_root("XXMSBuildXX", path, text)
build_systems.cmake.ctest.count_built_tests 16 -with tempfile.TemporaryDirectory(prefix="project_validator_") as build: || +with tempfile.TemporaryDirector
tools.coverage._lcov 16 -read = _configuration("lcov", command, facts, _named(_LCOV_PROGRAMS, "--config-file")(command), ".lcovrc") |
ecosystems.common_lisp._run_tests_action 16 -system = inputs.get("asdf-system", "") || +system = inputs.get("XXasdf-systemXX", "")
ecosystems.swift._swiftlint 16 -fixes = has_option(argv, "--fix", "--autocorrect") or argv[1:2] in (("--fix",), ("autocorrect",)) || +fixes
build_systems.cmake.presets.preset_facts 14 -subject="the CMake presets", || +subject="XXthe CMake presetsXX",
tools.scanners._semgrep 14 -how="semgrep ci", || +how="XXsemgrep ciXX",
ecosystems.perl._runs_test_action 14 -action = argv[2] if argv[:2] == ("perl", "Build") and len(argv) > 2 else argv[1] if len(argv) > 1 else "" |\
cli._add_scope_options 13 -metavar="DIR", || +metavar="XXDIRXX",
build_systems.cmake.build.cmake_configures 12 -excluded = ("--build", "--install", "-E", "-P", "--open", "--help", "--version") || +excluded = ("--build",
facts.formatting._keeps 12 -if argv[1] == "commit" or (argv[1] == "diff" and has_option(argv, "--exit-code", "--quiet")): || +if argv[1
tools.scanners._codeql 12 -name="CodeQL", || +name="XXCodeQLXX",
ecosystems.python._coverage_py 12 -return _lines_and(branch, _coverage_setting(command, facts, "run.branch"), reports=False) || +return _lines
text.check_json 11 -kind = match.lastgroup or "" || +kind = match.lastgroup or "XXXX"
ecosystems.scala._sbt_scalafmt 11 -if tasks & {"scalafmtCheckAll", "scalafmtCheck", "scalafmtSbtCheck"}: || +if tasks & {"XXscalafmtCheckAllXX
ecosystems.julia._runtest 11 -coverage = ("--code-coverage=user",) if inputs.get("coverage", "true").lower() != "false" else () || +cover
ecosystems.perl._runs_tests 11 -prefixes = (("prove",), ("make", "test"), ("Build", "test"), ("perl", "Build", "test"), ("dzil", "test")) |\
facts.outputs._glob_regex 10 -if glob.startswith("**", index): || +if glob.startswith("XX**XX", index):
tools.coverage._llvm_cov 10 -if subcommand not in ("show", "report", "export"): || +if subcommand not in ("show", "report", "XXexportXX"
ecosystems.dart._dart_format 10 -if has_option(argv, "--output", "-o") and any(word in ("none", "show", "json") for word in argv[2:]): || +i
ecosystems.tabular_data._frictionless_action 10 -sources = [inputs[name] for name in ("inquiry", "packages", "resources", "tables") if inputs.get(name)] ||
ecosystems.python._pytest_cov 10 -if not starts_with(argv, ("pytest",), ("py.test",)): || +if not starts_with(argv, ("pytest",), ("XXpy.testX
build_systems.cmake.evaluation.Evaluation.project 9 -("PROJECT_NAME", name), || +("XXPROJECT_NAMEXX", name),
ci.pre_commit_ci._facts 9 -skipped = ",".join(map(str, skip)) if isinstance(skip, list) else "" || +skipped = "XX,XX".join(map(str, sk
ecosystems.sql._squawk_action 9 -if inputs.get("fail-on-violations", "false").strip().lower() != "true": || +if inputs.get("fail-on-violatio
ecosystems.kubernetes._kube_linter_action 9 -configuration = ("--config", inputs["config"]) if inputs.get("config") else () || +configuration = ("XX--co
ecosystems.ruby.tools._ruby_tests 9 -ruby_runs_tests = name == "ruby" and ("-Itest" in argv or any(RUBY_TEST_FILES.path.search(word) for word in a

Proposal

  • Snapshot per check. Extend the snapshot fixtures so each check's message and hints, for the bare and well-configured fixtures, are compared verbatim. The snapshot machinery exists (UPDATE_SNAPSHOTS=1); this widens what it covers.
  • Assert the full Result (status, message, details) in policy tests, as several already do, instead of .status alone.
  • Keep heuristics out of scope: no test needs to pin every wording, only one per message template.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Activity

  1. leftibot commented on Nov 18, 2026

    @leftibot
    CollaboratorAuthor

    Progress in leftibot#80:

    tests/snapshots/registry.json (in tests/test_snapshots.py) snapshots:

    • every known tool's name and languages;
    • the levels each tool can provide, with how to get each: the text of every hint that names it;
    • what each tool reads;
    • every check's description and aliases.

    Changes show in review like the fixture snapshots (UPDATE_SNAPSHOTS=1).

    Most of this issue's survivors turned out to be string literals in recognisers (subcommands, aliases), which #301 and #302 cover. They were not message text.

    Remaining: asserting whole Results, not .status alone, in the policy tests.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

  2. added
    area: test-suiteThe validator's own tests, mutation testing and fuzzing
    trackingRoadmap or umbrella issue tracking several pieces of work
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: test-suiteThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions