Repository navigation
Open
Labels
area: test-suiteThe validator's own tests, mutation testing and fuzzingThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of workRoadmap or umbrella issue tracking several pieces of work
Description
Most message and hint text is never asserted
From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).
1,501 surviving mutants in 444 functions change only a string, and no test fails:
- an
XX…XXwrapper; - a case change;
- a hint replaced by
None.
Users read these messages, and their wording is part of the product (the hints say what to do). Most tests assert a result's status and subject, but not its message or hint.
Functions with the most survivors
| Function | Survivors | Example mutant |
|---|---|---|
cli._parser |
23 | -prog="project_validator", description="Validate a project's build, CI and tooling against best practices." | |
cli._add_check_options |
19 | -parser.add_argument("--build", action="store_true", help="configure and build the project to list its tests") |
scaffold.main |
17 | -parser = argparse.ArgumentParser(prog="python -m project_validator.scaffold", description=__doc__.split("\n") |
ecosystems.dotnet.tools._read_msbuild_properties |
17 | -root = xml_root("MSBuild", path, text) || +root = xml_root("XXMSBuildXX", path, text) |
build_systems.cmake.ctest.count_built_tests |
16 | -with tempfile.TemporaryDirectory(prefix="project_validator_") as build: || +with tempfile.TemporaryDirector |
tools.coverage._lcov |
16 | -read = _configuration("lcov", command, facts, _named(_LCOV_PROGRAMS, "--config-file")(command), ".lcovrc") | |
ecosystems.common_lisp._run_tests_action |
16 | -system = inputs.get("asdf-system", "") || +system = inputs.get("XXasdf-systemXX", "") |
ecosystems.swift._swiftlint |
16 | -fixes = has_option(argv, "--fix", "--autocorrect") or argv[1:2] in (("--fix",), ("autocorrect",)) || +fixes |
build_systems.cmake.presets.preset_facts |
14 | -subject="the CMake presets", || +subject="XXthe CMake presetsXX", |
tools.scanners._semgrep |
14 | -how="semgrep ci", || +how="XXsemgrep ciXX", |
ecosystems.perl._runs_test_action |
14 | -action = argv[2] if argv[:2] == ("perl", "Build") and len(argv) > 2 else argv[1] if len(argv) > 1 else "" |\ |
cli._add_scope_options |
13 | -metavar="DIR", || +metavar="XXDIRXX", |
build_systems.cmake.build.cmake_configures |
12 | -excluded = ("--build", "--install", "-E", "-P", "--open", "--help", "--version") || +excluded = ("--build", |
facts.formatting._keeps |
12 | -if argv[1] == "commit" or (argv[1] == "diff" and has_option(argv, "--exit-code", "--quiet")): || +if argv[1 |
tools.scanners._codeql |
12 | -name="CodeQL", || +name="XXCodeQLXX", |
ecosystems.python._coverage_py |
12 | -return _lines_and(branch, _coverage_setting(command, facts, "run.branch"), reports=False) || +return _lines |
text.check_json |
11 | -kind = match.lastgroup or "" || +kind = match.lastgroup or "XXXX" |
ecosystems.scala._sbt_scalafmt |
11 | -if tasks & {"scalafmtCheckAll", "scalafmtCheck", "scalafmtSbtCheck"}: || +if tasks & {"XXscalafmtCheckAllXX |
ecosystems.julia._runtest |
11 | -coverage = ("--code-coverage=user",) if inputs.get("coverage", "true").lower() != "false" else () || +cover |
ecosystems.perl._runs_tests |
11 | -prefixes = (("prove",), ("make", "test"), ("Build", "test"), ("perl", "Build", "test"), ("dzil", "test")) |\ |
facts.outputs._glob_regex |
10 | -if glob.startswith("**", index): || +if glob.startswith("XX**XX", index): |
tools.coverage._llvm_cov |
10 | -if subcommand not in ("show", "report", "export"): || +if subcommand not in ("show", "report", "XXexportXX" |
ecosystems.dart._dart_format |
10 | -if has_option(argv, "--output", "-o") and any(word in ("none", "show", "json") for word in argv[2:]): || +i |
ecosystems.tabular_data._frictionless_action |
10 | -sources = [inputs[name] for name in ("inquiry", "packages", "resources", "tables") if inputs.get(name)] || |
ecosystems.python._pytest_cov |
10 | -if not starts_with(argv, ("pytest",), ("py.test",)): || +if not starts_with(argv, ("pytest",), ("XXpy.testX |
build_systems.cmake.evaluation.Evaluation.project |
9 | -("PROJECT_NAME", name), || +("XXPROJECT_NAMEXX", name), |
ci.pre_commit_ci._facts |
9 | -skipped = ",".join(map(str, skip)) if isinstance(skip, list) else "" || +skipped = "XX,XX".join(map(str, sk |
ecosystems.sql._squawk_action |
9 | -if inputs.get("fail-on-violations", "false").strip().lower() != "true": || +if inputs.get("fail-on-violatio |
ecosystems.kubernetes._kube_linter_action |
9 | -configuration = ("--config", inputs["config"]) if inputs.get("config") else () || +configuration = ("XX--co |
ecosystems.ruby.tools._ruby_tests |
9 | -ruby_runs_tests = name == "ruby" and ("-Itest" in argv or any(RUBY_TEST_FILES.path.search(word) for word in a |
Proposal
- Snapshot per check. Extend the snapshot fixtures so each check's message and hints, for the bare and well-configured fixtures, are compared verbatim. The snapshot machinery exists (
UPDATE_SNAPSHOTS=1); this widens what it covers. - Assert the full
Result(status, message, details) in policy tests, as several already do, instead of.statusalone. - Keep heuristics out of scope: no test needs to pin every wording, only one per message template.
🤖 Generated with Claude Code
Metadata
Metadata
Assignees
Labels
area: test-suiteThe validator's own tests, mutation testing and fuzzingThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of workRoadmap or umbrella issue tracking several pieces of work
Activity
Snapshot what the validator says about its tools and checks
leftibot commentedon Nov 18, 2026
Progress in leftibot#80:
tests/snapshots/registry.json(intests/test_snapshots.py) snapshots:Changes show in review like the fixture snapshots (
UPDATE_SNAPSHOTS=1).Most of this issue's survivors turned out to be string literals in recognisers (subcommands, aliases), which #301 and #302 cover. They were not message text.
Remaining: asserting whole
Results, not.statusalone, in the policy tests.🤖 Generated with Claude Code
https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt