Repository navigation
Description
Tests check what a recogniser accepts, rarely what it must reject
From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).
429 surviving mutants in 283 functions widen a recognition condition, and no test fails:
and→or;- a condition forced with
or True/and False; in↔not in,==↔!=.
Each one is a latent false PASS, where the validator would credit a project with a practice it doesn't have. For a validator that is the most damaging kind of bug. The tests show that the right commands and settings are recognised, but rarely show that the near misses are not: another program with the same flags, a fixing run instead of a checking run, a setting that disables a linter, a test path passed to the wrong interpreter.
Examples
ecosystems.go._golangci_runs:linters.disablenaming the linter, andlinters.disable-all: true, are untested.return False→Truesurvives, so a golangci configuration that turns the linter off still counts.ecosystems.ruby.tools._ruby_tests:name == "ruby" and (…test path…)→orsurvives. No test passes a non-Ruby command with atest/path.ecosystems.swift._swiftlint:swiftlint --fixandswiftlint autocorrect(fixing runs) are not shown to be excluded.ecosystems.dart._test_coverage:dart testwithout--coverageis never asserted to measure nothing.facts.formatting._keeps: thegit diff --exit-code/--quietandcommitcases flip without failing.
Functions with the most survivors
| Function | Survivors | Example mutant |
|---|---|---|
ecosystems.go._golangci_runs |
6 | -if any(setting.key == "linters.disable" and _names(setting, linter) for setting in found): || +if any(setti |
facts.formatting._keeps |
5 | -if program(argv) != "git" or len(argv) < 2: || +if program(argv) != "git" and len(argv) < 2: |
ecosystems.perl._runs_test_action |
5 | -action = argv[2] if argv[:2] == ("perl", "Build") and len(argv) > 2 else argv[1] if len(argv) > 1 else "" |\ |
ecosystems.ruby.tools._ruby_tests |
5 | -ruby_runs_tests = name == "ruby" and ("-Itest" in argv or any(RUBY_TEST_FILES.path.search(word) for word in a |
facts.tools.defined_by |
4 | -if fact.reader in readers or PurePosixPath(fact.path).name in names: || +if fact.reader in readers and Pure |
questions.outputs._ignored |
4 | -elif "/" not in name and posixpath.basename(fact.origin.path) == name: || +elif "/" not in name or posixpat |
tools.audits._gitlab_dependency_scanning |
4 | -template = argv[:1] == ("gitlab-template",) and any("Dependency-Scanning" in word for word in argv) || +tem |
ci._heuristic_facts |
4 | -script = "\n".join(line for line in lines if line and line not in ("|", ">", "|-", ">-")) || +script = "\ |
ecosystems.swift._swiftlint |
4 | -fixes = has_option(argv, "--fix", "--autocorrect") or argv[1:2] in (("--fix",), ("autocorrect",)) || +fixes |
ecosystems.swift._body_length |
4 | -names = setting.value if isinstance(setting.value, tuple) else () || +names = setting.value if (isinstance( |
ecosystems.javascript._tasks |
4 | -targets = options.get("-t") or options.get("--target") or options.get("--targets") or "" || +targets = opti |
ecosystems.python.hatch._script |
4 | -scripts = table.get("scripts") if isinstance(table, Mapping) else None || +scripts = table.get("scripts") i |
build_systems.make._assign |
3 | -variables[name] = f"{variables[name]} {value}" if name in variables else value || +variables[name] = f"{var |
build_systems.make.parse_makefile |
3 | -current = [] if line.strip() and not line.lstrip().startswith("#") else current || +current = [] if (line.s |
build_systems.meson.evaluation.Evaluation.run_target |
3 | -if names and command: || +if names or command: |
build_systems.meson.operations.added |
3 | -more = right.items if isinstance(right, Array) else ((right, ALWAYS),) || +more = right.items if (isinstanc |
build_systems.cmake.evaluation.Evaluation._guarded |
3 | -nodes = self.parsed(path) or [] || +nodes = self.parsed(path) and [] |
build_systems.cmake.presets._combined |
3 | -return (either if kind == "anyOf" else both)(*(_condition(found) for found in conditions)) || +return (eith |
build_systems.cmake.commands._registered_test |
3 | -name = words[1] if len(words) > 1 else "" || +name = words[1] if (len(words) > 1) or True else "" |
build_systems.cmake.commands._resolve |
3 | -if invocation.name == "file" and raw[:1] == ["REAL_PATH"] and len(raw) >= 3: || +if invocation.name == "fil |
facts.formatting.flagged |
3 | -applying = None if applies is None else frozenset(applies) || +applying = None if (applies is None) or True |
tools.coverage._named |
3 | -if program(argv) in programs || +if (program(argv) in programs) and False |
tools.audits._trivy_action |
3 | -reference = inputs.get("image-ref", ".") if words == ("image",) else inputs.get("scan-ref", ".") || +refere |
policy.practices.judge_settings |
3 | -nothing_forbidden = practice.forbids and answer.answer is Answer.UNKNOWN and allow_unknown and not answer.fou |
ci.github_expressions._function |
3 | -if name in _STATUS and not arguments: || +if name in _STATUS or not arguments: |
ci.github_actions._own |
3 | -repository = load(REPOSITORY) if load is not None and path else None || +repository = load(REPOSITORY) if ( |
ecosystems.julia._bounded |
3 | -if deps and "name" in _project(text) and set(deps) <= set(compat): || +if deps and "name" in _project(text) |
ecosystems.clojure._cli_formatter |
3 | -return formatted(command, facts, checks=False) if "fix" in words else None || +return formatted(command, fa |
ecosystems.go._golangci_enables |
3 | -or (setting.key == "linters.enable-all" and setting.value is True) || +or (setting.key == "linters.enable-a |
ecosystems.go._excluded_everywhere |
3 | -sources = [source.path for source in facts.of(SourceFile) if source.language in GO_LANGUAGE] || +sources = |
ecosystems.beam._mix_format |
3 | -return None if "--dry-run" in task else formatted(command, facts, checks=False) || +return None if ("--dry- |
ecosystems.dotnet.tools._security_analysis |
3 | -if (found.key in ("AnalysisModeSecurity", "SecuritySeverity") and str(found.value).lower() in _SECURE) || + |
ecosystems.ruby.tools._read_rake_tasks |
3 | -tests = (match[2] or ("spec" if match[1].startswith("RSpec") else "test") for match in _TEST_TASK.finditer(te |
ecosystems.javascript._named_config |
3 | -if program(argv) == name || +if (program(argv) == name) and False |
ecosystems.javascript.analysis._strict_settings |
3 | -bases = _bases(setting.value) if setting.key == "extends" else () || +bases = _bases(setting.value) if (set |
ecosystems.python.hatch._environments |
3 | -document = tool.get("hatch", {}) if isinstance(tool, Mapping) else {} || +document = tool.get("hatch", {}) |
ecosystems.python._pytest_cov |
3 | -addopts = facts.setting("pytest", "addopts", paths=[configuration]) if configuration else None || +addopts |
ecosystems.python.tox._toml_tables |
3 | -document = document.get("tool", {}).get("tox") if isinstance(document, Mapping) else None || +document = do |
build_systems.make._logical_lines |
2 | -yield start, " ".join([*parts, line]) if parts else line || +yield start, " ".join([*parts, line]) if (part |
build_systems.make._rule |
2 | -if percent and long_enough and target.startswith(prefix) and target.endswith(suffix): || +if percent and lo |
(The full list of 283 functions is in the run's data.)
Proposal
-
A near-miss table for every claim and policy reader. Pair each accepted case with the lookalikes it must reject, asserting no claim, a different level or a warning:
- another program with the same flags;
- the same program in fix or apply mode;
- a disabling setting;
- a wrong subcommand;
- the path or flag in the wrong position.
The formatting tests already do this for
prettier --write; generalise that pattern with a small helper (assert_claims(tool, accepted=[…], rejected=[…])). -
Start with the readers that decide PASS/FAIL directly: static analysis, formatting, tests-run and coverage. Then the ecosystem profiles in survivor order.
-
Re-run mutmut on the touched functions (
mutmut run "project_validator.ecosystems.go.x__golangci_runs*") to confirm the survivors are killed.
🤖 Generated with Claude Code
Activity
Test what recognisers must reject, not only what they accept
A .NET command with --no-build does not run the project's analyzers
Test that -w takes back every warning flag and -Werror
A project-wide -w takes back every warning flag, and -Werror with them
Test the makefile reader's edges: recipes, assignments, includes, pre…
Test the Meson evaluator's edges
Test the CMake evaluator's edges
add_test without a name registers no test
leftibot commentedon Nov 18, 2026
Progress in leftibot#80:
Near-miss tests (
tests/test_near_misses.py, plus additions to the Go, Ruby, Perl, Swift, vcs-ignore, Make, Meson and CMake tests) cover:disable,disable-all,default: noneandenable-all;<name>_IS_TOP_LEVEL.Bugs found and fixed (failing test first, then the fix):
-w(and MSVC/w,/W0) passedcompiler-warningsandwarnings-as-errorsalthough it silences every warning.dotnet test/run/pack --no-buildcounted as a build that runs the analyzers.add_test()andadd_test(NAME)registered an empty test.Remaining: the long tail of the 283 functions. The highest-survivor and PASS/FAIL-deciding readers are covered; the rest of the ecosystem profiles are not yet.
🤖 Generated with Claude Code
https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt