Skip to content

Tests check what a recogniser accepts, rarely what it must reject (429 surviving mutants) #301

Open

Description

@leftibot

Tests check what a recogniser accepts, rarely what it must reject

From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).

429 surviving mutants in 283 functions widen a recognition condition, and no test fails:

  • and → or;
  • a condition forced with or True / and False;
  • in ↔ not in, == ↔ !=.

Each one is a latent false PASS, where the validator would credit a project with a practice it doesn't have. For a validator that is the most damaging kind of bug. The tests show that the right commands and settings are recognised, but rarely show that the near misses are not: another program with the same flags, a fixing run instead of a checking run, a setting that disables a linter, a test path passed to the wrong interpreter.

Examples

  • ecosystems.go._golangci_runs: linters.disable naming the linter, and linters.disable-all: true, are untested. return False → True survives, so a golangci configuration that turns the linter off still counts.
  • ecosystems.ruby.tools._ruby_tests: name == "ruby" and (…test path…) → or survives. No test passes a non-Ruby command with a test/ path.
  • ecosystems.swift._swiftlint: swiftlint --fix and swiftlint autocorrect (fixing runs) are not shown to be excluded.
  • ecosystems.dart._test_coverage: dart test without --coverage is never asserted to measure nothing.
  • facts.formatting._keeps: the git diff --exit-code/--quiet and commit cases flip without failing.

Functions with the most survivors

Function Survivors Example mutant
ecosystems.go._golangci_runs 6 -if any(setting.key == "linters.disable" and _names(setting, linter) for setting in found): || +if any(setti
facts.formatting._keeps 5 -if program(argv) != "git" or len(argv) < 2: || +if program(argv) != "git" and len(argv) < 2:
ecosystems.perl._runs_test_action 5 -action = argv[2] if argv[:2] == ("perl", "Build") and len(argv) > 2 else argv[1] if len(argv) > 1 else "" |\
ecosystems.ruby.tools._ruby_tests 5 -ruby_runs_tests = name == "ruby" and ("-Itest" in argv or any(RUBY_TEST_FILES.path.search(word) for word in a
facts.tools.defined_by 4 -if fact.reader in readers or PurePosixPath(fact.path).name in names: || +if fact.reader in readers and Pure
questions.outputs._ignored 4 -elif "/" not in name and posixpath.basename(fact.origin.path) == name: || +elif "/" not in name or posixpat
tools.audits._gitlab_dependency_scanning 4 -template = argv[:1] == ("gitlab-template",) and any("Dependency-Scanning" in word for word in argv) || +tem
ci._heuristic_facts 4 -script = "\n".join(line for line in lines if line and line not in ("|", ">", "|-", ">-")) || +script = "\
ecosystems.swift._swiftlint 4 -fixes = has_option(argv, "--fix", "--autocorrect") or argv[1:2] in (("--fix",), ("autocorrect",)) || +fixes
ecosystems.swift._body_length 4 -names = setting.value if isinstance(setting.value, tuple) else () || +names = setting.value if (isinstance(
ecosystems.javascript._tasks 4 -targets = options.get("-t") or options.get("--target") or options.get("--targets") or "" || +targets = opti
ecosystems.python.hatch._script 4 -scripts = table.get("scripts") if isinstance(table, Mapping) else None || +scripts = table.get("scripts") i
build_systems.make._assign 3 -variables[name] = f"{variables[name]} {value}" if name in variables else value || +variables[name] = f"{var
build_systems.make.parse_makefile 3 -current = [] if line.strip() and not line.lstrip().startswith("#") else current || +current = [] if (line.s
build_systems.meson.evaluation.Evaluation.run_target 3 -if names and command: || +if names or command:
build_systems.meson.operations.added 3 -more = right.items if isinstance(right, Array) else ((right, ALWAYS),) || +more = right.items if (isinstanc
build_systems.cmake.evaluation.Evaluation._guarded 3 -nodes = self.parsed(path) or [] || +nodes = self.parsed(path) and []
build_systems.cmake.presets._combined 3 -return (either if kind == "anyOf" else both)(*(_condition(found) for found in conditions)) || +return (eith
build_systems.cmake.commands._registered_test 3 -name = words[1] if len(words) > 1 else "" || +name = words[1] if (len(words) > 1) or True else ""
build_systems.cmake.commands._resolve 3 -if invocation.name == "file" and raw[:1] == ["REAL_PATH"] and len(raw) >= 3: || +if invocation.name == "fil
facts.formatting.flagged 3 -applying = None if applies is None else frozenset(applies) || +applying = None if (applies is None) or True
tools.coverage._named 3 -if program(argv) in programs || +if (program(argv) in programs) and False
tools.audits._trivy_action 3 -reference = inputs.get("image-ref", ".") if words == ("image",) else inputs.get("scan-ref", ".") || +refere
policy.practices.judge_settings 3 -nothing_forbidden = practice.forbids and answer.answer is Answer.UNKNOWN and allow_unknown and not answer.fou
ci.github_expressions._function 3 -if name in _STATUS and not arguments: || +if name in _STATUS or not arguments:
ci.github_actions._own 3 -repository = load(REPOSITORY) if load is not None and path else None || +repository = load(REPOSITORY) if (
ecosystems.julia._bounded 3 -if deps and "name" in _project(text) and set(deps) <= set(compat): || +if deps and "name" in _project(text)
ecosystems.clojure._cli_formatter 3 -return formatted(command, facts, checks=False) if "fix" in words else None || +return formatted(command, fa
ecosystems.go._golangci_enables 3 -or (setting.key == "linters.enable-all" and setting.value is True) || +or (setting.key == "linters.enable-a
ecosystems.go._excluded_everywhere 3 -sources = [source.path for source in facts.of(SourceFile) if source.language in GO_LANGUAGE] || +sources =
ecosystems.beam._mix_format 3 -return None if "--dry-run" in task else formatted(command, facts, checks=False) || +return None if ("--dry-
ecosystems.dotnet.tools._security_analysis 3 -if (found.key in ("AnalysisModeSecurity", "SecuritySeverity") and str(found.value).lower() in _SECURE) || +
ecosystems.ruby.tools._read_rake_tasks 3 -tests = (match[2] or ("spec" if match[1].startswith("RSpec") else "test") for match in _TEST_TASK.finditer(te
ecosystems.javascript._named_config 3 -if program(argv) == name || +if (program(argv) == name) and False
ecosystems.javascript.analysis._strict_settings 3 -bases = _bases(setting.value) if setting.key == "extends" else () || +bases = _bases(setting.value) if (set
ecosystems.python.hatch._environments 3 -document = tool.get("hatch", {}) if isinstance(tool, Mapping) else {} || +document = tool.get("hatch", {})
ecosystems.python._pytest_cov 3 -addopts = facts.setting("pytest", "addopts", paths=[configuration]) if configuration else None || +addopts
ecosystems.python.tox._toml_tables 3 -document = document.get("tool", {}).get("tox") if isinstance(document, Mapping) else None || +document = do
build_systems.make._logical_lines 2 -yield start, " ".join([*parts, line]) if parts else line || +yield start, " ".join([*parts, line]) if (part
build_systems.make._rule 2 -if percent and long_enough and target.startswith(prefix) and target.endswith(suffix): || +if percent and lo

(The full list of 283 functions is in the run's data.)

Proposal

  1. A near-miss table for every claim and policy reader. Pair each accepted case with the lookalikes it must reject, asserting no claim, a different level or a warning:

    • another program with the same flags;
    • the same program in fix or apply mode;
    • a disabling setting;
    • a wrong subcommand;
    • the path or flag in the wrong position.

    The formatting tests already do this for prettier --write; generalise that pattern with a small helper (assert_claims(tool, accepted=[…], rejected=[…])).

  2. Start with the readers that decide PASS/FAIL directly: static analysis, formatting, tests-run and coverage. Then the ecosystem profiles in survivor order.

  3. Re-run mutmut on the touched functions (mutmut run "project_validator.ecosystems.go.x__golangci_runs*") to confirm the survivors are killed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Activity

  1. leftibot commented on Nov 18, 2026

    @leftibot
    CollaboratorAuthor

    Progress in leftibot#80:

    Near-miss tests (tests/test_near_misses.py, plus additions to the Go, Ruby, Perl, Swift, vcs-ignore, Make, Meson and CMake tests) cover:

    • golangci disable, disable-all, default: none and enable-all;
    • Ruby runners against lookalikes;
    • Module::Build actions;
    • SwiftLint fixing runs;
    • git diff, commit and status modes;
    • the GitLab dependency-scanning template;
    • mix format and cljfmt modes;
    • one VCS's ignore entries against another's file;
    • GitHub status functions;
    • Julia compat;
    • the .NET security analyzers;
    • Make recipes, assignments, includes, prefixes and pattern rules;
    • Meson operations and subdirectories;
    • CMake presets, options and <name>_IS_TOP_LEVEL.

    Bugs found and fixed (failing test first, then the fix):

    • -w (and MSVC /w, /W0) passed compiler-warnings and warnings-as-errors although it silences every warning.
    • dotnet test/run/pack --no-build counted as a build that runs the analyzers.
    • add_test() and add_test(NAME) registered an empty test.

    Remaining: the long tail of the 283 functions. The highest-survivor and PASS/FAIL-deciding readers are covered; the rest of the ecosystem profiles are not yet.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

  2. added
    area: test-suiteThe validator's own tests, mutation testing and fuzzing
    trackingRoadmap or umbrella issue tracking several pieces of work
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: test-suiteThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions