Skip to content

Assert whole facts, not their presence: provenance, the heuristic flag and fields (300 surviving mutants) #304

Open

Description

@leftibot

Assert whole facts, not their presence: provenance, the heuristic flag and fields

From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).

Extractor tests check that a fact exists, or the final verdict, but not the fact's contents. Surviving mutants:

Kind Survivors Functions Example change
Provenance 143 40 origin=…, Origin(path, at=…) → None or dropped
Heuristic flag 91 30 heuristic=True flipped or dropped
Other fact fields 66 27 CiJob(os=, builds=, provider=, id=), Script(directory=), Unknown(reason=, subject=), Setting(key=)

Provenance drives every message's location and --explain. The heuristic flag decides how a claim is worded and weighed. A regression in either reaches users unnoticed.

Provenance

Function Survivors Example mutant
ci.gitlab._template_facts 14 -origin = Origin(path=path, at=("include", index)) || +origin = None
ci.gitlab.pipeline_facts 10 -origin = Origin(path=sources.get(name, path), at=(name,)) || +origin = None
ci.pre_commit_ci._facts 10 -origin = Origin(path=path, at=("ci",)) || +origin = Origin(path=path, at=None)
ecosystems.python.tox._env_scripts 10 -yield Unknown(origin=Origin(path=path), subject="the tox environments", reason=f"{path} does not parse") ||
tools.pre_commit._read_hooks 8 -yield Unknown(origin=Origin(path=path), subject="the pre-commit hooks", reason=str(error), hides="settings")
ci.gitlab._included 6 -origin = Origin(path=path, at=("include", index)) || +origin = None
build_systems.make._configured_by_cmake 5 -yield Unknown(origin=command.origin, subject="the makefile", reason="none is found where make runs") || +yi
ecosystems.zig._read_programs 5 -yield Setting(origin=Origin(path=path), tool=PROGRAMS, key="executable", value=True) || +yield Setting(orig
ecosystems.julia._bounded 5 -yield Locked(origin=Origin(path=path), manager="Pkg") || +yield Locked(origin=None, manager="Pkg")
ecosystems.fortran._read_programs 5 -yield Setting(origin=Origin(path=program), tool="Fortran programs", key="program", value=True) || +yield Se
ecosystems.go._read_programs 5 -yield Setting(origin=Origin(path=program), tool="Go programs", key="main", value=True) || +yield Setting(or
ecosystems.r._renv_project 5 -yield Application(origin=Origin(path=path), manager="R packages") || +yield Application(origin=None, manage
tools.pre_commit._configuration 4 -return Unknown(origin=command.origin, subject="the hooks", reason="no pre-commit configuration is found") |\
ci.gitlab._job_facts 4 -origin=Origin(path=path, at=(name,)), || +origin=Origin(path=None, at=(name,)),
ci.github_actions._computed_directory 4 -yield Unknown(origin=origin, subject=subject, reason="it is computed by an expression", hides="commands") |\
ecosystems.javascript._in_workspaces 4 -found.append(Script(text=script, origin=Origin(path=path, at=("scripts", name)), directory=member)) || +fou
build_systems.make._recipes 3 -origin=origin, || +origin=None,
build_systems.cmake.ctest._built_tests 3 -yield BuiltTests(origin=origin, count=count_built_tests(project.root, options)) || +yield BuiltTests(origin
ci.github_actions._Combination.run_facts 3 -command = Command(origin=at, job=self.identity, argv=argv, line=script, env=env, directory=directory) || +c
build_systems.make._goals 2 -yield Unknown(origin=command.origin, subject=f"the make target {goal}", reason=reason) || +yield Unknown(or
build_systems.meson.evaluation.Frame.origin 2 -return Origin(path=self.path, at=(name, self.line)) || +return Origin(path=self.path, at=None)
build_systems.meson.Collector.test 2 -self.facts.append(Setting(origin=Origin(path=frame.path), tool=BUILD_SYSTEM, key="test", value=True)) || +s
build_systems.meson._evaluated 2 -origin=Origin(path=path), || +origin=None,
build_systems.cmake.presets.preset_facts 2 -origin=Origin(path=path), || +origin=None,
build_systems.cmake._evaluated 2 -origin=Origin(path=path), || +origin=None,

Heuristic flag

Function Survivors Example mutant
ecosystems.ocaml._read_tests 6 -yield from settings(DUNE_TESTS, path, ((kind, name) for name in names or ["?"]), heuristic=True) || +yield
ecosystems.perl._read_actions 4 -return settings(ACTIONS, path, found, heuristic=True) || +return settings(ACTIONS, None, found, heuristic=T
ecosystems.ocaml._read_flags 4 -return settings("dune", path, ((_FLAGS, " ".join(field.split())) for field in fields), heuristic=True) || +
ecosystems.emacs_lisp._read_eask_commands 4 -return settings(EASK_COMMANDS, path, found, heuristic=True) || +return settings(EASK_COMMANDS, None, found,
ecosystems.shell.tools._read_pester 4 -yield from settings(PESTER_NAME, path, (("CodeCoverage.Enabled", True),), heuristic=True) || +yield from se
ecosystems.shell.tools._pester_coverage 4 -return reported(Granularity.STATEMENT, read=(setting,), heuristic=True) || +return reported(Granularity.STA
ecosystems.ruby.tools._read_rake_tasks 4 -yield from settings("Rake", path, found, heuristic=True) || +yield from settings("Rake", None, found, heuri
ecosystems.ruby.tools._read_simplecov_settings 4 -yield from settings("SimpleCov", path, (("start", True),) if started else (), heuristic=True) || +yield fro
facts.testing.tested 3 -return provided(TEST, RUNS, read=read, heuristic=heuristic) || +return provided(TEST, RUNS, read=read, heur
facts.settings.matched_settings 3 -yield from settings(tool, path, ((key, match[1] if value is None else value),), heuristic=True) || +yield f
facts.coverage.measured 3 -return provided(COVERAGE, *reached, read=read, heuristic=heuristic) || +return provided(COVERAGE, *reached,
facts.coverage.reported 3 -return measured(reported=granularities, gated=gated, read=read, heuristic=heuristic) || +return measured(re
facts.analysis.analysed 3 -return provided(ANALYSIS, CHECKS, read=read, heuristic=heuristic) || +return provided(ANALYSIS, CHECKS, rea
facts.security.scanned 3 -return provided(SECURITY, SCANS, read=read, heuristic=heuristic) || +return provided(SECURITY, SCANS, read=
facts.audit.audited 3 -return provided(AUDIT, RUNS, read=read, heuristic=heuristic) || +return provided(AUDIT, RUNS, read=read, he
facts.migrations.migrations_linted 3 -return provided(MIGRATIONS, LINTED, read=read, heuristic=heuristic) || +return provided(MIGRATIONS, LINTED,
facts.links.links_checked 3 -return provided(LINKS, CHECKED, read=read, heuristic=heuristic) || +return provided(LINKS, CHECKED, read=re
facts.schema.validated 3 -return provided(SCHEMA, VALIDATES, read=read, heuristic=heuristic) || +return provided(SCHEMA, VALIDATES, r
facts.schema.examples_checked 3 -return provided(EXAMPLES, CHECKED, read=read, heuristic=heuristic) || +return provided(EXAMPLES, CHECKED, r
facts.schema.compatibility_checked 3 -return provided(COMPATIBILITY, BREAKING, read=read, heuristic=heuristic) || +return provided(COMPATIBILITY,

Other fields

Function Survivors Example mutant
build_systems.cmake.evaluation._loop_items 6 -return [Item(word.text, word.when, word.origin) for word in words] || +return [Item(word.text, word.when, N
build_systems.cmake.values._variable 5 -Value(";".join(item.text for item in group), when, next(iter(group), Item("")).origin) || +Value(";".join(i
ecosystems.javascript._script_named 5 -_ScriptRun(_NPM_SCRIPT_COMMANDS[first], directory, explicit=True) if first in _NPM_SCRIPT_COMMANDS else None
core.Result.with_details 4 -return Result(status=self.status, message=self.message, details=self.details + details, places=self.places) \
build_systems.meson.evaluation.Evaluation.value 4 -return self.variables.get(name) or single(_BUILT_IN.get(name, Unknown(name))) || +return self.variables.get
build_systems.cmake.evaluation.Evaluation.separate_arguments 4 -found = texts or [Value(item.text, item.when, item.origin) for item in frame.scope.get(name) or ()] || +fou
build_systems.cmake.evaluation._parse_arguments 4 -values[current].append(Item(word.text, word.when, word.origin)) || +values[current].append(Item(word.text,
build_systems.cmake.values.expanded 3 -Value(value.text, both(name.when, value.when), value.origin or name.origin) || +Value(value.text, both(name
build_systems.meson.evaluation.Evaluation._operation 2 -return [(-value if isinstance(value, int) else Unknown(written(operation)), when) for value, when in left] |
build_systems.cmake.evaluation.Evaluation.words 2 -found.append(Value(text, value.when, value.origin, choice, index)) || +found.append(Value(text, value.when,
build_systems.cmake.evaluation.Evaluation._foreach 2 -frame.scope.assign(variable, (Item(item.text, ALWAYS, item.origin),), ALWAYS) || +frame.scope.assign(variab
build_systems.cmake.evaluation.Evaluation.list_command 2 -(Item(word.text, word.when, word.origin or origin) for word in added), || +(Item(word.text, word.when, None
build_systems.cmake.evaluation.Evaluation.find_program 2 -frame.scope.assign(words[0].text, (Item(names[0], ALWAYS, frame.origin(invocation)),), frame.condition) ||
build_systems.cmake.evaluation.Evaluation.parse_arguments 2 -arguments = [Value(item.text, item.when, item.origin) for item in frame.scope.get("ARGV") or ()] || +argume
build_systems.cmake.evaluation.Evaluation.project 2 -frame.scope.assign(variable, (Item(value, ALWAYS, origin),), frame.condition) || +frame.scope.assign(variab
build_systems.cmake.evaluation._set_list_file 2 -scope.variables["CMAKE_CURRENT_LIST_FILE"] = (Item(path),) || +scope.variables["CMAKE_CURRENT_LIST_FILE"] =
build_systems.cmake.evaluation._arguments 2 -item = Item(word.text, word.when, word.origin) || +item = Item(word.text, word.when, None)
build_systems.cmake.values._combined 2 -Value(prefix.text + suffix.text, when, _origin(prefix, suffix)) || +Value(prefix.text + suffix.text, when,
build_systems.cmake.commands._add_definitions 2 -flags = (Value(flag, word.when, word.origin) for word in words for flag in word.text.split()) || +flags = (
build_systems.cmake.generator_expressions._expression 2 -return [Value("1", when), Value("0", negated(when))] || +return [Value("1", ), Value("0", negated(when))]

Proposal

  • Compare whole facts in extractor tests. assertEqual(facts.of(CiJob), (CiJob(origin=Origin(path=…, at=(…)), id=…, provider=…, name=…, os=…, builds=…),)) rather than assertIn(name, …). A small builder in tests/support.py keeps the expected values short.
  • Assert heuristic in every claim test whose reader sets it.
  • Snapshot --explain output for a few fixtures, so provenance in messages is pinned end to end.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Activity

  1. leftibot commented on Nov 18, 2026

    @leftibot
    CollaboratorAuthor

    Progress in leftibot#80:

    tests/test_whole_facts.py compares whole facts:

    • GitLab template jobs and their marker commands;
    • GitLab jobs placed in the file they come from, with the runner their own or default tags choose;
    • the unknown at a job that extends an undefined template;
    • pre-commit.ci's job and command with skipped hooks;
    • tox commands by section and directory, under exact and partial factor selection, and the unknown for a tox.ini that does not parse;
    • pre-commit hooks by repository and index, and their parse-error unknown;
    • that the heuristic settings readers (dune tests and flags, Eask, Pester, Rake, SimpleCov, Module::Build) mark every setting heuristic at the file they read;
    • that the run helpers keep their reads and heuristic flag.

    Remaining: the CMake and Meson evaluators' Item/Value origins, and the smaller extractors.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

  2. added
    area: test-suiteThe validator's own tests, mutation testing and fuzzing
    trackingRoadmap or umbrella issue tracking several pieces of work
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: test-suiteThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions