Repository navigation
Open
Labels
area: test-suiteThe validator's own tests, mutation testing and fuzzingThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of workRoadmap or umbrella issue tracking several pieces of work
Description
Assert whole facts, not their presence: provenance, the heuristic flag and fields
From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).
Extractor tests check that a fact exists, or the final verdict, but not the fact's contents. Surviving mutants:
| Kind | Survivors | Functions | Example change |
|---|---|---|---|
| Provenance | 143 | 40 | origin=…, Origin(path, at=…) → None or dropped |
| Heuristic flag | 91 | 30 | heuristic=True flipped or dropped |
| Other fact fields | 66 | 27 | CiJob(os=, builds=, provider=, id=), Script(directory=), Unknown(reason=, subject=), Setting(key=) |
Provenance drives every message's location and --explain. The heuristic flag decides how a claim is worded and weighed. A regression in either reaches users unnoticed.
Provenance
| Function | Survivors | Example mutant |
|---|---|---|
ci.gitlab._template_facts |
14 | -origin = Origin(path=path, at=("include", index)) || +origin = None |
ci.gitlab.pipeline_facts |
10 | -origin = Origin(path=sources.get(name, path), at=(name,)) || +origin = None |
ci.pre_commit_ci._facts |
10 | -origin = Origin(path=path, at=("ci",)) || +origin = Origin(path=path, at=None) |
ecosystems.python.tox._env_scripts |
10 | -yield Unknown(origin=Origin(path=path), subject="the tox environments", reason=f"{path} does not parse") || |
tools.pre_commit._read_hooks |
8 | -yield Unknown(origin=Origin(path=path), subject="the pre-commit hooks", reason=str(error), hides="settings") |
ci.gitlab._included |
6 | -origin = Origin(path=path, at=("include", index)) || +origin = None |
build_systems.make._configured_by_cmake |
5 | -yield Unknown(origin=command.origin, subject="the makefile", reason="none is found where make runs") || +yi |
ecosystems.zig._read_programs |
5 | -yield Setting(origin=Origin(path=path), tool=PROGRAMS, key="executable", value=True) || +yield Setting(orig |
ecosystems.julia._bounded |
5 | -yield Locked(origin=Origin(path=path), manager="Pkg") || +yield Locked(origin=None, manager="Pkg") |
ecosystems.fortran._read_programs |
5 | -yield Setting(origin=Origin(path=program), tool="Fortran programs", key="program", value=True) || +yield Se |
ecosystems.go._read_programs |
5 | -yield Setting(origin=Origin(path=program), tool="Go programs", key="main", value=True) || +yield Setting(or |
ecosystems.r._renv_project |
5 | -yield Application(origin=Origin(path=path), manager="R packages") || +yield Application(origin=None, manage |
tools.pre_commit._configuration |
4 | -return Unknown(origin=command.origin, subject="the hooks", reason="no pre-commit configuration is found") |\ |
ci.gitlab._job_facts |
4 | -origin=Origin(path=path, at=(name,)), || +origin=Origin(path=None, at=(name,)), |
ci.github_actions._computed_directory |
4 | -yield Unknown(origin=origin, subject=subject, reason="it is computed by an expression", hides="commands") |\ |
ecosystems.javascript._in_workspaces |
4 | -found.append(Script(text=script, origin=Origin(path=path, at=("scripts", name)), directory=member)) || +fou |
build_systems.make._recipes |
3 | -origin=origin, || +origin=None, |
build_systems.cmake.ctest._built_tests |
3 | -yield BuiltTests(origin=origin, count=count_built_tests(project.root, options)) || +yield BuiltTests(origin |
ci.github_actions._Combination.run_facts |
3 | -command = Command(origin=at, job=self.identity, argv=argv, line=script, env=env, directory=directory) || +c |
build_systems.make._goals |
2 | -yield Unknown(origin=command.origin, subject=f"the make target {goal}", reason=reason) || +yield Unknown(or |
build_systems.meson.evaluation.Frame.origin |
2 | -return Origin(path=self.path, at=(name, self.line)) || +return Origin(path=self.path, at=None) |
build_systems.meson.Collector.test |
2 | -self.facts.append(Setting(origin=Origin(path=frame.path), tool=BUILD_SYSTEM, key="test", value=True)) || +s |
build_systems.meson._evaluated |
2 | -origin=Origin(path=path), || +origin=None, |
build_systems.cmake.presets.preset_facts |
2 | -origin=Origin(path=path), || +origin=None, |
build_systems.cmake._evaluated |
2 | -origin=Origin(path=path), || +origin=None, |
Heuristic flag
| Function | Survivors | Example mutant |
|---|---|---|
ecosystems.ocaml._read_tests |
6 | -yield from settings(DUNE_TESTS, path, ((kind, name) for name in names or ["?"]), heuristic=True) || +yield |
ecosystems.perl._read_actions |
4 | -return settings(ACTIONS, path, found, heuristic=True) || +return settings(ACTIONS, None, found, heuristic=T |
ecosystems.ocaml._read_flags |
4 | -return settings("dune", path, ((_FLAGS, " ".join(field.split())) for field in fields), heuristic=True) || + |
ecosystems.emacs_lisp._read_eask_commands |
4 | -return settings(EASK_COMMANDS, path, found, heuristic=True) || +return settings(EASK_COMMANDS, None, found, |
ecosystems.shell.tools._read_pester |
4 | -yield from settings(PESTER_NAME, path, (("CodeCoverage.Enabled", True),), heuristic=True) || +yield from se |
ecosystems.shell.tools._pester_coverage |
4 | -return reported(Granularity.STATEMENT, read=(setting,), heuristic=True) || +return reported(Granularity.STA |
ecosystems.ruby.tools._read_rake_tasks |
4 | -yield from settings("Rake", path, found, heuristic=True) || +yield from settings("Rake", None, found, heuri |
ecosystems.ruby.tools._read_simplecov_settings |
4 | -yield from settings("SimpleCov", path, (("start", True),) if started else (), heuristic=True) || +yield fro |
facts.testing.tested |
3 | -return provided(TEST, RUNS, read=read, heuristic=heuristic) || +return provided(TEST, RUNS, read=read, heur |
facts.settings.matched_settings |
3 | -yield from settings(tool, path, ((key, match[1] if value is None else value),), heuristic=True) || +yield f |
facts.coverage.measured |
3 | -return provided(COVERAGE, *reached, read=read, heuristic=heuristic) || +return provided(COVERAGE, *reached, |
facts.coverage.reported |
3 | -return measured(reported=granularities, gated=gated, read=read, heuristic=heuristic) || +return measured(re |
facts.analysis.analysed |
3 | -return provided(ANALYSIS, CHECKS, read=read, heuristic=heuristic) || +return provided(ANALYSIS, CHECKS, rea |
facts.security.scanned |
3 | -return provided(SECURITY, SCANS, read=read, heuristic=heuristic) || +return provided(SECURITY, SCANS, read= |
facts.audit.audited |
3 | -return provided(AUDIT, RUNS, read=read, heuristic=heuristic) || +return provided(AUDIT, RUNS, read=read, he |
facts.migrations.migrations_linted |
3 | -return provided(MIGRATIONS, LINTED, read=read, heuristic=heuristic) || +return provided(MIGRATIONS, LINTED, |
facts.links.links_checked |
3 | -return provided(LINKS, CHECKED, read=read, heuristic=heuristic) || +return provided(LINKS, CHECKED, read=re |
facts.schema.validated |
3 | -return provided(SCHEMA, VALIDATES, read=read, heuristic=heuristic) || +return provided(SCHEMA, VALIDATES, r |
facts.schema.examples_checked |
3 | -return provided(EXAMPLES, CHECKED, read=read, heuristic=heuristic) || +return provided(EXAMPLES, CHECKED, r |
facts.schema.compatibility_checked |
3 | -return provided(COMPATIBILITY, BREAKING, read=read, heuristic=heuristic) || +return provided(COMPATIBILITY, |
Other fields
| Function | Survivors | Example mutant |
|---|---|---|
build_systems.cmake.evaluation._loop_items |
6 | -return [Item(word.text, word.when, word.origin) for word in words] || +return [Item(word.text, word.when, N |
build_systems.cmake.values._variable |
5 | -Value(";".join(item.text for item in group), when, next(iter(group), Item("")).origin) || +Value(";".join(i |
ecosystems.javascript._script_named |
5 | -_ScriptRun(_NPM_SCRIPT_COMMANDS[first], directory, explicit=True) if first in _NPM_SCRIPT_COMMANDS else None |
core.Result.with_details |
4 | -return Result(status=self.status, message=self.message, details=self.details + details, places=self.places) \ |
build_systems.meson.evaluation.Evaluation.value |
4 | -return self.variables.get(name) or single(_BUILT_IN.get(name, Unknown(name))) || +return self.variables.get |
build_systems.cmake.evaluation.Evaluation.separate_arguments |
4 | -found = texts or [Value(item.text, item.when, item.origin) for item in frame.scope.get(name) or ()] || +fou |
build_systems.cmake.evaluation._parse_arguments |
4 | -values[current].append(Item(word.text, word.when, word.origin)) || +values[current].append(Item(word.text, |
build_systems.cmake.values.expanded |
3 | -Value(value.text, both(name.when, value.when), value.origin or name.origin) || +Value(value.text, both(name |
build_systems.meson.evaluation.Evaluation._operation |
2 | -return [(-value if isinstance(value, int) else Unknown(written(operation)), when) for value, when in left] | |
build_systems.cmake.evaluation.Evaluation.words |
2 | -found.append(Value(text, value.when, value.origin, choice, index)) || +found.append(Value(text, value.when, |
build_systems.cmake.evaluation.Evaluation._foreach |
2 | -frame.scope.assign(variable, (Item(item.text, ALWAYS, item.origin),), ALWAYS) || +frame.scope.assign(variab |
build_systems.cmake.evaluation.Evaluation.list_command |
2 | -(Item(word.text, word.when, word.origin or origin) for word in added), || +(Item(word.text, word.when, None |
build_systems.cmake.evaluation.Evaluation.find_program |
2 | -frame.scope.assign(words[0].text, (Item(names[0], ALWAYS, frame.origin(invocation)),), frame.condition) || |
build_systems.cmake.evaluation.Evaluation.parse_arguments |
2 | -arguments = [Value(item.text, item.when, item.origin) for item in frame.scope.get("ARGV") or ()] || +argume |
build_systems.cmake.evaluation.Evaluation.project |
2 | -frame.scope.assign(variable, (Item(value, ALWAYS, origin),), frame.condition) || +frame.scope.assign(variab |
build_systems.cmake.evaluation._set_list_file |
2 | -scope.variables["CMAKE_CURRENT_LIST_FILE"] = (Item(path),) || +scope.variables["CMAKE_CURRENT_LIST_FILE"] = |
build_systems.cmake.evaluation._arguments |
2 | -item = Item(word.text, word.when, word.origin) || +item = Item(word.text, word.when, None) |
build_systems.cmake.values._combined |
2 | -Value(prefix.text + suffix.text, when, _origin(prefix, suffix)) || +Value(prefix.text + suffix.text, when, |
build_systems.cmake.commands._add_definitions |
2 | -flags = (Value(flag, word.when, word.origin) for word in words for flag in word.text.split()) || +flags = ( |
build_systems.cmake.generator_expressions._expression |
2 | -return [Value("1", when), Value("0", negated(when))] || +return [Value("1", ), Value("0", negated(when))] |
Proposal
- Compare whole facts in extractor tests.
assertEqual(facts.of(CiJob), (CiJob(origin=Origin(path=…, at=(…)), id=…, provider=…, name=…, os=…, builds=…),))rather thanassertIn(name, …). A small builder intests/support.pykeeps the expected values short. - Assert
heuristicin every claim test whose reader sets it. - Snapshot
--explainoutput for a few fixtures, so provenance in messages is pinned end to end.
🤖 Generated with Claude Code
Metadata
Metadata
Assignees
Labels
area: test-suiteThe validator's own tests, mutation testing and fuzzingThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of workRoadmap or umbrella issue tracking several pieces of work
Activity
Compare extractors' whole facts: origins, fields and the heuristic flag
leftibot commentedon Nov 18, 2026
Progress in leftibot#80:
tests/test_whole_facts.pycompares whole facts:Remaining: the CMake and Meson evaluators'
Item/Valueorigins, and the smaller extractors.🤖 Generated with Claude Code
https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt