Skip to content

Limits and boundaries are never tested at the edge (243 surviving mutants) #303

Open

Description

@leftibot

Limits and boundaries are never tested at the edge

From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308).

243 surviving mutants in 125 functions move a boundary without failing:

  • a comparison (< → <=);
  • a constant (+ 1 → + 2, [-5:] → [-6:]);
  • a starting depth (depth = 0 → 1);
  • a recursion or nesting limit.

No test sits exactly at a limit or one past it:

Limit Value
ci/gitlab.py _DEPTH 10 (includes, extends, !reference)
build_systems/cmake/evaluation.py MOST_NESTED 24
ci/github_actions.py _CALL_DEPTH 4
ecosystems/javascript/analysis.py tsconfig extends hard-coded depth < 4
ecosystems/data.py _DEEP –

Functions with the most survivors

Function Survivors Example mutant
ci.gitlab._referenced 13 -def _referenced(value: object, pipeline: Mapping[str, object], depth: int = 0) -> object: || +def _referenc
ecosystems.data._deep 11 -if text.count("[") + text.count("{") <= _DEEP: || +if text.count("[") - text.count("{") <= _DEEP:
ecosystems.javascript.analysis._strict_settings 11 -def _strict_settings(path: str, facts: Facts, depth: int = 0) -> list[Setting]: || +def _strict_settings(pa
ci.gitlab._included 8 -def _included(path: str, document: Mapping[str, object], load: Load | None, depth: int = 0) -> _Pipeline: |
ci.gitlab._extended 7 -def _extended(name: str, pipeline: Mapping[str, object], depth: int = 0) -> Mapping[str, object] | None: |\
checks.large_files._size 6 -if size < 1024 * 1024: || +if size <= 1024 * 1024:
ecosystems.beam._bracketed 6 -depth = 0 || +depth = 1
build_systems.cmake.commands._option 5 -if len(arguments) >= 2: || +if len(arguments) > 2:
facts.outputs._glob_ignores 5 -for depth in range(1, len(components) + 1): || +for depth in range(len(components) + 1):
ecosystems.perl._runs_test_action 5 -action = argv[2] if argv[:2] == ("perl", "Build") and len(argv) > 2 else argv[1] if len(argv) > 1 else "" |\
build_systems.cmake.evaluation.Evaluation.parse_arguments 4 -_parse_arguments(frame.scope, texts[2:6], arguments[_number(texts[1]) :]) || +_parse_arguments(frame.scope,
build_systems.cmake.conditions._compared 4 -return {"EQUAL": difference == 0, "LESS": difference < 0, "GREATER": difference > 0}.get( || +return {"EQUA
facts.conditions._Diagram.variable 4 -return self.nodes[node][0] if node > _TRUE else 1 << 30 || +return self.nodes[node][0] if node >= _TRUE els
ecosystems.rust.cargo_argv 4 -while len(words) > 1 and words[0] in _CARGO_VALUED: || +while len(words) >= 1 and words[0] in _CARGO_VALUED
build_systems.cmake.evaluation.Evaluation.list_command 3 -if len(words) < 2: || +if len(words) <= 2:
build_systems.cmake.evaluation._arguments 3 -if word.argument >= 0 and index and words[index - 1].argument == word.argument: || +if word.argument > 0 an
build_systems.cmake.commands._registered_test 3 -name = words[1] if len(words) > 1 else "" || +name = words[1] if len(words) >= 1 else ""
build_systems.cmake.commands._set 3 -if "CACHE" in texts and texts.index("CACHE") >= 1: || +if "CACHE" in texts and texts.index("CACHE") > 1:
ci.github_actions._runner_os 3 -runner = EXPRESSION.sub(lambda match: rendered(match[0][3:-2], known) or match[0], runner) || +runner = EXP
pretty.print_reports 2 -name_width = max((len(report.check.name) for report in reports), default=0) || +name_width = max((len(repor
build_systems.make._logical_lines 2 -start, parts = 0, [] || +start, parts = 1, []
build_systems.meson.evaluation.Evaluation.run 2 -self._statements += 1 || +self._statements += 2
build_systems.meson.operations._known_element 2 -return all(when == ALWAYS for _, when in container.items) and -len(container.items) <= key < len(container.it
build_systems.cmake.evaluation.Evaluation._evaluate_all 2 -lists = sorted((path for path in self.files if PurePosixPath(path).name == LISTS_FILE), key=_depth) || +lis
build_systems.cmake.evaluation.Evaluation.run 2 -self._commands += 1 || +self._commands += 2
build_systems.cmake.evaluation.Evaluation._call 2 -if arguments in definition.seen or definition.name in frame.stack or len(frame.stack) > MOST_NESTED: || +if
build_systems.cmake.evaluation.Evaluation.string_command 2 -if len(words) >= 2 and words[0].text in ("APPEND", "PREPEND"): || +if len(words) > 2 and words[0].text in (
build_systems.cmake.evaluation.Evaluation.separate_arguments 2 -name, texts = words[0].text, words[2:] if len(words) > 2 and words[1].text.endswith("_COMMAND") else words[1:
build_systems.cmake.evaluation.Evaluation.include 2 -if len(frame.stack) > MOST_NESTED or (path in self.reached and self._guarded(path)): || +if len(frame.stack
build_systems.cmake.evaluation._loop_items 2 -if texts[:1] == ["RANGE"] or texts[:2] == ["IN", "ZIP_LISTS"]: || +if texts[:1] == ["RANGE"] or texts[:3] =
build_systems.cmake.values.expanded 2 -if end < 0: || +if end <= 0:
build_systems.cmake.commands._add_custom_command 2 -ends = [at for at, text in enumerate(texts[1:], 1) if text in _CUSTOM_COMMAND_KEYWORDS] or [len(texts)] ||
build_systems.cmake.commands._add_dependencies 2 -collector.dependencies.setdefault(words[0].text, []).extend(word.text for word in words[1:]) || +collector.
build_systems.cmake.generator_expressions._parts 2 -while (start := text.find(_START, position)) >= 0 and (end := _end(text, start + 2)) >= 0: || +while (start
checks.large_files._lfs_patterns 2 -if len(fields) > 1 and not fields[0].startswith("#") and "filter=lfs" in fields[1:]: || +if len(fields) >=

Proposal

  1. A shared boundary helper builds a chain of N and of N+1 for each limited reader:

    • nested GitLab includes;
    • tsconfig extends;
    • CMake functions calling functions;
    • reusable workflows calling reusable workflows.

    It asserts that N is followed and N+1 produces the documented unknown, naming the limit.

  2. One edge test per off-by-one survivor in the parsers: the stderr[-5:] tail, argv[1:2]/argv[2] positions, and range(1, …) starting points.

  3. Fold into Unify parsing: shared infrastructure, trees with spans, evaluation kept separate #300's single limits policy once it exists, so every limit is tested the same way.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Activity

  1. leftibot commented on Nov 18, 2026

    @leftibot
    CollaboratorAuthor

    Progress in leftibot#80:

    tests/test_limits.py has one helper that builds a chain at the limit and one link past it, for:

    • GitLab includes, extends and !reference (10);
    • GitHub reusable workflows (4);
    • tsconfig local bases (4);
    • CMake nested calls (24).

    It found:

    • a tsconfig extending a local base (./tsconfig.base.json, ../tsconfig.base.json) never found the base, because the joined path kept its ./ or ... Strict mode set in a base did not count. Fixed.
    • GitLab extends past the limit claimed the template was "not defined". It now says the templates nest too deep.
    • !reference past the limit dropped the job's commands silently. It is now an unknown.
    • CMake calls, includes and subdirectories past MOST_NESTED vanished. They now record "calls, includes or subdirectories nest more than 24 deep".

    Remaining: folding these limits into #300's single limits policy.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

  2. added
    area: test-suiteThe validator's own tests, mutation testing and fuzzing
    trackingRoadmap or umbrella issue tracking several pieces of work
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: test-suiteThe validator's own tests, mutation testing and fuzzingtrackingRoadmap or umbrella issue tracking several pieces of work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions