Skip to content

Pin small interfaces: allow_unknown, the CLI surface and parse-error positions #306

Closed

Description

@leftibot

Pin small interfaces: allow_unknown, the CLI surface and parse-error positions

From the full mutmut run of fixes at 1c2f869 (27,670 of 32,420 mutants checked; see #308). Three small, mechanical gaps.

1. allow_unknown defaults on policy judges (18 survivors, 18 functions)

Flipping allow_unknown: bool = False to True survives on every judge (judge_analysis, judge_coverage, judge_formatting, judge_tests, judge_pinning, judge_settings, …). Production always passes the value explicitly (judging.py), and test_cli pins the unknown-is-error rule, so these are equivalent today. But a new caller that forgets the argument would silently allow unknowns.

Fix: make allow_unknown a required keyword argument (no default). This removes the hazard and the 18 mutants.

2. CLI surface (128 survivors, 18 functions)

Function Survivors Example mutant
cli._parser 26 -prog="project_validator", description="Validate a project's build, CI and tooling against best practices." |
cli._add_check_options 22 -parser.add_argument("--build", action="store_true", help="configure and build the project to list its tests")
scaffold.main 16 -parser = argparse.ArgumentParser(prog="python -m project_validator.scaffold", description=__doc__.split("\n")
cli._add_output_options 10 -choices=("text", "json", "markdown"), || +choices=None,
cli._add_scope_options 8 -metavar="DIR", || +metavar=None,
cli._options 8 -build=arguments.build, || +build=None,
cli._print_listing 8 -print(json.dumps(check_list(checks), indent=2)) || +print(json.dumps(check_list(checks), indent=None))
cli._add_listing_options 6 -help="print what each check recognises for each ecosystem (markdown; as JSON with --format json) and exit", \
cli._run 5 -print(json.dumps(_dump(context, arguments), indent=2)) || +print(json.dumps(_dump(context, arguments), inde
cli._mebibytes 4 -size = 0 || +size = None
cli._format_json 3 -return json.dumps(report_dicts(reports), indent=2) || +return json.dumps(report_dicts(reports), indent=None
cli._format_list 3 -width = max((len(c.name) for c in checks), default=0) || +width = max((len(c.name) for c in checks), defaul
  • --build's store_true can become None, so no test passes --build.
  • The --format/--color choices can be dropped.
  • The defaults (path=., --cmake, --ctest) can be dropped.
  • The JSON indent can change.

Fix:

  • one test of parse_args([]) defaults;
  • one that rejects an invalid --format;
  • one that passes --build (it can mock the build);
  • a golden --help snapshot;
  • a golden --list --format json snapshot.

3. Parse-error positions (23 survivors, 6 functions)

Function Survivors Example mutant
ecosystems.docker.dockerfile.parse 11 -raise ParseError(f"unknown instruction {instruction.keyword}", instruction.line, 1) || +raise ParseError(f"
text.check_json 7 -raise JsonSyntaxError("malformed JSON", line, position - text.rfind("\n", 0, position)) || +raise JsonSynta
build_systems.cmake.ctest._run 2 -raise BuildError(f"could not run {command[0]}: {error}") from error || +raise BuildError(None) from error
build_systems.meson.syntax.parse 1 -raise MesonSyntaxError(parser.token.line, "nested too deeply to read") from error || +raise MesonSyntaxErro
ecosystems.data._check_deep_json 1 -raise ParseError(str(error), error.line, error.column) from error || +raise ParseError(str(None), error.lin
ecosystems.haskell.cabal._lines 1 -raise CabalSyntaxError(f"line {number}: tab in indentation") || +raise CabalSyntaxError(None)

Dockerfile and check_json errors are only checked with assertRaises. Their line and column, which users see, can change without failing.

Fix: assert (line, column) in parse-error tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Activity

  1. leftibot commented on Nov 18, 2026

    @leftibot
    CollaboratorAuthor

    Progress in leftibot#80:

    • allow_unknown is now a required keyword on every judge, including the Judge protocol, with a contract test (tests/test_judges.py). Making it required showed a real bug: the ci-lint check never passed --allow-unknown to its judge, so an unknown answer stayed an error. It is fixed, with a test. action-pinning, secret-scanning and license can never be unknown; they take the argument and discard it, like the rest.
    • CLI surface: tests/test_cli_options.py pins:
      • every default, and the Options each flag builds;
      • repeatable options and the enforced choices;
      • every option's help text and metavar (whitespace-normalised, so it does not depend on the Python version);
      • the JSON listings' indentation and trailing newline;
      • the scaffold's help, default --root, and the label and suffixes in the files it writes.
    • Parse-error positions: tests/test_parse_errors.py asserts line, column and message for JSON, Dockerfile, Meson, Cabal and build-command errors. It found a bug: JSON trailing content was reported at the end of the value, not where the content starts. Fixed.

    Nothing from this issue remains open.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

  2. leftibot commented on Nov 19, 2026

    @leftibot
    CollaboratorAuthor

    Fixed on main (as of e2e2286). Every problem listed here is covered by:

    • f47bb2e Report content after a JSON value where that content starts
    • fc2c806 Test that content after a JSON value is reported where it starts
    • c2145af Assert where parse errors are reported, and what they say
    • 94ecaad Pin the command line's surface: defaults, options, choices, help and listings
    • 86b28bf Require allow_unknown on every policy judge, and pass it from every check
    • bd7239c Test that --allow-unknown relaxes an unknown ci-lint answer
    • 893516a Test that every policy judge requires allow_unknown

    Checked against the commit messages and the regression suite (no expected failure is left for this issue). Reopen if a case is still wrong.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions