Skip to content

Corpus - #309

Merged
lefticus merged 139 commits into
cpp-best-practices:mainfrom
leftibot:corpus
Nov 19, 2026
Merged

Corpus#309
lefticus merged 139 commits into
cpp-best-practices:mainfrom
leftibot:corpus

Conversation

@lefticus

Copy link
Copy Markdown
Member

No description provided.

leftibot and others added 30 commits November 14, 2026 11:53
A symlink whose target does not exist was listed as a file, and the first content probe that opened it raised
FileNotFoundError, so the extraction failed and nothing was judged (argo-cd, jekyll and os-lib keep such symlinks as
test fixtures). Outside a git checkout the walk now skips them; the git listing already kept only real files.

Part of cpp-best-practices#180

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
ToolRun.covers asks owner() for every (run, unit) pair, and owner() walked every unit through PurePosixPath, so a
monorepo of U units and R runs cost O(U²·R) path operations: nushell took 208 s, uutils/coreutils 558 s, haskell/cabal
and rust-lang/cargo more than ten minutes. within() is now a string-prefix test on the facts' normalised paths and
owner() is cached per directory and set of units: nushell 4 s, coreutils 14 s, cabal 20 s, cargo 15 s, with the same
results. cabal and cargo join the corpus.

Part of cpp-best-practices#181

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The splitter stripped `if`/`then` but not `[[`, so `if [[ "$TOTAL" -eq 0 || "$TOTAL" -eq 1 ]]` became a command whose
program is $TOTAL, an unknown when nothing in the job is seen to set it (148 ERROR lines in uutils/coreutils); a test
continued on the next line made every continuation line a command (Humanizer, scikit-learn, Homebrew), and
`repos=("$X")` a command named after its first element. A conditional expression now runs nothing, over as many
lines as it takes, and `name=(…)` is an assignment.

Part of cpp-best-practices#190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The body of `cat > f <<'EOF' … EOF` was split into words like any other line, so prose became commands (llama.cpp's
96 ERROR lines quoting a story), an apostrophe in it opened a quote the lexer could not close (cmake_template's 22
checks in ERROR), and Lisp forms in package-lint's `$(cat - <<EOF …)` were unknown programs. The lines after `<<WORD`
up to the line that is `WORD` are now dropped before the script is split; here-strings (`<<<`) are untouched.

Part of cpp-best-practices#191

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
GitHub's `$/.github/actions/x` and `$/.github/workflows/w.yml` name an action or reusable workflow of the same
repository at the triggering commit; the Actions reader already followed them as local, but the pinning question only
knew `./`, so junit5 (18 of 55), koreader (17 of 17), pytest, poetry, vite and rspec were told to pin them.

Part of cpp-best-practices#209

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The local action's path was `f"{directory}/action.yml"`, and the root's directory is the empty string, so `uses: ./`
looked for `/action.yml` and every GitHub Action repository's self-test was "not read" on every check that reads CI
commands (1656 ERROR lines in docker/build-push-action). The root action is `action.yml`; a composite action's steps
are now read as the job's commands. The GitHub Action fixture's snapshots lose the unknowns and gain those commands.

Part of cpp-best-practices#210

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
`a && b` and `a || b` evaluated to true or false, but GitHub yields the operand that decides them, so ternaries
rendered as booleans: fmt's `CXXFLAGS="${{ matrix.sanitizers && '-fsanitize=address,undefined' || '' }}"` became
`CXXFLAGS=" true"` (its two sanitizer builds unseen), mkdocs' `+py=${{ matrix.py || matrix.python-version }}` became
`+py=true`. The short-circuits a condition relies on when a side is unknown are kept. A runner given by such an
expression (`runs-on: ${{ matrix.os || 'ubuntu-latest' }}`) is now evaluated before its OS is named, so flask's
Windows and macOS jobs are no longer linux.

Part of cpp-best-practices#214

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…bytes

The pattern held three literal backspace characters where `\b` was meant, so `error_prone_core` and
`-Xplugin:ErrorProne` could never match and a Maven build running Error Prone (guava) had "no static analyzer".

Part of cpp-best-practices#251

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
`.github/renovate.json5` (prettier, vite, okhttp, junit5, opentelemetry-proto, clap, starship, ruff), `.gitlab/
renovate.json(5)` and `.renovaterc.json5` were not looked at, so those projects had "no automated dependency
updates". Scala Steward, also named in the issue, is not covered here.

Part of cpp-best-practices#245

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
rbenv's and bash-it's CI run `test/run`, a script that ends in `exec bats …`; the script was followed but `exec` was
taken for the program, so bats never ran as far as tests-run could see. `exec` is a launcher, like `env` and `nice`.

Part of cpp-best-practices#193

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Composer scripts whose entry is `@php vendor/bin/phpstan analyse` (monolog, composer) and steps running
`php "vendor/bin/phpunit"` (PHP_CodeSniffer) recorded a command whose program is `php`, so PHPStan and PHPUnit never
ran as far as the checks could see. `php` is now a launcher of the script after its options, and symfony's
`simple-phpunit` (composer) is one of PHPUnit's program names, for tests-run as for coverage.

Part of cpp-best-practices#200

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
spf13/cobra runs its tests through richgo, which only colours go test's output; tests-run, coverage and the race
detector now recognise it wherever they recognise `go test`.

Part of cpp-best-practices#259

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
mkdocs keeps its tests in `mkdocs/tests/*_tests.py` and discovers them with `unittest discover -p "*tests.py"`; only
`test_*.py` and `*_test.py` were test files, so it had "no tests found". The suffix counts under a `test`/`tests`
directory, where it cannot be mistaken for a module named that way.

Part of cpp-best-practices#255

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…hout a space

nushell runs every lint job as `cargo --config lints.toml clippy …`: only a toolchain (`+nightly`) was skipped before
the subcommand, so clippy "did not run". serde denies `-Dclippy::pedantic`, which the `\bclippy::pedantic` pattern
missed (no word boundary after `-D`). A shared `cargo_argv` now drops the toolchain and cargo's global options for the
clippy and unsafe-code matchers, and the lint-settings pattern allows the same options.

Part of cpp-best-practices#287

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
ripgrep was credited "-D warnings" from `RUSTDOCFLAGS: -D warnings` on its `cargo doc` step, which denies rustdoc's
lints only; nothing passed it to rustc or clippy. A line that sets RUSTDOCFLAGS no longer counts.

Part of cpp-best-practices#287

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Rust had no warnings-as-errors practice, so the check was "not applicable to Rust" although tokio and serde set
`RUSTFLAGS: -Dwarnings` on every job (and ripgrep's failure hint, from its shell scripts, said to run shellcheck).
Denying warnings anywhere the build or CI reads it now passes: RUSTFLAGS, clippy's `-D warnings`, `[lints.rust]
warnings = "deny"`, `#![deny(warnings)]`. The Rust fixtures' snapshots gain the result (well-configured PASS, bare
FAIL); the profile page documents it.

Part of cpp-best-practices#287

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
… || yields its operand

`java-version: ${{ matrix.java || '17' }}` (scalameta/scalafmt) rendered as "true" before && and || returned the
deciding operand; it now names JDK 17 and JDK 21, so the regression test's decorator goes. Also wraps two test lines
the renamed Python test-file pattern made too long.

Part of cpp-best-practices#274

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
rustup runs the channel `rust-toolchain.toml` pins for every plain `cargo`, but a job that named no version read as
"Rust default" (nushell pins 1.97.1). A profile's toolchain patterns can now name the files that pin the default
(`pinned_by`, read once per project into the CI jobs); Rust names `rust-toolchain.toml` and the older
`rust-toolchain`.

Part of cpp-best-practices#287

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…n it names none

mlugg/setup-zig with no `version` installs the manifest's `minimum_zig_version`, but such jobs (zig-clap, zls, libxev,
ghostty) read as "Zig default". The Zig profile now names build.zig.zon as the file that pins it.

Part of cpp-best-practices#288

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
`role()` stopped at the first source root (`src`, `pkg`, `internal`), because the directory names below one are the
code's own (`com/example/`). Test data kept there was therefore product data: deliberately malformed fixtures failed
data-wellformed (golangci-lint's `pkg/…/testdata/`, liquibase's `src/test/resources/`, vite's `src/node/…/__tests__/
fixtures/`) and fixture projects became product units. Below a source root, a test directory (`test`, `tests`,
`testdata`, `__tests__`, `fixtures`) now gives the test role; nothing else does. The JVM and Scala fixtures' test
sources change role in the facts snapshots; no result changes. vite keeps the issue for its `playground/`.

Part of cpp-best-practices#184

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
A `.dir-locals.el` holds an editor's per-directory settings; it made clj-kondo, GNOME/glib, rubocop, frog and others
Emacs Lisp projects, failing six checks each for a language they do not have. Editor settings files are no sources.
The rest of the issue (languages found only in fixtures, tooling or docs assets) is left: it would change the rule
that every non-vendored source is the product's until `--auxiliary` names it.

Part of cpp-best-practices#182

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
dbus-broker's CI passes `${CI_MESON_ARGS}`, a workflow variable holding `-D werror=true`; the variable was resolved,
but only `-Dwerror=true` in one word counted, so warnings-as-errors failed. Meson takes `-D`'s option in the same word
or the next, for `werror` and `warning_level` alike.

Part of cpp-best-practices#196

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…unknown

Recipes were split with every `$(VAR)` kept as written, so `GO ?= go` then `$(GO) test -race -coverprofile …` (gin,
protovalidate), `BIN := ./_boot/dune.exe` then `$(BIN) runtest` (dune) and `make test-with-flags TEST_FLAGS=…`
(golang-migrate) ran nothing the checks knew, and no unknown said so. The makefile's assignments (`=`, `:=`, `?=`,
`+=`, with `override` or `export`), make's built-in variables (`CC`, `CXX`, `AR`, `RM`…) and the command line's
`NAME=value` are now expanded into each recipe, and a recipe whose program is still a variable is an unknown naming
it. Two Make tests expected `$(CC)` literally; they now expect the compiler their makefile sets.

Part of cpp-best-practices#197

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…n rules, CMake's own

Four gaps, each silent until now. A rule line ending in a comment (`lint: deps ## Lint the spec`, the
self-documenting idiom of digitalocean/openapi and docker-stacks) was not a rule. `include other.mk` was not followed,
and a goal no rule defined expanded to nothing; included makefiles the project holds are now read and an undefined
goal is an unknown (cowboy's erlang.mk targets). Pattern rules (`%:` of sphinx-quickstart's Makefile, clap's
`lint-%:`) now match the goals they cover, with `$@` and `$*` filled in. And make in a directory with no makefile but
a CMakeLists.txt (uber/h3's in-source `cmake .`) builds that directory's CMake targets instead of being unknown. A
CMake test that listed `make lint` among the commands the CMake expander ignores no longer lists it.

Part of cpp-best-practices#198

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…ow-unknown

With `--allow-unknown`, an unknown answer of Swift's hardening practice (it forbids `-Ounchecked`) fell through to
FAIL with nothing after the colon ("runtime safety checks are compiled away: ") whenever an unrelated reusable
workflow was unread (swift-argument-parser, swift-nio, vapor, SwiftLint, dart-lang/http, bloc). Judged by what could
be read, nothing forbidden found is a pass.

Part of cpp-best-practices#243

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
pyrightconfig.json (llama.cpp), turbo.json (graphql-eslint), .luarc.json (koreader), asv.conf.json (scikit-learn) and
Nerdbank.GitVersioning's version.json (Humanizer) are documented to take `//` comments and trailing commas, and their
tools read them so; data-wellformed failed each project for one. They join tsconfig.json and the other known names,
with .eslintrc.json, biome.json and deno.json.

Part of cpp-best-practices#232

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…urses

The corpus's first CI run recorded "1:100001: Expecting value" for JSONTestSuite's 100000-deep array where the local
recording had "1:100001: malformed JSON": the JSON reader falls back to its non-recursive checker only when Python's
decoder raises RecursionError, which depends on the interpreter. The test fixes the decoder's behaviour each way with a
mock and asks for one report; it is an expected failure until the reader decides by depth.

Part of cpp-best-practices#290

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
… decoder would do

The reader tried Python's decoder first and fell back to its own non-recursive checker on RecursionError; whether the
decoder recursed that deep depended on the interpreter, so the same file was "Expecting value" on one machine and
"malformed JSON" on another. Nesting is now measured first (one pass, strings skipped) and anything deeper than 500
goes straight to the checker.

Part of cpp-best-practices#290

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…itten

A glob entry ignored an output only when its text was one of the spellings the output names, so `*.pyc*`
(scikit-learn), `site*/` (mkdocs), `blib*`/`cover_db*` (tidyall), `*.gz` (l3build), pandoc's whitelist (`/*` then
`!tracked`), deno std's `sub/target` and magit's `/lisp/*.elc` all "did not ignore" what they ignore. An entry is now
also read as git reads it (`*` within a component, `**` across, a trailing `/` for directories, a slash anchoring it)
against the output's paths placed where the language writes them: its units' directories, and, for a file output,
its sources' directories. `src/build/` still does not ignore a build directory at the root.

Part of cpp-best-practices#236

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…tten

Only the root ignore files were read, and an output was placed at the project's root, so a unit's own .gitignore
(PostgREST's and golangci-lint's docs/.gitignore, koyo's and laravel's npm units, guava's Gradle integration tests)
and a root entry anchored to the unit (`docs/_build/`, rich) did not count. An output is now placed in the units that
write it (those built with its build system, else those of its language; a format's beside its own files, Sphinx's
_build next to its conf.py), and an ignore file of the same name in a directory above it applies there. A nested
file's entry is reported with its path.

Part of cpp-best-practices#237

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
leftibot and others added 25 commits November 18, 2026 09:59
JSON and deep-JSON errors give the line and column of the token that breaks the value, Dockerfile
problems the instruction's line, Meson and Cabal errors name the line, and a failing build command
reports its last five lines of errors, or names the program it could not run. Until now these were
only checked with assertRaises, so the mutation run's 23 survivors could move them freely.

Part of cpp-best-practices#306

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
check_json reports trailing content at the end of the value ("[1]\n x" at 1:4) instead of where
the content that breaks it starts (2:2), as its docstring promises. Found while asserting
parse-error positions.

Part of cpp-best-practices#306

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
check_json skips the whitespace after a complete value before reporting trailing content, so the
line and column point at the content that breaks the file, as its docstring says.

Part of cpp-best-practices#306

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Near-miss tests for the recognisers with the most surviving condition mutants:
- golangci-lint configurations that disable go vet or staticcheck, turn the defaults off, or
  enable them again;
- Ruby test runners against lookalike programs, scripts and rake tasks;
- Module::Build actions only for ./Build and actions that depend on test;
- SwiftLint's fixing runs, and the plugin only for builds;
- git commands that check, keep, show or ignore changes;
- GitLab's dependency scanning template only;
- mix format and cljfmt by mode;
- one VCS's ignore entries not counting for another's file;
- GitHub status functions only without arguments;
- Julia compat bounding every dependency;
- the .NET security analyzer settings.

`dotnet test --no-build` counting as a build is recorded as an expected failure.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
dotnet test, run, pack and publish build the project first, so they run the analyzers it references;
with --no-build they run what an earlier step built and analyze nothing. The well-configured .NET
fixture's `dotnet test --no-build` step no longer counts as an analysis (its `dotnet build` step
still does), so only its facts snapshot changes.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
A project-wide -w silences every warning, so compiler-warnings should not pass on the -W flags it
overrides, and warnings-as-errors has nothing left to treat as an error. Both pass today. Found
while designing the score's disabled-diagnostics model (cpp-best-practices#295).

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
-w (MSVC's /w and /W0) silences every warning, so like a flag's own -Wno- form it takes back the -W
flags and -pedantic that compiler-warnings looks for, and leaves warnings-as-errors nothing to treat
as an error. As with -Wno-, only a project-wide option counts; one a target sets does not.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…fixes, patterns

Comments and blank lines keep a recipe open and other lines close it; += appends, ?= keeps the
first value, and a comment after a value is dropped; includes and prerequisites accumulate across
lines and rules; continued lines are numbered by their first line; the @, - and + prefixes are not
part of a command; and a pattern rule covers only the goals its prefix and suffix match, with the
stem in $* and the goal in $@.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
`in` needs an array; adding to an array appends an item or each item, keeping its condition; a
comparison with a truth takes its condition from either side; indexing needs a dictionary and a
string key; expressions are written back as they read; is_variable holds only for an assigned
variable; a subdirectory reached twice is evaluated once, under the condition it is reached under;
and a run target without a command runs nothing.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Presets' anyOf and allOf conditions with more than one condition; an option without a default is
OFF; a project's own <name>_IS_TOP_LEVEL says whether it is the top project; and add_test without
a name registers nothing, which today it does (an empty test), recorded as an expected failure.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
`add_test()` and `add_test(NAME)` are errors in CMake and register nothing; they were recorded as
a test with an empty name. The command still counts as the project using test registration.

Part of cpp-best-practices#301

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
A tsconfig that extends a local base ("./tsconfig.base.json", "./tsconfig.base", or
"../tsconfig.base.json" from a subdirectory) never finds the base's strict setting: the joined
path keeps its "./" or "..", so it never matches the base file's path. Found while testing the
extends depth limit.

Part of cpp-best-practices#303

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The path of a local base ("./tsconfig.base", "../tsconfig.base.json") is normalised before it is
looked up, so the base's settings are found and strict mode set there counts.

Part of cpp-best-practices#303

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
A shared helper builds a chain at the limit and one link past it for GitLab includes, extends and
!reference tags, GitHub reusable workflows, tsconfig bases and CMake nested calls. Each is followed
at its limit and not past it. Three readers stop without saying why:
- GitLab extends past the limit claim the template is "not defined";
- !reference tags past it drop their commands with no unknown;
- CMake calls past it vanish with no unknown.
These are recorded as expected failures.

Part of cpp-best-practices#303

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…, CMake nesting

- GitLab extends nesting past its limit is unknown "the templates it extends nest more than 10
  levels deep" rather than "a template it extends is not defined".
- A !reference left unresolved past the limit makes its job unknown instead of dropping its
  commands silently.
- A CMake call, include or add_subdirectory nested past MOST_NESTED records
  "calls, includes or subdirectories nest more than 24 deep" for its file, where it used to vanish.

Part of cpp-best-practices#303

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
- GitLab template jobs and their marker commands, and GitLab jobs placed in the file they come from
  with the runner their own or default tags choose, and an unknown at the job that extends an
  undefined template.
- pre-commit.ci's job and command with the hooks it skips.
- tox commands placed in their section, running where tox.ini is, under exact and partial factor
  selection, and an unknown at a tox.ini that does not parse.
- pre-commit hooks placed at their repository and index, and an unknown that hides the settings of
  a configuration that does not parse.
- The settings readers that match code they do not parse mark every setting heuristic at the file
  they read.
- The run helpers keep the reads and heuristic flag they are given.

Part of cpp-best-practices#304

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…ers' defaults

Each spelling reads the same as its siblings and differently from leaving it out: lcov reporting
(genhtml, --list, -l, --summary) and measuring (geninfo, --capture), branch and condition coverage
by option or --rc, llvm-cov's single- and double-dash options, tsc's project and build options,
the JVM Jazzer's class, classpath and jar, NuGet auditing on any restore, mill's format checks,
dart format's output options, and Haskell coverage on any build. Each action reader's defaults and
inputs are pinned. `dart format --output=none` (the option joined to its value) is not recognised
as only showing: recorded as an expected failure.

Part of cpp-best-practices#302

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The output mode is read as the --output or -o option's value, so `--output=none`, `-o=show` and
the space-separated forms all only show the result instead of counting as a run that rewrites.

Part of cpp-best-practices#302

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
tests/snapshots/registry.json records each known tool's name, languages, the levels it can provide
with how to get each (the text of every hint that names it), what it reads, and each check's name,
description and aliases. A change to any of them shows in review like the fixture snapshots, and
is recorded with UPDATE_SNAPSHOTS=1. The mutation run's message survivors were mostly this text.

Part of cpp-best-practices#305

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…ut can not see

- A check built by answered_check is run: each ecosystem's answer is judged with the context's
  allow_unknown, and the results are labelled and folded. The real checks are built at import, so
  mutmut never exercised the run closure.
- The mutation step caps each worker's memory: runaway mutants grew to about 14 GB in the full run.
- The fresh-import architecture test is left out of mutation runs, where it only cost 70 s per
  import-time mutant. These two are recorded as expected failures.

Part of cpp-best-practices#307

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…t of mutation runs

The mutation step sets `ulimit -v 4194304` before mutmut runs, so a runaway mutant fails as a kill
instead of exhausting the runner. pyproject.toml's mutmut configuration deselects the fresh-import
architecture test, which mutmut mapped to every function that runs at import but which fails only
on a mutant that breaks an import: those mutants now report as having no tests, which is what they
have, instead of surviving after 70 seconds each.

Part of cpp-best-practices#307

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
`--score` adds the score designed in cpp-best-practices#295 to the report, read from the reports alone; the statuses and the exit code
do not change. Text and markdown end with the summary line and the top five next steps; with `--format json` the
output becomes `{"results": [...], "score": {...}}`, the results unchanged.

- Level: a gated tech tree. Level N when every core and project cell of levels 1 to N passes and each of their checks
  ran; level 4 when every check ran and every cell passes, auxiliary ones included.
- Completeness: covered cells over applicable ones, per unit, averaged; a warning covers half a cell, auxiliary cells
  weigh half, a skip is not applicable and an error stays in the maximum.
- Depth: each tool in a covered cell by its rank (10/5/2/1 core, 5/2/1 auxiliary, a flat 5 for a project cell
  without tools), counted only once the levels below are complete; the rest is locked_depth.
- Opportunities: every cell not covered, with the completeness and depth it is worth and the level it blocks or
  alone unlocks, the next level's blockers first.

This is version 0, a preview: strength, advisory runs, tool families and verified points need facts the readers do
not record yet. docs/score.md describes it.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…ub Pages

The leaderboard designed in cpp-best-practices#295: leaderboard/projects.toml lists the projects (26 seeds, one per language, with
Compiler Explorer as the TypeScript one), and every night each is fetched at the head of its branch, scored with the
preview score, and published with the documentation site, one board per language.

- leaderboard/: the manifest (name, board, consent, optional branch, auxiliary, exclude, license); the entry rules
  (a public, unarchived repository under an OSI-approved licence, at most 2 GB, on a known board, with consent, listed
  once); evaluation (shallow fetch, every check strictly and with --allow-unknown, recording how many checks only the
  latter saved as validator gaps; a crash or a failed fetch is recorded, unscored); the history on the
  leaderboard-data branch (latest, previous, one line per night) and what changed since last night, including when a
  project changed with no new commit, which is the validator's doing; and the pages: the boards ranked by level, then
  completeness, with depth breaking ties, a page per project (history sparkline, next steps, cells, badge) and a
  shields.io endpoint per project.
- tests/build_site.py --leaderboard DATA renders the leaderboard with the site's navigation and style; before the
  first night it lists the projects.
- Workflows:
  - leaderboard: nightly and on demand; a read-only matrix job per project, then the one job that writes (to the data
    branch alone), then pages.yml, now callable, to publish.
  - leaderboard-check: a pull request that adds or changes entries is checked and previewed, read-only, never
    commenting itself.
  - leaderboard-comment: posts the preview from the artifact, as text, on the pull request whose head was checked
    (workflow_run, with the zizmor finding suppressed and why).
- lizard, jscpd and coverage cover leaderboard/ too; mutmut copies it.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Fix the corpus audit's issues, test first: crashes, shell, Make, ignore files, Actions, Rust and more
Score projects' tooling (--score, preview) and publish a nightly leaderboard
@lefticus
lefticus merged commit 63d182b into cpp-best-practices:main Nov 19, 2026
20 of 21 checks passed
@lefticus
lefticus deleted the corpus branch November 19, 2026 04:06
@github-actions

Copy link
Copy Markdown

Leaderboard preview

serilog/serilog on the .NET board

Level 0 · 52% complete · 45 depth

Next steps that would raise it most:

  • compiler-warnings (Shell in .)
  • tests-run (Shell in .)
  • warnings-as-errors (Shell in .)
  • code-formatting (.NET in .)
  • compiler-warnings (.NET in .)

gleam-lang/stdlib on the BEAM board

Level 0 · 47% complete · 30 depth

Next steps that would raise it most:

  • code-formatting (BEAM in .)
  • code-formatting (JavaScript/TypeScript in .)
  • compiler-warnings (BEAM in .)
  • compiler-warnings (JavaScript/TypeScript in .)
  • tests-run (BEAM in .)

uber/h3 on the C/C++ board

Level 0 · 51% complete · 90 depth

Next steps that would raise it most:

  • tests-run (JavaScript/TypeScript in website)
  • code-formatting (JavaScript/TypeScript in .)
  • code-formatting (Shell in .)
  • compiler-warnings (JavaScript/TypeScript in .)
  • compiler-warnings (Shell in .)

metosin/malli on the Clojure board

Level 0 · 45% complete · 50 depth

Next steps that would raise it most:

  • code-formatting (Clojure in .)
  • code-formatting (JavaScript/TypeScript in .)
  • code-formatting (Shell in .)
  • compiler-warnings (JavaScript/TypeScript in .)
  • compiler-warnings (Shell in .)

edicl/cl-ppcre on the Common Lisp board

Level 0 · 32% complete · 10 depth

Next steps that would raise it most:

  • compiler-warnings (Common Lisp in .)
  • warnings-as-errors (Common Lisp in .)
  • ci-present
  • vcs-ignore
  • tests-run (Common Lisp in .)

felangel/bloc on the Dart board

Level 0 · 38% complete · 205 depth

Next steps that would raise it most:

  • code-formatting (Dart in bricks/bloc/hooks)
  • code-formatting (Dart in bricks/cubit/hooks)
  • code-formatting (Dart in bricks/flutter_bloc_feature/hooks)
  • code-formatting (Dart in bricks/hydrated_bloc/hooks)
  • code-formatting (Dart in bricks/hydrated_cubit/hooks)

magit/magit on the Emacs Lisp board

Level 0 · 41% complete · 25 depth

Next steps that would raise it most:

  • code-formatting (Shell in .)
  • compiler-warnings (Emacs Lisp in .)
  • compiler-warnings (Shell in .)
  • tests-run (Shell in .)
  • warnings-as-errors (Emacs Lisp in .)

fortran-lang/stdlib on the Fortran board

Level 0 · 52% complete · 100 depth

Next steps that would raise it most:

  • code-formatting (C/C++ in .)
  • code-formatting (Fortran in .)
  • code-formatting (Python in .)
  • compiler-warnings (Python in .)
  • tests-run (Python in .)

spf13/cobra on the Go board

Level 1 · 71% complete · 75 depth

Next steps that would raise it most:

  • coverage (Go in .)
  • dependency-audit (Go in .)
  • fuzz-testing (Go in .)
  • sanitizers (Go in .)
  • action-pinning

haskell/cabal on the Haskell board

Level 0 · 63% complete · 2147.5 depth

Next steps that would raise it most:

  • tests-run (Haskell in cabal-install/tests/IntegrationTests2/build/keep-going/p)
  • tests-run (Haskell in cabal-install/tests/IntegrationTests2/build/keep-going/q)
  • tests-run (Haskell in cabal-install/tests/IntegrationTests2/build/local-tarball/q)
  • tests-run (Haskell in cabal-install/tests/IntegrationTests2/regression/3324/p)
  • tests-run (Haskell in cabal-install/tests/IntegrationTests2/regression/3324/q)

compiler-explorer/compiler-explorer on the JavaScript/TypeScript board

Level 0 · 49% complete · 365 depth

Next steps that would raise it most:

  • code-formatting (Python in etc/scripts/shortlinkmigration)
  • tests-run (Shell in etc/scripts/ce-properties-wizard)
  • code-formatting (Python in etc/scripts/gh_tool)
  • tests-run (JavaScript/TypeScript in etc/scripts/docenizers)
  • tests-run (Shell in etc/scripts/docenizers)

jump-dev/JuMP.jl on the Julia board

Level 0 · 76% complete · 55 depth

Next steps that would raise it most:

  • vcs-ignore (Julia in .)
  • type-checking (Julia in .)
  • action-pinning
  • security-scanning (Julia in .)
  • ci-lint

apache/commons-lang on the JVM board

Level 0 · 67% complete · 25 depth

Next steps that would raise it most:

  • code-formatting (JVM in .)
  • compiler-warnings (JVM in .)
  • warnings-as-errors (JVM in .)
  • vcs-ignore (JVM in .)
  • coverage (JVM in .)

lunarmodules/Penlight on the Lua board

Level 0 · 73% complete · 35 depth

Next steps that would raise it most:

  • code-formatting (Lua in .)
  • vcs-ignore (Lua in .)
  • type-checking (Lua in .)
  • security-scanning (Lua in .)
  • function-limits (Lua in .)

ocaml-community/yojson on the OCaml board

Level 0 · 52% complete · 30 depth

Next steps that would raise it most:

  • compiler-warnings (OCaml in .)
  • warnings-as-errors (OCaml in .)
  • code-formatting (OCaml in .)
  • tests-run (OCaml in .)
  • coverage (OCaml in .)

mojolicious/mojo on the Perl board

Level 0 · 49% complete · 40 depth

Next steps that would raise it most:

  • code-formatting (JavaScript/TypeScript in .)
  • compiler-warnings (JavaScript/TypeScript in .)
  • compiler-warnings (Perl in .)
  • tests-run (JavaScript/TypeScript in .)
  • warnings-as-errors (JavaScript/TypeScript in .)

Seldaek/monolog on the PHP board

Level 0 · 69% complete · 45 depth

Next steps that would raise it most:

  • compiler-warnings (PHP in .)
  • warnings-as-errors (Documentation site in .)
  • warnings-as-errors (PHP in .)
  • code-formatting (PHP in .)
  • tests-run (Documentation site in .)

psf/requests on the Python board

Level 0 · 70% complete · 55 depth

Next steps that would raise it most:

  • compiler-warnings (Python in .)
  • warnings-as-errors (Documentation site in .)
  • tests-run (Documentation site in .)
  • coverage (Python in .)
  • static-analysis (Documentation site in .)

tidyverse/ggplot2 on the R board

Level 0 · 67% complete · 42.5 depth

Next steps that would raise it most:

  • compiler-warnings (R in .)
  • license
  • vcs-ignore (R in .)
  • static-analysis (R in .)
  • action-pinning

jackfirth/rebellion on the Racket board

Level 0 · 57% complete · 25 depth

Next steps that would raise it most:

  • code-formatting (Racket in .)
  • tests-run (Racket in .)
  • coverage (Racket in .)
  • action-pinning
  • platform-matrix (Racket in .)

sinatra/sinatra on the Ruby board

Level 0 · 61% complete · 40 depth

Next steps that would raise it most:

  • code-formatting (Ruby in .)
  • compiler-warnings (Ruby in .)
  • coverage (Ruby in .)
  • type-checking (Ruby in .)
  • dependency-audit (Ruby in .)

starship/starship on the Rust board

Level 0 · 61% complete · 80 depth

Next steps that would raise it most:

  • tests-run (Shell in docs)
  • code-formatting (JSON Schema in .)
  • code-formatting (Lua in .)
  • compiler-warnings (Lua in .)
  • compiler-warnings (Shell in .)

com-lihaoyi/os-lib on the Scala board

Level 0 · 47% complete · 50 depth

Next steps that would raise it most:

  • code-formatting (JVM in .)
  • compiler-warnings (JVM in .)
  • compiler-warnings (Scala in .)
  • compiler-warnings (Shell in .)
  • tests-run (JVM in .)

acmesh-official/acme.sh on the Shell board

Level 0 · 64% complete · 40 depth

Next steps that would raise it most:

  • tests-run (Shell in .)
  • coverage (Shell in .)
  • static-analysis (Docker in .)
  • dependency-audit (Docker in .)
  • action-pinning

Alamofire/Alamofire on the Swift board

Level 0 · 47% complete · 50 depth

Next steps that would raise it most:

  • code-formatting (JavaScript/TypeScript in .)
  • code-formatting (Ruby in .)
  • compiler-warnings (JavaScript/TypeScript in .)
  • compiler-warnings (Ruby in .)
  • tests-run (JavaScript/TypeScript in .)

zigtools/zls on the Zig board

Level 0 · 60% complete · 50 depth

Next steps that would raise it most:

  • code-formatting (JSON Schema in .)
  • code-formatting (Nix in .)
  • compiler-warnings (Nix in .)
  • tests-run (Nix in .)
  • warnings-as-errors (Nix in .)

Scored by score version 0 (a preview). The board ranks by level, then completeness.

@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.80021% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.61%. Comparing base (304e949) to head (db57800).
⚠️ Report is 145 commits behind head on main.

Files with missing lines Patch % Lines
project_validator/ecosystems/python/hatch.py 83.33% 7 Missing ⚠️
project_validator/ecosystems/nix.py 86.20% 4 Missing ⚠️
...roject_validator/build_systems/cmake/evaluation.py 85.71% 3 Missing ⚠️
project_validator/ecosystems/julia.py 90.47% 2 Missing ⚠️
project_validator/ecosystems/kubernetes.py 90.90% 2 Missing ⚠️
project_validator/ecosystems/jvm/__init__.py 94.11% 1 Missing ⚠️
project_validator/ecosystems/jvm/analysis.py 93.33% 1 Missing ⚠️
project_validator/ecosystems/rust/__init__.py 97.22% 1 Missing ⚠️
project_validator/ecosystems/rust/tools.py 94.73% 1 Missing ⚠️
project_validator/facts/flags.py 94.44% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #309      +/-   ##
==========================================
- Coverage   99.77%   99.61%   -0.16%     
==========================================
  Files         273      282       +9     
  Lines       14065    15709    +1644     
  Branches     1544     1750     +206     
==========================================
+ Hits        14033    15649    +1616     
- Misses         32       60      +28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

leftibot added a commit to leftibot/project_validator that referenced this pull request Nov 19, 2026
… kill at the budget's end

Since the corpus (cpp-best-practices#309) the stats pass that maps the tests to the functions runs past the 20-minute budget, and
mutmut ignores timeout's INT while in it: the KILL two minutes later ended the job in 137, a failure, with no report.
The budget is 30 minutes (the job 45), and 137 reads as the budget running out, as 124 does: the summary says it
stopped and gives what was checked.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants