Skip to content

Show what the validator found (console, HTML report, TUI, site), score v1, and separate C/C++ and JS/TS - #310

Merged
lefticus merged 10 commits into
cpp-best-practices:mainfrom
leftibot:ux
Nov 19, 2026
Merged

lefticus merged 10 commits into
cpp-best-practices:mainfrom
leftibot:ux

Conversation

@leftibot

Copy link
Copy Markdown
Collaborator

The validator knew a great deal about a project (the languages it found and why, the units, the code in each role, the CI jobs, about 1,500 facts on a mid-size C++ project), but almost none of it reached the reader. This PR makes it visible: what was detected, how it was read, the results, and a score whose 100% means something, in the terminal, in an HTML report, in a TUI and on the site. It also makes C and C++, and JavaScript and TypeScript, separate languages.

The analysis and the decisions are in docs/design/2026-11-19-reporting-and-languages.md.

1. Separate languages (b5740e2)

  • C and C++ are separate ecosystems. What they share is the new project_validator/c_family package.
    • Each is judged by its own flags: CMAKE_C_FLAGS, $<COMPILE_LANGUAGE:C> and c_std for C, and the CXX forms for C++. Flags for every language count for both.
    • C is not asked for the C++ standard library's assertions (_GLIBCXX_ASSERTIONS).
    • A .h header takes the language of most sources below its nearest directory that has any.
    • CMake's default project() languages give way to the sources, so a C++ project is no longer also C.
    • cxx-extensions is renamed compiler-extensions, since it judges C and Fortran too; the old name still selects it.
  • JavaScript and TypeScript are separate ecosystems.
    • TypeScript uses JavaScript's platform and tools, and only TypeScript is asked for type checking.
    • A package.json's main, or JavaScript tool configuration such as eslint.config.js, no longer makes a TypeScript project a JavaScript one.
  • Families. Ecosystems gain a family field, which holds these rules generically.
  • Seeds and corpus. The leaderboard's seeds follow the split: uber/h3 is the C seed, Catch2 the new C++ seed, Compiler Explorer the TypeScript seed and express the new JavaScript seed. The corpus's ecosystem lists were re-detected.

2. Score v1 (b5740e2)

The design proposed on #295 after the preview:

  • 100% means every gate that applies to each part of the project is in place. One check's result for a language in a unit is a slot. A pass fills it, a warning fills half, and an error stays in the total.
  • Pooled across the project. A folder of data files with one gate weighs one slot.
  • Auxiliary code weighs half, and isn't asked for product-only gates (coverage, fuzzing, mutation, sanitizers, hardening, …).
  • Not scored: status-badges.
  • Extra credit goes only to gates where more tools find more: analysers, scanners, auditors, fuzzers, sanitizers and type checkers. A second tool earns +¼ of a slot and a third +⅒, and tools of one family count once. It is held until 100%.
  • Levels stay as milestones: Foundations, Analysis, Hardening, Complete.
  • Next steps give their points and the detail that says what to do.
  • docs/score.md describes it. The leaderboard ranks by it.

3. The views (29c787f)

Every output renders one Analysis, with progress events from extraction and the runner.

  • Plain text keeps its greppable [STATUS] check: message lines. They now come after a "Detected" section (languages and what recognised them, units, code by role with the --auxiliary flag that would apply, CI, facts read). Details are deduplicated.
  • The console report (rich, [pretty]) has:
    • a header with the score bar;
    • what was detected;
    • the results grouped by category, with a line per language and unit;
    • details cut unless --verbose, and skipped checks counted rather than listed;
    • the score's breakdown and next steps.
      It also shows live progress on stderr.
  • The HTML report (--html FILE or --format html) is one self-contained file, light and dark, with:
    • the score ring, what was detected, and the CI jobs;
    • every result with why it matters and its places and tools;
    • a matrix and the next steps;
    • a searchable browser of every fact, linked to the file on GitHub for a checkout;
    • how long each stage took.
  • The TUI (--tui, textual, [tui]) shows the stages as they are read, then a tree of the overview, detection, results, next steps and facts by kind.
  • --explain CHECK says what a check looks for, why it matters and which tools it recognises per language. Every check now has a rationale (project_validator/explanations.py).
  • CI requirements: rich and textual are pinned in CI's test, coverage and typing requirements, so this code is tested and covered.

4. The site and leaderboard (bc2f4d8)

  • One design. The docs site, the leaderboard and the report share one stylesheet, with a top bar and a theme toggle. The support matrix gets sticky headers.
  • The board has a tab per language, score bars and level chips. Each project page has a score ring, a nightly history, what changed, the next steps, every gate and a link to the project's full nightly HTML report.
  • Reports and the data branch. The nightly evaluation writes the reports. The leaderboard-data branch keeps only the latest ones and is now one commit replaced each night (force-pushed), so the reports never grow its history; history/ still keeps every night's numbers.

On five seeds today: spf13/cobra 72%, expressjs/express 68%, catchorg/Catch2 47%, uber/h3 47% and Compiler Explorer 45%.

Decisions to check

  • The HTML report's script is JavaScript in the product with no JavaScript toolchain yet. It is excluded from this project's self-validation visibly in ci.yml (--exclude project_validator/views/assets), rather than hidden in a Python string. The alternative is to add eslint, prettier and a test for it.
  • The force-pushed data branch: if branch protection covers all branches, leaderboard-data needs force pushes allowed.
  • Detected roles stay judged as the product unless --auxiliary names them, as before. The reports now say so and give the flag.

Something the new report surfaced: project_validator/tools/ is detected as tooling code because of its directory name. It's still judged as the product, but the role detection could learn about package directories.

Gates

All green on bc2f4d8:

  • 2,145 tests, with 28 expected failures;
  • ruff, ruff format and whole-repo mypy;
  • lizard (the report's JavaScript included);
  • coverage at 99%;
  • jscpd: 0 clones;
  • strict self-validation;
  • actionlint and zizmor on the workflows.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

leftibot and others added 3 commits November 18, 2026 22:32
…; score v1

C and C++ are separate ecosystems now, sharing their build systems, compilers
and tools (project_validator/c_family). Each is judged by its own flags
(CMAKE_C_FLAGS vs CMAKE_CXX_FLAGS, $<COMPILE_LANGUAGE:...>); C is not asked
for the C++ standard library's assertions. A .h header takes the language of
most sources below its nearest directory that has any, and CMake's default
project() languages give way to the sources. cxx-extensions is renamed
compiler-extensions (it judges C and Fortran too); the old name still selects
it.

JavaScript and TypeScript are separate ecosystems too, sharing the JavaScript
profile's platform and tools. Type checking is asked of TypeScript only. A
package.json's main and JavaScript tool configuration (eslint.config.js) make
no TypeScript project a JavaScript one.

Ecosystems gain a family (C family, ECMAScript): build-declared languages and
tool configuration of a family give way to its sources.

Score v1: 100% means every gate that applies to each part of the project is
in place. Slots are pooled, auxiliary code weighs half and is not asked for
product-only gates, status-badges is reported but not scored, and extra
credit for complementary tools (one per tool family) is held until 100%.
The leaderboard ranks by it.

Also: an Analysis model every view will render, with progress events from
extraction and the runner; an explanation of why each check matters, and
--explain CHECK.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
… TUI

Every view renders one Analysis (project_validator/views):

- Plain text keeps its [STATUS] check: message lines, now after what was
  detected (each language and what recognised it, the units, the code in
  each role and how it is judged, the CI and how much was read), with each
  detail said once, and the totals.
- The console report (rich, [pretty]): a header with the score, what was
  detected, the results grouped by category with a line per language and
  unit, details cut unless --verbose, the skipped checks counted, and the
  score's breakdown by language and next steps. Progress on a terminal's
  stderr while the project is read.
- The HTML report (--html FILE, or --format html): one self-contained file,
  light and dark, with the score ring, what was detected, every result with
  why it matters and its places and tools, a matrix, the next steps, a
  searchable browser of every fact (linked to the file on the web for a
  checkout with a known remote) and how long each stage took.
- The TUI (--tui, textual, [tui]): shows each stage as it is read, then a
  tree of the overview, detection, results, next steps and facts by kind.

The score's hint for a next step is the detail that says what to do. The
old pretty module is replaced by the console view. rich and textual are
pinned in CI's test, coverage and typing requirements.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The documentation site and the leaderboard share the HTML report's design:
one stylesheet (views/assets), light and dark, a top bar with the sections
and a theme toggle. The support matrix has sticky headers and marks the
cells where a check applies.

The leaderboard is one board with a tab per language, score bars and level
chips; each project's page has its score ring, the facts of its night, its
score night by night, what changed, the next steps, every gate, and a link
to its full HTML report. The nightly evaluation writes that report beside
each result; the data branch keeps the latest only, and is one commit
replaced each night so the reports never grow its history (history/ keeps
every night's numbers).

docs/score.md describes score v1; the README shows what each view shows and
documents --html, --format html, --verbose, --explain and --tui. The HTML
report's script is left out of this project's self-validation, visibly in
ci.yml, until it has a JavaScript toolchain of its own.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
@github-actions

Copy link
Copy Markdown

Leaderboard preview

uber/h3 on the C board

47% · Level 0 Getting started · 52 of 85 gates in place

Next steps that would raise it most:

  • code-formatting (JavaScript): +1.2 points
  • code-formatting (Shell): +1.2 points
  • compiler-warnings (JavaScript): +1.2 points
  • compiler-warnings (Shell): +1.2 points
  • compiler-warnings (TypeScript): +1.2 points

catchorg/Catch2 on the C++ board

47% · Level 0 Getting started · 31 of 51 gates in place

Next steps that would raise it most:

  • code-formatting (Python): +2.0 points
  • code-formatting (Shell): +2.0 points
  • compiler-warnings (Python): +2.0 points
  • compiler-warnings (Shell): +2.0 points
  • tests-run (Python): +2.0 points

expressjs/express on the JavaScript board

68% · Level 0 Getting started · 21 of 25 gates in place

Next steps that would raise it most:

  • code-formatting (JavaScript): +4.0 points
  • warnings-as-errors (JavaScript): +4.0 points
  • tests-run (JavaScript): +2.0 points
  • dependency-audit (JavaScript): +2.0 points
  • dependency-management (JavaScript): +2.0 points

compiler-explorer/compiler-explorer on the TypeScript board

45% · Level 0 Getting started · 218 of 321 gates in place

Next steps that would raise it most:

  • code-formatting (Clojure): +0.4 points
  • code-formatting (C++): +0.4 points
  • code-formatting (Julia): +0.4 points
  • code-formatting (Perl): +0.4 points
  • code-formatting (Python): +0.4 points

Scored by score version 1. The board ranks by the score: 100% is every quality gate that applies in place.

@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.93108% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.74%. Comparing base (63d182b) to head (98e44b7).

Files with missing lines Patch % Lines
project_validator/views/tui_app.py 99.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #310      +/-   ##
==========================================
+ Coverage   99.61%   99.74%   +0.12%     
==========================================
  Files         282      294      +12     
  Lines       15709    16823    +1114     
  Branches     1750     1867     +117     
==========================================
+ Hits        15649    16780    +1131     
+ Misses         60       43      -17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

leftibot and others added 7 commits November 19, 2026 05:55
… pins

- The leaderboard data still holds version 0 scores: the site, the ranking and the movement now treat a score of
  another version as not scored (and say so), and the history draws only this version's nights.
- A project in a directory its surrounding checkout ignores (mutmut's mutants/) counts every file as tracked, so
  committed-artifacts judges the copied fixtures as it judges the originals.
- The HTML escaping test carries its markup in a CI job's name: Windows forbids "<" in a file name.
- Self-validation leaves out build/, the copy of the package (the report's script included) `pip install .` leaves.
- The test, coverage and typing requirements pin markdown-it-py, mdit-py-plugins and platformdirs as the others do,
  so pip-audit can install them together.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Tests for a git that can not run, an explanation without a rationale, markdown with the score, a project that
could not be read, code named auxiliary, the console's score colours, branch and unknown suffixes, the TUI's CI
jobs, details, empty next steps, the facts cut-off, its pages and its exit code. short_path and iter_details were
never called.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
… kill at the budget's end

Since the corpus (cpp-best-practices#309) the stats pass that maps the tests to the functions runs past the 20-minute budget, and
mutmut ignores timeout's INT while in it: the KILL two minutes later ended the job in 137, a failure, with no report.
The budget is 30 minutes (the job 45), and 137 reads as the budget running out, as 124 does: the summary says it
stopped and gives what was checked.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The CI test job now runs the unittest suite with `pytest -n auto`, one worker per core.
On Linux with 4 workers (the cores a GitHub Linux or Windows runner has) the suite took
178 s instead of 540 s, with the same 2,137 passed and 28 expected failures.

pytest collects only the unittest classes: not the helper functions named test/tested,
not product classes named Test* (TestFiles), and not the fixture projects' own tests.
`python -m unittest` still runs the same suite locally.

Part of cpp-best-practices#310

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
pip-compile on Linux leaves out pytest's Windows-only colorama dependency, and the
hash-checked install on Windows then refused it as unpinned.

Part of cpp-best-practices#310

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The test runs the mutation workflow's status line under bash. On a Windows runner the
bash found first is System32's WSL launcher, which has no Linux installed and exits 1
for everything. The workflow itself runs only on Linux, and Linux and macOS still run
the test.

Part of cpp-best-practices#310

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
@lefticus
lefticus merged commit e2e2286 into cpp-best-practices:main Nov 19, 2026
24 checks passed
@lefticus
lefticus deleted the ux branch November 19, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants