Repository navigation
Show what the validator found (console, HTML report, TUI, site), score v1, and separate C/C++ and JS/TS - #310
Merged
Merged
Conversation
…; score v1 C and C++ are separate ecosystems now, sharing their build systems, compilers and tools (project_validator/c_family). Each is judged by its own flags (CMAKE_C_FLAGS vs CMAKE_CXX_FLAGS, $<COMPILE_LANGUAGE:...>); C is not asked for the C++ standard library's assertions. A .h header takes the language of most sources below its nearest directory that has any, and CMake's default project() languages give way to the sources. cxx-extensions is renamed compiler-extensions (it judges C and Fortran too); the old name still selects it. JavaScript and TypeScript are separate ecosystems too, sharing the JavaScript profile's platform and tools. Type checking is asked of TypeScript only. A package.json's main and JavaScript tool configuration (eslint.config.js) make no TypeScript project a JavaScript one. Ecosystems gain a family (C family, ECMAScript): build-declared languages and tool configuration of a family give way to its sources. Score v1: 100% means every gate that applies to each part of the project is in place. Slots are pooled, auxiliary code weighs half and is not asked for product-only gates, status-badges is reported but not scored, and extra credit for complementary tools (one per tool family) is held until 100%. The leaderboard ranks by it. Also: an Analysis model every view will render, with progress events from extraction and the runner; an explanation of why each check matters, and --explain CHECK. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
… TUI Every view renders one Analysis (project_validator/views): - Plain text keeps its [STATUS] check: message lines, now after what was detected (each language and what recognised it, the units, the code in each role and how it is judged, the CI and how much was read), with each detail said once, and the totals. - The console report (rich, [pretty]): a header with the score, what was detected, the results grouped by category with a line per language and unit, details cut unless --verbose, the skipped checks counted, and the score's breakdown by language and next steps. Progress on a terminal's stderr while the project is read. - The HTML report (--html FILE, or --format html): one self-contained file, light and dark, with the score ring, what was detected, every result with why it matters and its places and tools, a matrix, the next steps, a searchable browser of every fact (linked to the file on the web for a checkout with a known remote) and how long each stage took. - The TUI (--tui, textual, [tui]): shows each stage as it is read, then a tree of the overview, detection, results, next steps and facts by kind. The score's hint for a next step is the detail that says what to do. The old pretty module is replaced by the console view. rich and textual are pinned in CI's test, coverage and typing requirements. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The documentation site and the leaderboard share the HTML report's design: one stylesheet (views/assets), light and dark, a top bar with the sections and a theme toggle. The support matrix has sticky headers and marks the cells where a check applies. The leaderboard is one board with a tab per language, score bars and level chips; each project's page has its score ring, the facts of its night, its score night by night, what changed, the next steps, every gate, and a link to its full HTML report. The nightly evaluation writes that report beside each result; the data branch keeps the latest only, and is one commit replaced each night so the reports never grow its history (history/ keeps every night's numbers). docs/score.md describes score v1; the README shows what each view shows and documents --html, --format html, --verbose, --explain and --tui. The HTML report's script is left out of this project's self-validation, visibly in ci.yml, until it has a JavaScript toolchain of its own. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Leaderboard previewuber/h3 on the C board47% · Level 0 Getting started · 52 of 85 gates in place Next steps that would raise it most:
catchorg/Catch2 on the C++ board47% · Level 0 Getting started · 31 of 51 gates in place Next steps that would raise it most:
expressjs/express on the JavaScript board68% · Level 0 Getting started · 21 of 25 gates in place Next steps that would raise it most:
compiler-explorer/compiler-explorer on the TypeScript board45% · Level 0 Getting started · 218 of 321 gates in place Next steps that would raise it most:
Scored by score version 1. The board ranks by the score: 100% is every quality gate that applies in place. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #310 +/- ##
==========================================
+ Coverage 99.61% 99.74% +0.12%
==========================================
Files 282 294 +12
Lines 15709 16823 +1114
Branches 1750 1867 +117
==========================================
+ Hits 15649 16780 +1131
+ Misses 60 43 -17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… pins - The leaderboard data still holds version 0 scores: the site, the ranking and the movement now treat a score of another version as not scored (and say so), and the history draws only this version's nights. - A project in a directory its surrounding checkout ignores (mutmut's mutants/) counts every file as tracked, so committed-artifacts judges the copied fixtures as it judges the originals. - The HTML escaping test carries its markup in a CI job's name: Windows forbids "<" in a file name. - Self-validation leaves out build/, the copy of the package (the report's script included) `pip install .` leaves. - The test, coverage and typing requirements pin markdown-it-py, mdit-py-plugins and platformdirs as the others do, so pip-audit can install them together. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Tests for a git that can not run, an explanation without a rationale, markdown with the score, a project that could not be read, code named auxiliary, the console's score colours, branch and unknown suffixes, the TUI's CI jobs, details, empty next steps, the facts cut-off, its pages and its exit code. short_path and iter_details were never called. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
… kill at the budget's end Since the corpus (cpp-best-practices#309) the stats pass that maps the tests to the functions runs past the 20-minute budget, and mutmut ignores timeout's INT while in it: the KILL two minutes later ended the job in 137, a failure, with no report. The budget is 30 minutes (the job 45), and 137 reads as the budget running out, as 124 does: the summary says it stopped and gives what was checked. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The CI test job now runs the unittest suite with `pytest -n auto`, one worker per core. On Linux with 4 workers (the cores a GitHub Linux or Windows runner has) the suite took 178 s instead of 540 s, with the same 2,137 passed and 28 expected failures. pytest collects only the unittest classes: not the helper functions named test/tested, not product classes named Test* (TestFiles), and not the fixture projects' own tests. `python -m unittest` still runs the same suite locally. Part of cpp-best-practices#310 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
pip-compile on Linux leaves out pytest's Windows-only colorama dependency, and the hash-checked install on Windows then refused it as unpinned. Part of cpp-best-practices#310 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
The test runs the mutation workflow's status line under bash. On a Windows runner the bash found first is System32's WSL launcher, which has no Linux installed and exits 1 for everything. The workflow itself runs only on Linux, and Linux and macOS still run the test. Part of cpp-best-practices#310 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The validator knew a great deal about a project (the languages it found and why, the units, the code in each role, the CI jobs, about 1,500 facts on a mid-size C++ project), but almost none of it reached the reader. This PR makes it visible: what was detected, how it was read, the results, and a score whose 100% means something, in the terminal, in an HTML report, in a TUI and on the site. It also makes C and C++, and JavaScript and TypeScript, separate languages.
The analysis and the decisions are in
docs/design/2026-11-19-reporting-and-languages.md.1. Separate languages (b5740e2)
project_validator/c_familypackage.CMAKE_C_FLAGS,$<COMPILE_LANGUAGE:C>andc_stdfor C, and theCXXforms for C++. Flags for every language count for both._GLIBCXX_ASSERTIONS)..hheader takes the language of most sources below its nearest directory that has any.project()languages give way to the sources, so a C++ project is no longer also C.cxx-extensionsis renamedcompiler-extensions, since it judges C and Fortran too; the old name still selects it.main, or JavaScript tool configuration such aseslint.config.js, no longer makes a TypeScript project a JavaScript one.familyfield, which holds these rules generically.2. Score v1 (b5740e2)
The design proposed on #295 after the preview:
status-badges.docs/score.mddescribes it. The leaderboard ranks by it.3. The views (29c787f)
Every output renders one
Analysis, with progress events from extraction and the runner.[STATUS] check: messagelines. They now come after a "Detected" section (languages and what recognised them, units, code by role with the--auxiliaryflag that would apply, CI, facts read). Details are deduplicated.[pretty]) has:--verbose, and skipped checks counted rather than listed;It also shows live progress on stderr.
--html FILEor--format html) is one self-contained file, light and dark, with:--tui, textual,[tui]) shows the stages as they are read, then a tree of the overview, detection, results, next steps and facts by kind.--explain CHECKsays what a check looks for, why it matters and which tools it recognises per language. Every check now has a rationale (project_validator/explanations.py).4. The site and leaderboard (bc2f4d8)
leaderboard-databranch keeps only the latest ones and is now one commit replaced each night (force-pushed), so the reports never grow its history;history/still keeps every night's numbers.On five seeds today: spf13/cobra 72%, expressjs/express 68%, catchorg/Catch2 47%, uber/h3 47% and Compiler Explorer 45%.
Decisions to check
ci.yml(--exclude project_validator/views/assets), rather than hidden in a Python string. The alternative is to add eslint, prettier and a test for it.leaderboard-dataneeds force pushes allowed.--auxiliarynames them, as before. The reports now say so and give the flag.Something the new report surfaced:
project_validator/tools/is detected as tooling code because of its directory name. It's still judged as the product, but the role detection could learn about package directories.Gates
All green on bc2f4d8:
🤖 Generated with Claude Code
https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt