Repository navigation
Score projects' tooling (--score, preview) and publish a nightly leaderboard - #81
Merged
Merged
Conversation
`--score` adds the score designed in cpp-best-practices#295 to the report, read from the reports alone; the statuses and the exit code do not change. Text and markdown end with the summary line and the top five next steps; with `--format json` the output becomes `{"results": [...], "score": {...}}`, the results unchanged. - Level: a gated tech tree. Level N when every core and project cell of levels 1 to N passes and each of their checks ran; level 4 when every check ran and every cell passes, auxiliary ones included. - Completeness: covered cells over applicable ones, per unit, averaged; a warning covers half a cell, auxiliary cells weigh half, a skip is not applicable and an error stays in the maximum. - Depth: each tool in a covered cell by its rank (10/5/2/1 core, 5/2/1 auxiliary, a flat 5 for a project cell without tools), counted only once the levels below are complete; the rest is locked_depth. - Opportunities: every cell not covered, with the completeness and depth it is worth and the level it blocks or alone unlocks, the next level's blockers first. This is version 0, a preview: strength, advisory runs, tool families and verified points need facts the readers do not record yet. docs/score.md describes it. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…ub Pages The leaderboard designed in cpp-best-practices#295: leaderboard/projects.toml lists the projects (26 seeds, one per language, with Compiler Explorer as the TypeScript one), and every night each is fetched at the head of its branch, scored with the preview score, and published with the documentation site, one board per language. - leaderboard/: the manifest (name, board, consent, optional branch, auxiliary, exclude, license); the entry rules (a public, unarchived repository under an OSI-approved licence, at most 2 GB, on a known board, with consent, listed once); evaluation (shallow fetch, every check strictly and with --allow-unknown, recording how many checks only the latter saved as validator gaps; a crash or a failed fetch is recorded, unscored); the history on the leaderboard-data branch (latest, previous, one line per night) and what changed since last night, including when a project changed with no new commit, which is the validator's doing; and the pages: the boards ranked by level, then completeness, with depth breaking ties, a page per project (history sparkline, next steps, cells, badge) and a shields.io endpoint per project. - tests/build_site.py --leaderboard DATA renders the leaderboard with the site's navigation and style; before the first night it lists the projects. - Workflows: - leaderboard: nightly and on demand; a read-only matrix job per project, then the one job that writes (to the data branch alone), then pages.yml, now callable, to publish. - leaderboard-check: a pull request that adds or changes entries is checked and previewed, read-only, never commenting itself. - leaderboard-comment: posts the preview from the artifact, as text, on the pull request whose head was checked (workflow_run, with the zizmor finding suppressed and why). - lizard, jscpd and coverage cover leaderboard/ too; mutmut copies it. Part of cpp-best-practices#295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #80 (
fixes): review and merge that first; this PR's own changes are the last two commits (a1509b2, 4c9b640).The public leaderboard from cpp-best-practices#295. A leaderboard needs a score to rank by, and none existed yet, so this PR also adds the first, preview version of the score. It has two parts.
1.
--score: the tooling score, preview v0 (a1509b2)The score is read from the reports alone. Statuses and exit codes do not change. The design is cpp-best-practices#295's;
docs/score.mddocuments it.locked_depth.--format json --scorethe output is{"results": [...], "score": {...}}, andresultsis unchanged.What v0 leaves out, because the readers don't record the facts yet: strength (disabled diagnostics, suppression density), advisory runs at half points, tool families, and verified points. Until then, a covered cell earns its first rank even when its check names no tool.
On uber/h3 today:
2. The leaderboard (4c9b640)
leaderboard/projects.tomlholds the 26 seeds, one per language, each markedconsent = "seed"until a maintainer confirms. Compiler Explorer is the TypeScript seed, withauxiliary = ["example", "test"](see A language present only in auxiliary files (fixtures, tooling, docs assets, editor dotfiles) becomes a product ecosystem and its FAILs decide the project cpp-best-practices/project_validator#182). Four entries name theirlicensebecause GitHub reports NOASSERTION for them; I checked each licence file: os-lib MIT, cabal BSD-3-Clause, JuMP MPL-2.0, ggplot2 MIT.python -m leaderboard:matrixlists the projects as JSON for the workflow's job matrix.evaluateshallow-fetches the head of a project's branch, runs every check strictly and with--allow-unknown, and scores the latter. How many checks only--allow-unknownsaved is recorded as validator gaps. A crash or a failed fetch is recorded, unscored, with why.recordadds a night to the data directory:latest/,previous/, andhistory/*.jsonlwith one line per night. A rerun of the same night replaces it.checkchecks the entries a PR adds or changes: a public, unarchived repository; an OSI-approved licence; at most 2 GB; a known board; consent; not listed twice. It then writes a Markdown preview with the score and the top next steps.The site:
tests/build_site.py --leaderboard DATArenders the leaderboard pages with the docs site's navigation and style:project_validator | Level 2 · 75%) anddata/latest.json.Before the first night, the board just lists the projects.
Workflows:
leaderboard.ymlruns nightly at 03:17 UTC and on demand:listjob builds the matrix;evaluatejob per project (20-minute budget, no secrets);recordis the only job withcontents: write, and it pushes only to the orphanleaderboard-databranch;publishcallspages.yml, nowworkflow_call-able, which fetches the data branch and builds the site.leaderboard-check.ymlruns onpull_requestforprojects.tomlchanges. It runs the PR's code with a read-only token, writes the preview to the job summary and an artifact, and never comments.leaderboard-comment.ymlruns onworkflow_runwithpull-requests: write. It never checks out or runs PR code. It takes only digits from the artifact's PR number, comments only if that PR's head is the checked commit, and updates its earlier preview comment instead of adding new ones. zizmor flagsworkflow_run; the finding is suppressed inline with the reason.tests/test_leaderboard.pycovers the manifest, the rules, evaluation (on fixtures and local git repositories, never the network), history, movement, ranking, the pages, the preview, the CLI, and each workflow's permissions and checkout credentials.leaderboard/is now in the lizard, jscpd and coverage scopes (100% covered) and in mutmut'salso_copy.A live run on four seeds took 16 s in total:
Before the board goes live
npm run test -- --coverage. The validator still misses the forwarded--coverage. Score a project's tooling, not just pass/warn/fail cpp-best-practices/project_validator#295 lists it as a fix needed before launch.recordjob needscontents: writeon theleaderboard-databranch, so if branch protection covers all branches, that branch needs an exception.Gates
All green on 4c9b640:
leaderboard/;leaderboard/is 100%, andscore.py100% branch);🤖 Generated with Claude Code
https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt