Skip to content

Score projects' tooling (--score, preview) and publish a nightly leaderboard - #81

Merged
lefticus merged 2 commits into
corpusfrom
leaderboard
Nov 19, 2026
Merged

lefticus merged 2 commits into
corpusfrom
leaderboard

Conversation

@leftibot

Copy link
Copy Markdown
Owner

Stacked on #80 (fixes): review and merge that first; this PR's own changes are the last two commits (a1509b2, 4c9b640).

The public leaderboard from cpp-best-practices#295. A leaderboard needs a score to rank by, and none existed yet, so this PR also adds the first, preview version of the score. It has two parts.

1. --score: the tooling score, preview v0 (a1509b2)

The score is read from the reports alone. Statuses and exit codes do not change. The design is cpp-best-practices#295's; docs/score.md documents it.

  • Level is a gated tech tree. A project reaches level N when every core and project cell of the level 1 to N checks passes and each of those checks ran:
    • Level 1, foundations: CI, warnings, warnings as errors, formatting, tests, licence, VCS ignore.
    • Level 2, analysis: static analysis, type checking, coverage, dependency management and audit.
    • Level 3, hardening: sanitizers, fuzzing, mutation testing, security scanning, pinning.
    • Level 4: every check ran and every cell passes, auxiliary ones included.
  • Completeness is covered cells over applicable ones, per unit, then averaged across units. A warning covers half a cell, auxiliary cells weigh half, a skip is not applicable, and an error stays in the maximum. An untooled language lowers completeness.
  • Depth counts each tool in a covered cell by its rank: 10/5/2/1 for core, 5/2/1 for auxiliary, and a flat 5 for a project cell without tools. It counts only once the levels below are complete; the rest is reported as locked_depth.
  • Opportunities are every uncovered cell, with the completeness and depth it is worth and the level it blocks or alone unlocks. The next level's blockers come first.
  • Output: text and markdown end with the summary and the top five next steps. With --format json --score the output is {"results": [...], "score": {...}}, and results is unchanged.

What v0 leaves out, because the readers don't record the facts yet: strength (disabled diagnostics, suppression density), advisory runs at half points, tool families, and verified points. Until then, a covered cell earns its first rank even when its check names no tool.

On uber/h3 today:

Score (preview v0): Level 0 · 49% complete · 90 depth
Next:
  - tests-run (JavaScript/TypeScript in unit website): +2.8% completeness, +10 depth, toward Level 1
  - code-formatting (JavaScript/TypeScript in the root): +0.7% completeness, +10 depth, toward Level 1
  ...

2. The leaderboard (4c9b640)

  • leaderboard/projects.toml holds the 26 seeds, one per language, each marked consent = "seed" until a maintainer confirms. Compiler Explorer is the TypeScript seed, with auxiliary = ["example", "test"] (see A language present only in auxiliary files (fixtures, tooling, docs assets, editor dotfiles) becomes a product ecosystem and its FAILs decide the project cpp-best-practices/project_validator#182). Four entries name their license because GitHub reports NOASSERTION for them; I checked each licence file: os-lib MIT, cabal BSD-3-Clause, JuMP MPL-2.0, ggplot2 MIT.

  • python -m leaderboard:

    • matrix lists the projects as JSON for the workflow's job matrix.
    • evaluate shallow-fetches the head of a project's branch, runs every check strictly and with --allow-unknown, and scores the latter. How many checks only --allow-unknown saved is recorded as validator gaps. A crash or a failed fetch is recorded, unscored, with why.
    • record adds a night to the data directory: latest/, previous/, and history/*.jsonl with one line per night. A rerun of the same night replaces it.
    • check checks the entries a PR adds or changes: a public, unarchived repository; an OSI-approved licence; at most 2 GB; a known board; consent; not listed twice. It then writes a Markdown preview with the score and the top next steps.
  • The site: tests/build_site.py --leaderboard DATA renders the leaderboard pages with the docs site's navigation and style:

    • an all-languages board and one board per language, ranked by level, then completeness, with depth only breaking ties;
    • a page per project: commit, validator and score version, a history sparkline, "since last night" (tools that arrived or left, level, completeness and depth changes, and "same commit: the validator changed" when nothing new was committed), the next steps, every cell, and the badge snippet;
    • a shields.io endpoint per project (project_validator | Level 2 · 75%) and data/latest.json.

    Before the first night, the board just lists the projects.

  • Workflows:

    • leaderboard.yml runs nightly at 03:17 UTC and on demand:
      • a list job builds the matrix;
      • one read-only evaluate job per project (20-minute budget, no secrets);
      • record is the only job with contents: write, and it pushes only to the orphan leaderboard-data branch;
      • publish calls pages.yml, now workflow_call-able, which fetches the data branch and builds the site.
    • leaderboard-check.yml runs on pull_request for projects.toml changes. It runs the PR's code with a read-only token, writes the preview to the job summary and an artifact, and never comments.
    • leaderboard-comment.yml runs on workflow_run with pull-requests: write. It never checks out or runs PR code. It takes only digits from the artifact's PR number, comments only if that PR's head is the checked commit, and updates its earlier preview comment instead of adding new ones. zizmor flags workflow_run; the finding is suppressed inline with the reason.
  • tests/test_leaderboard.py covers the manifest, the rules, evaluation (on fixtures and local git repositories, never the network), history, movement, ranking, the pages, the preview, the CLI, and each workflow's permissions and checkout credentials. leaderboard/ is now in the lizard, jscpd and coverage scopes (100% covered) and in mutmut's also_copy.

A live run on four seeds took 16 s in total:

Project Level Completeness Depth
spf13/cobra 1 71% 75
uber/h3 0 51% 90
compiler-explorer/compiler-explorer 0 49% 365
edicl/cl-ppcre 0 32% 10

Before the board goes live

Gates

All green on 4c9b640:

  • 2,066 tests, with 28 expected failures;
  • ruff, ruff format and whole-repo mypy;
  • lizard, now including leaderboard/;
  • coverage at 99% (leaderboard/ is 100%, and score.py 100% branch);
  • jscpd: 0 clones;
  • strict self-validation;
  • actionlint and zizmor on the workflows.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

leftibot and others added 2 commits November 18, 2026 13:47
`--score` adds the score designed in cpp-best-practices#295 to the report, read from the reports alone; the statuses and the exit code
do not change. Text and markdown end with the summary line and the top five next steps; with `--format json` the
output becomes `{"results": [...], "score": {...}}`, the results unchanged.

- Level: a gated tech tree. Level N when every core and project cell of levels 1 to N passes and each of their checks
  ran; level 4 when every check ran and every cell passes, auxiliary ones included.
- Completeness: covered cells over applicable ones, per unit, averaged; a warning covers half a cell, auxiliary cells
  weigh half, a skip is not applicable and an error stays in the maximum.
- Depth: each tool in a covered cell by its rank (10/5/2/1 core, 5/2/1 auxiliary, a flat 5 for a project cell
  without tools), counted only once the levels below are complete; the rest is locked_depth.
- Opportunities: every cell not covered, with the completeness and depth it is worth and the level it blocks or
  alone unlocks, the next level's blockers first.

This is version 0, a preview: strength, advisory runs, tool families and verified points need facts the readers do
not record yet. docs/score.md describes it.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
…ub Pages

The leaderboard designed in cpp-best-practices#295: leaderboard/projects.toml lists the projects (26 seeds, one per language, with
Compiler Explorer as the TypeScript one), and every night each is fetched at the head of its branch, scored with the
preview score, and published with the documentation site, one board per language.

- leaderboard/: the manifest (name, board, consent, optional branch, auxiliary, exclude, license); the entry rules
  (a public, unarchived repository under an OSI-approved licence, at most 2 GB, on a known board, with consent, listed
  once); evaluation (shallow fetch, every check strictly and with --allow-unknown, recording how many checks only the
  latter saved as validator gaps; a crash or a failed fetch is recorded, unscored); the history on the
  leaderboard-data branch (latest, previous, one line per night) and what changed since last night, including when a
  project changed with no new commit, which is the validator's doing; and the pages: the boards ranked by level, then
  completeness, with depth breaking ties, a page per project (history sparkline, next steps, cells, badge) and a
  shields.io endpoint per project.
- tests/build_site.py --leaderboard DATA renders the leaderboard with the site's navigation and style; before the
  first night it lists the projects.
- Workflows:
  - leaderboard: nightly and on demand; a read-only matrix job per project, then the one job that writes (to the data
    branch alone), then pages.yml, now callable, to publish.
  - leaderboard-check: a pull request that adds or changes entries is checked and previewed, read-only, never
    commenting itself.
  - leaderboard-comment: posts the preview from the artifact, as text, on the pull request whose head was checked
    (workflow_run, with the zizmor finding suppressed and why).
- lizard, jscpd and coverage cover leaderboard/ too; mutmut copies it.

Part of cpp-best-practices#295

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt
Base automatically changed from fixes to corpus November 19, 2026 04:05
@lefticus
lefticus merged commit db57800 into corpus Nov 19, 2026
19 of 22 checks passed
@lefticus
lefticus deleted the leaderboard branch November 19, 2026 04:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants