Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
0838262
Skip symlinks to nothing when listing a project's files
leftibot Nov 14, 2026
764502e
Find a directory's unit once, not once per run and unit
leftibot Nov 14, 2026
5dc2c13
Read `[[ … ]]` tests and array assignments as the shell does
leftibot Nov 14, 2026
f1b0e25
Skip here-document bodies when splitting a script
leftibot Nov 14, 2026
5f0a8cd
Count `$/` references as local when judging action pinning
leftibot Nov 14, 2026
5445472
Read the repository's own action for `uses: ./`
leftibot Nov 14, 2026
76e7064
Evaluate && and || to the deciding operand, as GitHub does
leftibot Nov 14, 2026
284d349
Spell the Error Prone pattern's word boundaries as \b, not backspace …
leftibot Nov 14, 2026
77d43cc
Find Renovate's configuration at every location it documents
leftibot Nov 14, 2026
08fbfaa
Read `exec <program>` as running the program
leftibot Nov 14, 2026
a4f77c3
Read `php <script>` as running the script, and simple-phpunit as PHPUnit
leftibot Nov 14, 2026
af323cf
Read `richgo test` as `go test`
leftibot Nov 14, 2026
6df5859
Count `*_tests.py` under a tests directory as Python test files
leftibot Nov 14, 2026
2aaf6b5
Read clippy behind cargo's global options, and -Dclippy::pedantic wit…
leftibot Nov 14, 2026
d32296d
Do not credit RUSTDOCFLAGS' -D warnings as a Rust lint setting
leftibot Nov 14, 2026
d6517ce
Judge warnings-as-errors for Rust
leftibot Nov 14, 2026
ef3281f
Expect scalafmt's JDK from a fallback expression to be read, now that…
leftibot Nov 14, 2026
1ab8d6c
Take a job's Rust toolchain from rust-toolchain.toml when it names none
leftibot Nov 14, 2026
54e49b9
Take a job's Zig version from build.zig.zon's minimum_zig_version whe…
leftibot Nov 14, 2026
3b349f1
Give test data below a source root the test role
leftibot Nov 14, 2026
7e498b5
Do not read Emacs' .dir-locals.el as an Emacs Lisp source
leftibot Nov 14, 2026
7668cb8
Read Meson's `-D werror=true` with a space, as Meson does
leftibot Nov 14, 2026
e09d7fd
Expand Make variables in recipes, and say when a recipe's program is …
leftibot Nov 14, 2026
a598db2
Read Makefiles as make does: comments on rule lines, includes, patter…
leftibot Nov 14, 2026
8ba783c
Pass a practice that forbids settings when none was read, under --all…
leftibot Nov 14, 2026
936ce58
Read the JSON files their tools read with comments as JSON with comments
leftibot Nov 14, 2026
063dfc8
Test that deeply nested JSON is reported the same whatever Python rec…
leftibot Nov 14, 2026
89d1fbb
Read JSON nested deeper than 500 without recursion, whatever Python's…
leftibot Nov 14, 2026
7e76617
Judge ignore entries with gitignore semantics, where the output is wr…
leftibot Nov 14, 2026
6d480b7
Judge each output against the ignore files that apply where it is wri…
leftibot Nov 14, 2026
3fe08c0
Bring four tests in line with the fixes for #184, #198, #236 and #290
leftibot Nov 14, 2026
87a60fa
Split where an output is written out of BuildOutput.ignored_by
leftibot Nov 14, 2026
ab91b06
Count one toolchain version however it is spelled, and no "default" b…
leftibot Nov 14, 2026
892d881
Read every version a setup action installs, and the version file it r…
leftibot Nov 14, 2026
5894c80
Leave ci-lint unknown when a job or command that could run the linter…
leftibot Nov 14, 2026
efc1de9
Read deno task's options before the task name
leftibot Nov 14, 2026
23ff1a5
Read tox's posargs and TOX_ENV, and hatch's environment scripts
leftibot Nov 14, 2026
d6efec7
Read a repository's own reusable workflows when they are called by th…
leftibot Nov 14, 2026
56e4ef3
Run GitHub steps in the default working directory, and say when it is…
leftibot Nov 14, 2026
ae5a6a9
Test that loop and case headers are not commands
leftibot Nov 14, 2026
888db9d
Read `for`, `select` and `case` headers as the keywords they are
leftibot Nov 14, 2026
4041835
Test that arithmetic expansion is a value, not a command
leftibot Nov 14, 2026
6c379c7
Keep arithmetic expansion whole when splitting a script
leftibot Nov 14, 2026
4190199
Test that `dune fmt --preview` checks the formatting
leftibot Nov 14, 2026
1e03d83
Read `dune fmt --preview` as checking the formatting
leftibot Nov 14, 2026
c059106
Count OSS-Fuzz's CIFuzz and a loop over fuzzer binaries as running th…
leftibot Nov 14, 2026
9911df4
Credit coverage uploads as reported, and Codecov targets and Vitest t…
leftibot Nov 14, 2026
c401d53
Stop toolchain patterns matching free text: cache keys, refs, ranges,…
leftibot Nov 14, 2026
655679a
Say a function-length rule is off, not that it runs without a limit
leftibot Nov 14, 2026
20e88fa
Recognise more licence texts and Data Package declarations
leftibot Nov 14, 2026
14cf87b
Continue a Meson expression after a line ending in `and` or `or`
leftibot Nov 14, 2026
8b29b66
Recognise the pre-3.21 top-level guard idioms and resolved-path checks
leftibot Nov 14, 2026
4874cf7
Read COMPILE_FLAGS and keep a branched property value as one argument
leftibot Nov 14, 2026
b1cdd98
Skip configure_file templates when reading the CMake files
leftibot Nov 14, 2026
c73409d
Run an add_custom_command's commands for the targets depending on it
leftibot Nov 14, 2026
66e92bd
Follow `cmake -P script` into the commands the script executes
leftibot Nov 14, 2026
73025f6
Take every value of an option default set in branches
leftibot Nov 14, 2026
474d09d
Read -Wshadow=global, -Weverything and -Wno-* negations
leftibot Nov 14, 2026
ae643c9
Read YAML 1.2 flow scalars starting with `:` and skip @NAME@ templates
leftibot Nov 14, 2026
17c8d70
Show job names as GitHub renders them and strip block-scalar uses
leftibot Nov 14, 2026
b8abbb4
Know pre-commit.ci by its README badge and prek-action as a hook runner
leftibot Nov 14, 2026
baa9785
Say what the tests-run check found and which runner CI uses
leftibot Nov 14, 2026
bd1f34f
Count unread data files by reason and say when only test data is found
leftibot Nov 14, 2026
c565a12
Say each unread subject and each clause once
leftibot Nov 14, 2026
f552885
Print the standard found, a raco fmt hint, and reusable workflows by …
leftibot Nov 14, 2026
84dad27
Credit a tool only where its configuration turns it on
leftibot Nov 14, 2026
fe71f89
Accept SECURITY_CONTACTS and a written-out address as security policies
leftibot Nov 14, 2026
ad3db4f
Read status badges by what they show
leftibot Nov 14, 2026
0fcd682
Do not report suffixes of claimed languages or known data files
leftibot Nov 14, 2026
d2c629d
Do not take a language's source files for license files
leftibot Nov 14, 2026
fd7e9a3
Compare only the project's own manifests; a combined file is its firs…
leftibot Nov 14, 2026
35a7090
Read what dependency files say: submodules, notices, Composer, Pkg, R
leftibot Nov 14, 2026
14ccb83
Judge committed outputs by what they hold
leftibot Nov 14, 2026
dd8d480
Credit a definition inside a nested unit to that unit alone
leftibot Nov 14, 2026
93f0709
Give a job a default compiler only when it compiles
leftibot Nov 14, 2026
a87e3a6
Read Swift settings from code, and judge Swift's sanitizers
leftibot Nov 14, 2026
14fdd50
Find JUnit 5 and JUnit 3 tests, multiplatform test sets, and .NET tes…
leftibot Nov 14, 2026
5345920
Read more JavaScript test runners, xo, nub and strict tsconfig bases
leftibot Nov 14, 2026
888ceb4
Run hadolint for its hadolint-docker hook; a git-diff exit is a forma…
leftibot Nov 14, 2026
1589d5a
Keep nvim's -c from busted's coverage flag; read treefmt's formatters
leftibot Nov 14, 2026
0af029d
Read zsh themes as Shell and run zunit tests
leftibot Nov 14, 2026
cf53dcc
Skip Zig hardening for example-only executables; ignore commented AFL…
leftibot Nov 14, 2026
294aaec
Credit forbidden unsafe code only when it covers every crate
leftibot Nov 14, 2026
9efa558
Read OCaml lexer flags, cram fixtures, ocamlv matrices and lint-fmt
leftibot Nov 14, 2026
d801d49
Run Typst tests with tytanic and format with the typstyle action
leftibot Nov 14, 2026
ff38a78
Split the terraform-min-max case of the #262 regression test into its…
leftibot Nov 14, 2026
c8b3e84
Follow terraform -chdir= into its subcommand; .tpl files are templates
leftibot Nov 14, 2026
3af583c
Read sbt aliases, project/*.scala options, sbt-typelevel and binary +…
leftibot Nov 14, 2026
133a0ae
Read Perl build actions, Test::Perl::Critic runs, test classes and inc/
leftibot Nov 14, 2026
cded8a5
Split the #256 regression test by case
leftibot Nov 14, 2026
1b6f1cd
Count R's tests/*.R scripts as tests
leftibot Nov 14, 2026
751fa83
Split the #257 regression test by case
leftibot Nov 14, 2026
12111e4
Read a Rakefile's warning setting and ruby running a test file
leftibot Nov 14, 2026
bd8a5b5
Read every Dart lint-set form, integration tests and the emulator action
leftibot Nov 14, 2026
9064d5a
Split the antsibull-nox case of the #261 regression test into its own…
leftibot Nov 14, 2026
8f58b75
Read helm-unittest's unittests/ directory and chart-testing's commands
leftibot Nov 14, 2026
6d45936
Detect API descriptions by their versioned top-level key
leftibot Nov 14, 2026
13df2ee
Run an Eask command's ERT, and Leiningen's profiles and aliases
leftibot Nov 14, 2026
ecc8d33
Keep the steps a conditional GITHUB_ENV export guards
leftibot Nov 14, 2026
579ce19
Split the script of a custom shell that runs it with bash
leftibot Nov 14, 2026
f6a7b42
Do not read a binary launcher or a generated configure as shell
leftibot Nov 14, 2026
c16c871
Read a Cargo custom profile's build type from what it inherits
leftibot Nov 14, 2026
0f28c5a
Judge Zig's fuzzing: std.testing.fuzz run by zig build test --fuzz
leftibot Nov 14, 2026
dd5b430
Recognise PostgreSQL extensions as SQL and run their pg_regress tests
leftibot Nov 14, 2026
0fd3d95
Run the POM's defaultGoal for a bare mvn
leftibot Nov 14, 2026
e3f4f01
Share the parenthesis scan of command and arithmetic substitutions
leftibot Nov 14, 2026
9e272ed
Test that mutation testing copies every package the tests import and …
leftibot Nov 15, 2026
1c2f869
Copy the corpus package for mutmut and fail the mutation job when mut…
leftibot Nov 15, 2026
ec21c05
Test that one C/C++ static analyzer is enough
leftibot Nov 16, 2026
1ce7498
Require one C/C++ static analyzer, not both clang-tidy and cppcheck
leftibot Nov 16, 2026
893516a
Test that every policy judge requires allow_unknown
leftibot Nov 18, 2026
bd7239c
Test that --allow-unknown relaxes an unknown ci-lint answer
leftibot Nov 18, 2026
86b28bf
Require allow_unknown on every policy judge, and pass it from every c…
leftibot Nov 18, 2026
94ecaad
Pin the command line's surface: defaults, options, choices, help and …
leftibot Nov 18, 2026
c2145af
Assert where parse errors are reported, and what they say
leftibot Nov 18, 2026
fc2c806
Test that content after a JSON value is reported where it starts
leftibot Nov 18, 2026
f47bb2e
Report content after a JSON value where that content starts
leftibot Nov 18, 2026
11bd853
Test what recognisers must reject, not only what they accept
leftibot Nov 18, 2026
ea110a6
A .NET command with --no-build does not run the project's analyzers
leftibot Nov 18, 2026
e1c849d
Test that -w takes back every warning flag and -Werror
leftibot Nov 18, 2026
28a81db
A project-wide -w takes back every warning flag, and -Werror with them
leftibot Nov 18, 2026
a13e3c4
Test the makefile reader's edges: recipes, assignments, includes, pre…
leftibot Nov 18, 2026
29de5f4
Test the Meson evaluator's edges
leftibot Nov 18, 2026
0737bae
Test the CMake evaluator's edges
leftibot Nov 18, 2026
dce9fd8
add_test without a name registers no test
leftibot Nov 18, 2026
5480dec
Test that tsc is strict through a local base configuration
leftibot Nov 18, 2026
f729e33
Follow a tsconfig's local base by its normalised path
leftibot Nov 18, 2026
0c92f4a
Test every chain-following reader exactly at its limit
leftibot Nov 18, 2026
4f3e39f
Say why a chain stopped at its limit: GitLab templates and references…
leftibot Nov 18, 2026
d3c2b0a
Compare extractors' whole facts: origins, fields and the heuristic flag
leftibot Nov 18, 2026
1085ce0
Test every alternative spelling a reader accepts, and the action read…
leftibot Nov 18, 2026
ba0c30f
dart format only shows with --output joined to its value too
leftibot Nov 18, 2026
aae8acc
Snapshot what the validator says about its tools and checks
leftibot Nov 18, 2026
ae058f1
Test the mutation job's memory cap, and the answered checks that mutm…
leftibot Nov 18, 2026
aa96ded
Cap each mutation worker's memory, and leave the fresh-import test ou…
leftibot Nov 18, 2026
a1509b2
Score a project's tooling: level, completeness and depth (preview)
leftibot Nov 18, 2026
4c9b640
Add the public leaderboard: nightly scores of opt-in projects on GitH…
leftibot Nov 18, 2026
8d43ef0
Merge pull request #80 from leftibot/fixes
lefticus Nov 19, 2026
db57800
Merge pull request #81 from leftibot/leaderboard
lefticus Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ jobs:
run: python -m pip install --require-hashes -r .github/requirements/lizard.txt

- name: Limit function length
run: lizard --length 30 --warnings_only project_validator tests
run: lizard --length 30 --warnings_only project_validator tests leaderboard

ci-lint:
name: ci lint
Expand Down
55 changes: 55 additions & 0 deletions .github/workflows/leaderboard-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: leaderboard-check

# A pull request that adds or changes a leaderboard entry: each entry is checked (a public repository under an
# OSI-approved licence, not too large, on a known board, with its maintainers' consent, not listed twice) and scored,
# and the preview is uploaded for leaderboard-comment to post on the pull request. This workflow runs the pull
# request's code, so it holds a read-only token and no secrets, and it never comments itself.
on:
pull_request:
paths:
- leaderboard/projects.toml

permissions:
contents: read

jobs:
preview:
name: check and preview the entries
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.14"

- name: Install PyYAML
run: python -m pip install --require-hashes -r .github/requirements/test.txt

- name: Check and score the entries the pull request adds or changes
env:
GITHUB_TOKEN: ${{ github.token }}
BASE_REF: ${{ github.base_ref }}
NUMBER: ${{ github.event.pull_request.number }}
VALIDATOR: ${{ github.sha }}
run: |
mkdir -p preview
echo "$NUMBER" > preview/number
git fetch --quiet --depth 1 origin "$BASE_REF"
git show "FETCH_HEAD:leaderboard/projects.toml" > "${RUNNER_TEMP}/base.toml" 2> /dev/null || true
status=0
python -m leaderboard check --base "${RUNNER_TEMP}/base.toml" --out preview/preview.md \
--cache "${RUNNER_TEMP}/checkouts" --validator "$VALIDATOR" || status=$?
cat preview/preview.md >> "$GITHUB_STEP_SUMMARY"
exit "$status"

- name: Upload the preview
if: ${{ !cancelled() }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: leaderboard-preview
path: preview/
retention-days: 7
61 changes: 61 additions & 0 deletions .github/workflows/leaderboard-comment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: leaderboard-comment

# Posts leaderboard-check's preview on its pull request, updating the earlier preview comment when there is one.
# This runs in the base repository's context, with permission to comment, so it never checks out or runs the pull
# request's code: it only reads the preview artifact, treats it as text, and comments on the pull request whose
# head is the commit that was checked.
on:
# The dangerous part of workflow_run is running or trusting the pull request's code; this reads the artifact as
# text, writes nothing to GITHUB_ENV, takes only digits from it, and checks the pull request's head.
workflow_run: # zizmor: ignore[dangerous-triggers] the preview is only read as text and commented
workflows: [leaderboard-check]
types: [completed]

permissions: {}

jobs:
comment:
name: comment the preview
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion != 'cancelled'
runs-on: ubuntu-latest
permissions:
actions: read
pull-requests: write
steps:
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: leaderboard-preview
path: ${{ runner.temp }}/preview
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}

- name: Post or update the comment
env:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
run: |
preview="${RUNNER_TEMP}/preview"
number="$(head -c 20 "${preview}/number" | tr -cd '0-9')"
if [ -z "$number" ]; then
echo "::error::the preview names no pull request"
exit 1
fi
# The artifact was written by the pull request's code: comment only on the pull request it claims if that
# pull request's head is the commit checked.
head="$(gh api "repos/${REPOSITORY}/pulls/${number}" --jq .head.sha)"
if [ "$head" != "$HEAD_SHA" ]; then
echo "::error::pull request ${number} is not at the checked commit"
exit 1
fi
marker='<!-- leaderboard-preview -->'
{ echo "$marker"; tail -n +2 "${preview}/preview.md" | head -c 60000; } > "${RUNNER_TEMP}/body.md"
existing="$(gh api "repos/${REPOSITORY}/issues/${number}/comments" --paginate \
--jq ".[] | select(.user.login == \"github-actions[bot]\" and (.body | startswith(\"$marker\"))) | .id" \
| head -n 1)"
if [ -n "$existing" ]; then
method=PATCH endpoint="repos/${REPOSITORY}/issues/comments/${existing}"
else
method=POST endpoint="repos/${REPOSITORY}/issues/${number}/comments"
fi
gh api --method "$method" "$endpoint" -F "body=@${RUNNER_TEMP}/body.md" > /dev/null
135 changes: 135 additions & 0 deletions .github/workflows/leaderboard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
name: leaderboard

# The leaderboard (leaderboard/README.md): every night, each project on leaderboard/projects.toml is fetched at the
# head of its branch and scored by this validator, the results are added to the `leaderboard-data` branch, and the
# site is published with the boards. Evaluating runs no project code (the validator only parses), with read-only
# permissions and no secrets; only the job that records the results writes, and only to the data branch.
on:
schedule:
- cron: "17 3 * * *"
workflow_dispatch:

# One night at a time; a run is never cancelled halfway through recording.
concurrency:
group: leaderboard
cancel-in-progress: false

permissions:
contents: read

jobs:
list:
name: list the projects
runs-on: ubuntu-latest
outputs:
projects: ${{ steps.list.outputs.projects }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.14"

- name: Install PyYAML
run: python -m pip install --require-hashes -r .github/requirements/test.txt

- id: list
name: List the projects
run: echo "projects=$(python -m leaderboard matrix)" >> "$GITHUB_OUTPUT"

evaluate:
name: score ${{ matrix.project }}
needs: list
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
max-parallel: 8
matrix:
project: ${{ fromJSON(needs.list.outputs.projects) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.14"

- name: Install PyYAML
run: python -m pip install --require-hashes -r .github/requirements/test.txt

# A project that can not be fetched is recorded as such, unscored; the step itself does not fail.
- name: Score the project
id: score
env:
PROJECT: ${{ matrix.project }}
VALIDATOR: ${{ github.sha }}
run: |
python -m leaderboard evaluate results "$PROJECT" --cache "${RUNNER_TEMP}/checkouts" --validator "$VALIDATOR"
echo "slug=${PROJECT//\//--}" >> "$GITHUB_OUTPUT"

- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: leaderboard-${{ steps.score.outputs.slug }}
path: results/
retention-days: 7

record:
name: record the night
needs: evaluate
# Record whatever was scored, even when a project's job failed or timed out.
if: ${{ !cancelled() }}
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.14"

- name: Install PyYAML
run: python -m pip install --require-hashes -r .github/requirements/test.txt

- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: leaderboard-*
path: ${{ runner.temp }}/results
merge-multiple: true

# The data branch holds only the results (an orphan branch, created on the first night). The token is used for
# this push alone, not kept in the checkout.
- name: Add the night to the leaderboard-data branch
env:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
run: |
data="${RUNNER_TEMP}/data"
remote="https://x-access-token:${GH_TOKEN}@github.com/${REPOSITORY}.git"
if git ls-remote --exit-code --heads "$remote" leaderboard-data > /dev/null; then
git clone --quiet --depth 1 --branch leaderboard-data "$remote" "$data"
else
git init --quiet -b leaderboard-data "$data"
fi
python -m leaderboard record "${RUNNER_TEMP}/results" "$data" --date "$(date -u +%F)"
cd "$data"
git add --all
git -c user.name="github-actions[bot]" -c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit --quiet --allow-empty -m "Leaderboard for $(date -u +%F)"
git push --quiet "$remote" HEAD:leaderboard-data

publish:
name: publish the site
needs: record
if: ${{ !cancelled() && needs.record.result == 'success' }}
permissions:
contents: read
pages: write
id-token: write
uses: ./.github/workflows/pages.yml # zizmor: ignore[self-repository] the pinned actionlint rejects $/ yet
6 changes: 6 additions & 0 deletions .github/workflows/mutation-testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ jobs:
run: python -m pip install --require-hashes -r .github/requirements/mutation.txt

- name: Mutate the functions this pull request changes
# Explicit, so that GitHub adds pipefail: "mutmut run | tee" then fails when mutmut does.
shell: bash
env:
BASE: ${{ github.base_ref }}
run: |
Expand All @@ -39,6 +41,10 @@ jobs:
echo "No function with mutants changed: nothing to mutate." >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
# A mutant that breaks a loop's progress while it appends grows without bound, faster than mutmut's CPU-time
# limit stops it (the full run saw 14 GB in minutes): capping each process's address space at 4 GiB makes it
# fail as a kill instead of exhausting the runner.
ulimit -v 4194304
# mutmut copies the tree into mutants/, which is not a git repository, so the tests that call
# "git ls-files" fail there. The first run only creates the copy (its stats pass stops at the
# first failures); make the copy a repository, then run for real. The priming run is expected to fail
Expand Down
17 changes: 15 additions & 2 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: pages

# The documentation site: the README, docs/ and the generated support matrix, as HTML, built from main on every push
# and published to GitHub Pages. Each page says which commit it was built from.
# and published to GitHub Pages, with the leaderboard from its nightly results (the `leaderboard-data` branch; the
# leaderboard workflow calls this one after each night). Each page says which commit it was built from.
on:
push:
branches: [main]
Expand All @@ -10,6 +11,8 @@ on:
- .github/workflows/pages.yml
- .github/requirements/pages.txt
- tests/build_site.py
- leaderboard/**
workflow_call:
workflow_dispatch:

# One deployment at a time, and never cancelled: a cancelled deployment leaves the site in an unknown state.
Expand All @@ -36,13 +39,23 @@ jobs:
- name: Install the site's dependencies
run: python -m pip install --require-hashes -r .github/requirements/pages.txt

# Before the first night there is no data branch: the leaderboard then lists its projects.
- name: Fetch the leaderboard's results
env:
REPOSITORY: ${{ github.repository }}
run: |
git clone --quiet --depth 1 --branch leaderboard-data "https://github.com/${REPOSITORY}.git" \
"${RUNNER_TEMP}/leaderboard" || mkdir -p "${RUNNER_TEMP}/leaderboard"

- name: Build the site
env:
PYTHONPATH: ${{ github.workspace }}
REPOSITORY: ${{ github.repository }}
REF: ${{ github.ref_name }}
REVISION: ${{ github.sha }}
run: python tests/build_site.py _site --repository "$REPOSITORY" --ref "$REF" --revision "$REVISION"
run: |
python tests/build_site.py _site --repository "$REPOSITORY" --ref "$REF" --revision "$REVISION" \
--leaderboard "${RUNNER_TEMP}/leaderboard"

- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
Expand Down
2 changes: 1 addition & 1 deletion .jscpd.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@
"gitignore": true,
"ignore": ["**/__pycache__/**", "**/.venv/**", "**/*.egg-info/**"],
"ignorePattern": ["from [\\w.]+ import \\([^)]*\\)", "from [\\w.]+ import [\\w, ]+"],
"path": ["project_validator", "tests"]
"path": ["project_validator", "tests", "leaderboard"]
}
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ python3 -m project_validator --only coverage,platform-matrix --format json path/
| `--answers` | print the answers to every question about the project as JSON, per language in each unit of the project: each `yes`, `no`, `unknown` (something that could answer it was not read) or `impossible` (no known tool provides it for the language), with what shows it (the tools CI runs and the files that define them, the coverage at each granularity, the settings met, the platforms, the manifests and lockfiles, …) and what the known tools could do |
| `--only CHECKS` / `--skip CHECKS` | comma separated check names to run / not run |
| `--format text\|json\|markdown` | output format; `json` gives each check's result and, in `places`, its result for each ecosystem in each unit (status, message, details, the tools that ran, what `--auxiliary` named); `markdown` lays those places out as a grid of checks against ecosystems, for a CI job summary (and says so when no ecosystem was recognised) |
| `--score` | also score the project's tooling (a preview, [docs/score.md](docs/score.md)): its **level** (a gated tech tree: foundations, analysis, hardening, excellence), **completeness** (covered cells over applicable ones, averaged across units; auxiliary code weighs half) and **depth** (each tool in a covered cell by its rank), and the next steps that raise them most. Text and markdown end with the summary and the top five next steps; JSON becomes `{"results": [...], "score": {...}}`. The statuses and the exit code do not change |
| `--color auto\|always\|never` | color the text output (needs rich); `auto` colors only on a terminal when `NO_COLOR` is unset |
| `--exclude DIR` | leave `DIR`, relative to the project, out of the validation (e.g. test fixtures); may be repeated |
| `--build` | configure and build the project in a temporary directory and ask `ctest` for its tests; a build that cannot run makes whether there are tests unknown |
Expand All @@ -75,8 +76,8 @@ Every language is judged in each unit (build root or workspace member) of the pr
|---|---|
| `ci-present` | at least one CI configuration is found |
| `tests-run` | each language's tests exist in each unit (a separate test project's count for all of them) and CI runs them (`ctest-runs` is accepted as an old name). Warns when CI never runs the tests |
| `fuzz-testing` | each language with a fuzzer has fuzz targets in each unit (libFuzzer, cargo-fuzz, atheris, Go fuzzing, ...) and CI runs them (a ClusterFuzzLite action, or a CTest registration that passes a libFuzzer option, counts). Warns when CI never runs the targets, when CI runs a fuzzer but no targets are recognised, or when only property-based tests exist |
| `coverage` | A coverage tool CI runs measures branch coverage, or the finest coverage any known tool can measure for the language when none measures branches. The result says how far CI takes it: measured, reported (a report shows the numbers) or gated (CI fails under a threshold), and suggests the next rung. Path or MC/DC coverage is a bonus. A setting counts when CI runs the tool that reads it |
| `fuzz-testing` | each language with a fuzzer has fuzz targets in each unit (libFuzzer, cargo-fuzz, atheris, Go fuzzing, ...) and CI runs them (a ClusterFuzzLite or CIFuzz action, or a CTest registration that passes a libFuzzer option, counts). Warns when CI never runs the targets, when CI runs a fuzzer but no targets are recognised, or when only property-based tests exist |
| `coverage` | A coverage tool CI runs measures branch coverage, or the finest coverage any known tool can measure for the language when none measures branches. The result says how far CI takes it: measured, reported (a report shows the numbers; an upload to Codecov or Coveralls counts) or gated (CI fails under a threshold; a Codecov status target counts), and suggests the next rung. Path or MC/DC coverage is a bonus. A setting counts when CI runs the tool that reads it |
| `platform-matrix` | CI covers at least 2 operating systems, 2 toolchains (compilers, or the ecosystem's language versions) and 3 OS/toolchain combinations; for SQL, 2 database engine versions (the service containers CI runs: `postgres:16`), whatever the OS |
| `ci-build-types` | every OS/toolchain combination in CI builds Debug and Release, each with and without sanitizers |
| `sanitizers` | sanitizers are used in some configuration: for C/C++, AddressSanitizer and UndefinedBehaviorSanitizer (`-fsanitize=address,undefined`, MSVC's `/fsanitize=address`, cmake_template's `ENABLE_SANITIZER_*` options, Meson's `b_sanitize`), warning when only one of the two is; for Rust, `cargo miri test` or a nightly `-Zsanitizer` build; for Go, `go test -race` (the race detector; `-asan` and `-msan` count too). Fails when none is used; `ci-build-types` asks that each platform builds with and without them |
Expand Down
Loading
Loading