Skip to content

ci: answer the Scorecard report, and pin what the scanner cannot see - #134

Merged
donislawdev merged 4 commits into
masterfrom
ci/scorecard-findings
Aug 19, 2026
Merged

ci: answer the Scorecard report, and pin what the scanner cannot see#134
donislawdev merged 4 commits into
masterfrom
ci/scorecard-findings

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

What this is

The first OpenSSF Scorecard report raised 16 findings on master. This closes the
five that are worth closing in the tree, answers four in writing, and leaves seven
to time or to arithmetic. The interesting half is what it does not do.

Fixed

release.yml grants write on the job, not on the file (Token-Permissions, high,
0/10). contents: write, id-token: write and attestations: write moved onto the
single release job; the file grants contents: read. ci.yml, pages.yml and
scorecard.yml already worked this way, so this was an inconsistency, not a
decision. It is not cosmetic while there is one job: a permission written at the top
is inherited by every job added later, by an author with no reason to scroll up, in
the one workflow that publishes under this project's name.

pip install --upgrade pip is gone from all five places it appeared. It fetched
an unpinned pip from the index and then used it to verify the artefact hashes on the
next line - the one unchecked link in a hash-checked chain. The pip that checks them
is now the one setup-python shipped, and that action is pinned by SHA.

requirements-lint.txt is hash-checked and holds its whole closure (13 packages:
ruff, mypy, diff-cover and the ten underneath them). Resolved with pip for
Python 3.14 and identical on manylinux2014_x86_64 and win_amd64, which is why no
line carries an environment marker.

requirements-scan.txt is new and is the one file here pinned by version and not
by bytes: semgrep and pip-audit. Reasoning is written in the file.

The tests job installs the lint file whole instead of grepping diff-cover out
of it. A hash cannot travel through a command substitution - grep hands pip a bare
pin and pip stays out of hash-checking mode.

The measurement behind it

🔴 Scorecard does not read workflow steps that run in a Windows shell. From its
source: "Skip unsupported shells. We don't support Windows shells". Three of the
five --upgrade pip lines were in windows-latest jobs, as were two
pip install -r requirements-lint.txt lines it never mentioned. It listed 8 of the
13
unpinned pip commands in this repository.

Fixing exactly what it named would have produced Pinned-Dependencies 10/10 with five
unpinned installs still in the tree. An alert list is a sample, not an inventory -
which is why the guard for this lives in the suite and not in the scanner.

Also read from source rather than assumed: --require-hashes is the only thing
that makes a pip command count as pinned (a version in the file does not), and the
flag cannot be added for show, because pip refuses the install without hashes behind
it.

Deliberately not done

  • requirements-dev.txt stays unpinned. pytest, hypothesis and pytest-cov
    track latest on purpose - that drift is what the weekly run exists to see. Freezing
    them to satisfy a scanner trades a real signal for a number. Two alerts closed as
    won't-fix.
  • Branch-Protection (4/10) cannot rise here. computeFinalScore returns early
    when a tier is not maxed, so the nine required status checks this repository
    already has (Tier 3) are never reached while Tier 2 (at least one approving review)
    is unmet - and nobody can approve their own pull request. Raising the approval count
    behind an admin bypass would move the number and make the rule theatre. Code-Review
    (0/29 approved) is the same wall from the other side.
  • Fuzzing (0/10) is unreachable with the fuzzing already here. Scorecard
    recognises Python through exactly one pattern, import atheris in a *.py file.
    Property-based testing counts for six other languages and not for Python, so the
    hypothesis suites on the filter mini-language and the CLI are invisible to it.
  • Maintained (0/10), SAST (9/10) and CI-Tests (9/10) are arithmetic. A repository
    younger than 90 days, and one pull request out of 29 predating those jobs. They
    close themselves.

Guards

Five checks in tests/test_version_and_release.py and four mutation-registry
entries, all four caught on the first run: the two requirement files each held to
its own promise, no workflow bootstrapping pip from the index, every install of a
hashed file passing --require-hashes, and no write permission at the file level in
release.yml.

Verification

  • Full suite: 1201 passed. GUI smoke: OK. ruff --select F,B,C90: clean. mypy: clean.
  • The lint file installed into a clean virtual environment under --require-hashes.
  • The tests job's three install commands replayed in order, because pytest and the
    lint closure share pluggy.
  • The regeneration command documented in the file was run, and returns the same 13
    packages.

Also in this branch

The first commit is unrelated to the above: a changelog line telling users they can
check where a download came from, which the previous change shipped without
announcing.

🤖 Generated with Claude Code

donislawdev and others added 4 commits August 19, 2026 12:06
The build-provenance attestation shipped with the last change and both READMEs
explain how to verify it, but CHANGELOG.md never said so - and a user reads
the changelog to find out what is new. Rule: anything visible from outside
goes in there, and a new way to verify a release is as visible as it gets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…flow

The top-level block held `contents: write`, `id-token: write` and
`attestations: write`, so every job added to this file later would inherit
them - in the one workflow that can publish an asset under this project's
name. They now sit on the single `release` job, and the file itself grants
`contents: read`. ci.yml, pages.yml and scorecard.yml already worked this
way; this was the last file breaking the rule.

Raised by OpenSSF Scorecard as Token-Permissions, 0 out of 10.

The install step also loses `python -m pip install --upgrade pip`, which
fetched an unpinned pip from the index and then used it to verify the
artefact hashes on the very next line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
requirements-lint.txt is now installed with --require-hashes, which is why it
carries its whole closure rather than three names: ruff, mypy, diff-cover and
the ten packages underneath them. pip turns hash-checking on for an entire
install as soon as one requirement carries a hash, so every package it
resolves has to be written down. The closure was resolved with pip for Python
3.14 and came back identical on manylinux2014_x86_64 and win_amd64, so no
line needs an environment marker.

semgrep and pip-audit move to a new requirements-scan.txt, pinned by version
and deliberately not by hash. Their closures are tens of packages that move on
their own schedule, and against that: both run with a read-only token, both
produce a report, and neither can put a byte into a release, because the build
and release jobs install only from requirements.txt and requirements-build.txt
and both of those are hash-checked. semgrep cannot be frozen in the sense that
matters anyway - the pin fixes the engine, and the rules are fetched from the
registry at scan time. The cost is named in that file rather than hidden: a
scanner will keep counting the semgrep job's install as unpinned, and it is
right to.

The tests job installs the lint file whole instead of grepping diff-cover out
of it. A hash cannot travel through a command substitution: grep hands pip a
bare pin, and pip stays out of hash-checking mode. With semgrep moved out,
installing the file no longer drags a 57 MB engine onto both matrix legs.

`python -m pip install --upgrade pip` is gone from the four places left in
these two workflows, for the reason given in the previous commit.

Verified rather than assumed: the lint file installs into a clean virtual
environment under --require-hashes, and the tests job's three install commands
were replayed in order, because pytest and the lint closure share pluggy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five checks in tests/test_version_and_release.py, and four mutation-registry
entries that were all caught on the first run:

* the two requirement files, each held to its own promise - the lint file
  fully hashed (a shared helper now checks it and the runtime file with one
  piece of code), the scan file pinned with == at least;
* no workflow may bootstrap pip from the index again;
* every install of a hashed file must pass --require-hashes. pip does turn
  hash-checking on by itself, so without the flag the day a line loses its
  hash block is the day the check silently stops being mandatory;
* release.yml may not grant a write permission at the file level.

The pip-bootstrap guard exists because the scanner is blind to most of what it
covers. OpenSSF Scorecard skips workflow steps that run in a Windows shell,
and three of the five bootstraps were in windows-latest jobs, so it named two
and said nothing about the rest - 8 of the 13 unpinned pip commands in this
repository. Fixing exactly what a scanner lists would have produced a perfect
score with five unpinned installs still in the tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev merged commit 8bae7f1 into master Aug 19, 2026
13 of 14 checks passed
@donislawdev
donislawdev deleted the ci/scorecard-findings branch August 19, 2026 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant