Skip to content

ci: publish the provenance bundle, and add a second reader on pull requests - #137

Merged
donislawdev merged 6 commits into
masterfrom
ci/release-bundle-and-claude-review
Aug 19, 2026
Merged

ci: publish the provenance bundle, and add a second reader on pull requests#137
donislawdev merged 6 commits into
masterfrom
ci/release-bundle-and-claude-review

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

Steps 1 to 3 of the agreed plan. Step 4 (a conditional approving review) waits for a
real cost datapoint from step 3; steps 5 and 6 (the two-phase signed release, and the
ruleset change) come after.

The provenance bundle now ships as an asset

Both attestations lived only in GitHub's attestation store. That is enough for
gh attestation verify <zip> -R <repo> and not enough for anyone holding the download
with no path to that API, or anyone who took the files from a mirror. The release now
uploads BeanNetworkTester-<tag>.sigstore.json, and --bundle verifies offline.

🔴 Why it was worth finding: OpenSSF Scorecard's Signed-Releases check reads release
assets by file extension and never opens the attestation store, so it scored 0/10 on
releases that already carried two signed statements. Read in probes/releasesAreSigned
and probes/releasesHaveProvenance, not guessed.

The asset is .sigstore.json because that is what the action writes. .intoto.jsonl
would score two points higher on that check and is a different format, not another name
for this one. And the check scores floor(sum / releases) over the last five
releases, so the first signed release scores 2, not 10 - worth knowing before reading
that number as a failure.

A second reader on pull requests

Four analysers read the code for shapes. This reads the change for meaning, with a
context the session that wrote it does not have. It comments; it cannot fail a pull
request.

The trigger is the cost control, not the model. opened and ready_for_review
only, never synchronize - that one fires on every push and would multiply the bill by
how often a branch is amended. The cost of the choice is stated rather than hidden: a
problem introduced by a later push is not seen, and a fresh pass is asked for with a
comment.

Three locks on who can spend it: the condition in the workflow, the action's own refusal
of actors without write access and of bots, and GitHub withholding secrets from fork
pull requests in a public repository. The job holds no write permission.

🔴 The part that needed finding: CLAUDE.md is git-ignored here, so a runner checks
out a tree without it. The review would arrive knowing none of this repository's rules
and would spend its findings on what CI already enforces.
.github/claude-review-rules.md is a public digest of those rules - every line already
in CONTRIBUTING and the READMEs - copied into place for the length of the run. Without
that step the job still succeeds, just uselessly, which is why it is guarded: a
wasted run is harder to notice than a red one.

Testing policy

CONTRIBUTING.md said the policy twice as examples and never as a policy. It now says
it once and names what enforces it.

Verification

  • Full suite, GUI smoke, ruff, mypy: green on these changes.
  • Two mutations, both caught on the first run: the bundle dropped from the publish
    command, and synchronize added back to the review trigger.
  • The existing attestation guard was silently reading only the first line of the
    publish command, which now spans lines. Fixed in the same commit.

⚠️ Unrelated: a property-based test, test_describe_reparses_to_the_same_matcher,
found a genuine round-trip defect in the filter mini-language while this branch was
being verified. It predates these changes and touches none of them, but it may redden
the test job here. It is being handled separately rather than folded into this branch.

What this needs on the repository side

The Claude GitHub App installed, and a CLAUDE_CODE_OAUTH_TOKEN secret. Until both
exist the review job fails on authentication rather than doing anything unexpected.

🤖 Generated with Claude Code

donislawdev and others added 5 commits August 19, 2026 15:49
The rule was already followed and already written down twice - a new failure mode
gets an exit code, a test and a README row; a new mechanism gets a numbered spot in
the pipeline plus unit tests - but only as two examples of what happens, never as
the policy behind them. A reader looking for the policy found instances of it.

Says it once, and names what enforces it: the mutation registry records which broken
behaviour each test is supposed to catch, and CI re-breaks them to prove the test
actually reddens.

Replaces a line added in #136 that repeated the two bullets above it verbatim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both attestations went into GitHub's attestation store and nowhere else, which is
enough for `gh attestation verify <zip> -R <repo>` and not enough for two other
readers. A person holding the download and no path to that API cannot check it, and
neither can anyone who took the files from a mirror - the proof has to travel with
the archive. The release now uploads the bundle as
`BeanNetworkTester-<tag>.sigstore.json`, and `--bundle` verifies offline.

The second reader is a scanner, and it is why this was worth finding: OpenSSF
Scorecard's Signed-Releases check reads release assets BY FILE EXTENSION and never
opens the attestation store, so it scored 0 out of 10 on releases that already
carried two signed statements. Read in probes/releasesAreSigned and
probes/releasesHaveProvenance rather than guessed.

The name says what the file is. The action writes a Sigstore bundle, so the asset is
`.sigstore.json`. `.intoto.jsonl` would score two points higher on that check and is
a different format, not another name for this one.

Worth knowing before reading the first number: the check scores
floor(sum / releases) over the last five releases, so one signed release scores 2,
not 10. The full value arrives after five in a row.

The step reads `bundle-path` off the provenance action through `env:`, never
interpolated into the script, and checks the file parses as JSON before uploading a
name that promises it does. Guarded by a test that also pins the extension; the
existing attestation guard had to learn that the publish command now spans lines,
which it was silently not reading.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four analysers already read the code for shapes. This reads the change for meaning -
a wrong answer, an edge case nobody tried, a contract quietly altered, a comment that
stopped matching the code under it - with a context the session that wrote the change
does not have. It comments; it cannot fail a pull request.

The trigger is the cost control, not the model. `opened` and `ready_for_review` only,
never `synchronize`, which fires on every push and would multiply the bill by how
often a branch is amended. The cost of that choice is stated rather than hidden: a
problem introduced by a later push is not seen, and a fresh pass is asked for with a
comment. Opus 5 deliberately, on a project that ships a kernel driver.

Three locks on who can spend it: the condition in the workflow, the action's own
refusal of actors without write access and of bots, and GitHub withholding secrets
from fork pull requests in a public repository. The job holds no write permission.

The part that needed finding: `CLAUDE.md` is git-ignored here, so a runner checks out
a tree without it and the review would arrive knowing none of this repository's rules
- flat hyphens, English-only history, the changelog rule, fail-open - and would spend
its findings on what CI already enforces. `.github/claude-review-rules.md` is a public
digest of those rules, every line of it already in CONTRIBUTING and the READMEs, and
the workflow copies it into place for the length of the run.

That is also the failure mode the guard is for: without the copy the job still
succeeds, just uselessly. A wasted run is harder to notice than a red one, so the
test pins the copy step, the trigger, the actor condition and the absence of write
permission. Two mutations, both caught.

Setup this needs on the repository side: the Claude GitHub App installed, and a
`CLAUDE_CODE_OAUTH_TOKEN` secret so runs bill against a subscription rather than
opening a second meter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project is registered and passes at 100 percent of that level, so the badge row
says so next to the Scorecard one. Scorecard's own CII-Best-Practices check reads the
bestpractices.dev API by repository URL rather than the README, so this changes
nothing about the score - it is there for a person deciding whether to install a tool
that loads a kernel driver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… term

`Matcher.describe` joins terms with ", ", so a term ending in a backslash escaped
that comma on the way back in and two terms silently became one. The filter a user
reads stopped being the filter that runs, which is the class of lie the project's
fifth convention exists to prevent, and it reaches them through the filter display,
the log line and the repro report.

Measured on the shape this tool meets most often - a Windows path typed with its
trailing separator, then a process name. Two terms in, one term out.

Only an ODD run of trailing backslashes is dangerous, so the fix drops one rather
than stripping the tail: an even run escapes itself and already round-tripped.
Measured across eleven shapes, three broken before and none after. The dropped
character carries no meaning in any kind this language has - a regex ending in a
backslash is not a valid regex at all - so this repairs a term rather than
truncating one.

`split_terms` is exported from the facade, so this is a change to public behaviour
rather than only an internal repair. Its readers are `parse_matcher`, which builds
every filter the GUI, the CLI and saved profiles use, and the GUI's add-to-filter
path. A stored profile with such a term now parses one character shorter; it already
displayed something other than what it did.

Found by the property test that found the comma half of this - and then that test
PASSED on CI the same afternoon it failed locally, because Hypothesis explores at
random and its counterexample lives in a local database. So the regression test here
is deterministic and spells both inputs out. Two mutations, both caught: the fix
removed, and the fix widened into a plain rstrip, which is the likelier future
mistake because it looks tidier.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev marked this pull request as draft August 19, 2026 14:42
@donislawdev
donislawdev marked this pull request as ready for review August 19, 2026 14:42
Measured on this branch: the app refuses to hand out a token unless the workflow file
is byte-identical to the copy on the default branch, so the action exits with
"Exiting due to workflow validation skip" - a success, in 18 seconds, with no model
call and nothing spent.

That is the right rule. Without it a pull request could edit this file and walk off
with the token. But it has two consequences worth a comment rather than a rediscovery:
the workflow cannot be tested before it is merged, and every later change to it skips
its own review.

Worth knowing for a different reason too: a green "Claude review" check is not
evidence that a review happened. The run duration is.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev merged commit e24cc27 into master Aug 19, 2026
13 of 14 checks passed
@donislawdev
donislawdev deleted the ci/release-bundle-and-claude-review branch August 19, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant