ci: publish the provenance bundle, and add a second reader on pull requests - #137
Merged
Merged
Conversation
The rule was already followed and already written down twice - a new failure mode gets an exit code, a test and a README row; a new mechanism gets a numbered spot in the pipeline plus unit tests - but only as two examples of what happens, never as the policy behind them. A reader looking for the policy found instances of it. Says it once, and names what enforces it: the mutation registry records which broken behaviour each test is supposed to catch, and CI re-breaks them to prove the test actually reddens. Replaces a line added in #136 that repeated the two bullets above it verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both attestations went into GitHub's attestation store and nowhere else, which is enough for `gh attestation verify <zip> -R <repo>` and not enough for two other readers. A person holding the download and no path to that API cannot check it, and neither can anyone who took the files from a mirror - the proof has to travel with the archive. The release now uploads the bundle as `BeanNetworkTester-<tag>.sigstore.json`, and `--bundle` verifies offline. The second reader is a scanner, and it is why this was worth finding: OpenSSF Scorecard's Signed-Releases check reads release assets BY FILE EXTENSION and never opens the attestation store, so it scored 0 out of 10 on releases that already carried two signed statements. Read in probes/releasesAreSigned and probes/releasesHaveProvenance rather than guessed. The name says what the file is. The action writes a Sigstore bundle, so the asset is `.sigstore.json`. `.intoto.jsonl` would score two points higher on that check and is a different format, not another name for this one. Worth knowing before reading the first number: the check scores floor(sum / releases) over the last five releases, so one signed release scores 2, not 10. The full value arrives after five in a row. The step reads `bundle-path` off the provenance action through `env:`, never interpolated into the script, and checks the file parses as JSON before uploading a name that promises it does. Guarded by a test that also pins the extension; the existing attestation guard had to learn that the publish command now spans lines, which it was silently not reading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four analysers already read the code for shapes. This reads the change for meaning - a wrong answer, an edge case nobody tried, a contract quietly altered, a comment that stopped matching the code under it - with a context the session that wrote the change does not have. It comments; it cannot fail a pull request. The trigger is the cost control, not the model. `opened` and `ready_for_review` only, never `synchronize`, which fires on every push and would multiply the bill by how often a branch is amended. The cost of that choice is stated rather than hidden: a problem introduced by a later push is not seen, and a fresh pass is asked for with a comment. Opus 5 deliberately, on a project that ships a kernel driver. Three locks on who can spend it: the condition in the workflow, the action's own refusal of actors without write access and of bots, and GitHub withholding secrets from fork pull requests in a public repository. The job holds no write permission. The part that needed finding: `CLAUDE.md` is git-ignored here, so a runner checks out a tree without it and the review would arrive knowing none of this repository's rules - flat hyphens, English-only history, the changelog rule, fail-open - and would spend its findings on what CI already enforces. `.github/claude-review-rules.md` is a public digest of those rules, every line of it already in CONTRIBUTING and the READMEs, and the workflow copies it into place for the length of the run. That is also the failure mode the guard is for: without the copy the job still succeeds, just uselessly. A wasted run is harder to notice than a red one, so the test pins the copy step, the trigger, the actor condition and the absence of write permission. Two mutations, both caught. Setup this needs on the repository side: the Claude GitHub App installed, and a `CLAUDE_CODE_OAUTH_TOKEN` secret so runs bill against a subscription rather than opening a second meter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project is registered and passes at 100 percent of that level, so the badge row says so next to the Scorecard one. Scorecard's own CII-Best-Practices check reads the bestpractices.dev API by repository URL rather than the README, so this changes nothing about the score - it is there for a person deciding whether to install a tool that loads a kernel driver. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… term `Matcher.describe` joins terms with ", ", so a term ending in a backslash escaped that comma on the way back in and two terms silently became one. The filter a user reads stopped being the filter that runs, which is the class of lie the project's fifth convention exists to prevent, and it reaches them through the filter display, the log line and the repro report. Measured on the shape this tool meets most often - a Windows path typed with its trailing separator, then a process name. Two terms in, one term out. Only an ODD run of trailing backslashes is dangerous, so the fix drops one rather than stripping the tail: an even run escapes itself and already round-tripped. Measured across eleven shapes, three broken before and none after. The dropped character carries no meaning in any kind this language has - a regex ending in a backslash is not a valid regex at all - so this repairs a term rather than truncating one. `split_terms` is exported from the facade, so this is a change to public behaviour rather than only an internal repair. Its readers are `parse_matcher`, which builds every filter the GUI, the CLI and saved profiles use, and the GUI's add-to-filter path. A stored profile with such a term now parses one character shorter; it already displayed something other than what it did. Found by the property test that found the comma half of this - and then that test PASSED on CI the same afternoon it failed locally, because Hypothesis explores at random and its counterexample lives in a local database. So the regression test here is deterministic and spells both inputs out. Two mutations, both caught: the fix removed, and the fix widened into a plain rstrip, which is the likelier future mistake because it looks tidier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
donislawdev
marked this pull request as draft
August 19, 2026 14:42
donislawdev
marked this pull request as ready for review
August 19, 2026 14:42
Measured on this branch: the app refuses to hand out a token unless the workflow file is byte-identical to the copy on the default branch, so the action exits with "Exiting due to workflow validation skip" - a success, in 18 seconds, with no model call and nothing spent. That is the right rule. Without it a pull request could edit this file and walk off with the token. But it has two consequences worth a comment rather than a rediscovery: the workflow cannot be tested before it is merged, and every later change to it skips its own review. Worth knowing for a different reason too: a green "Claude review" check is not evidence that a review happened. The run duration is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Steps 1 to 3 of the agreed plan. Step 4 (a conditional approving review) waits for a
real cost datapoint from step 3; steps 5 and 6 (the two-phase signed release, and the
ruleset change) come after.
The provenance bundle now ships as an asset
Both attestations lived only in GitHub's attestation store. That is enough for
gh attestation verify <zip> -R <repo>and not enough for anyone holding the downloadwith no path to that API, or anyone who took the files from a mirror. The release now
uploads
BeanNetworkTester-<tag>.sigstore.json, and--bundleverifies offline.🔴 Why it was worth finding: OpenSSF Scorecard's Signed-Releases check reads release
assets by file extension and never opens the attestation store, so it scored 0/10 on
releases that already carried two signed statements. Read in
probes/releasesAreSignedand
probes/releasesHaveProvenance, not guessed.The asset is
.sigstore.jsonbecause that is what the action writes..intoto.jsonlwould score two points higher on that check and is a different format, not another name
for this one. And the check scores
floor(sum / releases)over the last fivereleases, so the first signed release scores 2, not 10 - worth knowing before reading
that number as a failure.
A second reader on pull requests
Four analysers read the code for shapes. This reads the change for meaning, with a
context the session that wrote it does not have. It comments; it cannot fail a pull
request.
The trigger is the cost control, not the model.
openedandready_for_reviewonly, never
synchronize- that one fires on every push and would multiply the bill byhow often a branch is amended. The cost of the choice is stated rather than hidden: a
problem introduced by a later push is not seen, and a fresh pass is asked for with a
comment.
Three locks on who can spend it: the condition in the workflow, the action's own refusal
of actors without write access and of bots, and GitHub withholding secrets from fork
pull requests in a public repository. The job holds no write permission.
🔴 The part that needed finding:
CLAUDE.mdis git-ignored here, so a runner checksout a tree without it. The review would arrive knowing none of this repository's rules
and would spend its findings on what CI already enforces.
.github/claude-review-rules.mdis a public digest of those rules - every line alreadyin CONTRIBUTING and the READMEs - copied into place for the length of the run. Without
that step the job still succeeds, just uselessly, which is why it is guarded: a
wasted run is harder to notice than a red one.
Testing policy
CONTRIBUTING.mdsaid the policy twice as examples and never as a policy. It now saysit once and names what enforces it.
Verification
command, and
synchronizeadded back to the review trigger.publish command, which now spans lines. Fixed in the same commit.
test_describe_reparses_to_the_same_matcher,found a genuine round-trip defect in the filter mini-language while this branch was
being verified. It predates these changes and touches none of them, but it may redden
the test job here. It is being handled separately rather than folded into this branch.
What this needs on the repository side
The Claude GitHub App installed, and a
CLAUDE_CODE_OAUTH_TOKENsecret. Until bothexist the review job fails on authentication rather than doing anything unexpected.
🤖 Generated with Claude Code