docs: re-measure the field scan, so the numbers match the rules that produced them - #78
Open
bgard68 wants to merge 1 commit into
Open
docs: re-measure the field scan, so the numbers match the rules that produced them#78bgard68 wants to merge 1 commit into
bgard68 wants to merge 1 commit into
Conversation
…produced them The results table predated the precision work and contradicted the section above it: the prose said three rules establish need before reporting, the numbers said GHA002 fires 346 times at High. Re-run across the same 14 repositories, on their default branches today. findings in total 2,601 -> 2,563 GHA002 findings 346 -> 319 GHA002 at High 346 -> 104 GHA004 at Critical 27 -> 0 Thirty-eight findings went away and none were hidden: 27 are grants an action in the same job cannot work without, now listed as examined and accepted, and 11 are checkouts whose job goes on to git push with the credential the finding asked them to remove. The larger change is the 242 findings that left High and Critical for the severity their scope actually carries. Per-repository counts and the totals reconcile: 564 files, 532 with findings, 2,563 findings. The reproduction steps now say the enumeration is that directory only. dotnet/runtime keeps four .eval.yaml files in workflows/evals/ which define no jobs; recursing into it reports four parse failures that are not parse failures, which is what happened on the first attempt at this re-measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N7HjzX5e8Wv7PKENDo2Uj7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The results table predated the precision work and contradicted the section above it: the prose says three rules establish need before reporting, the numbers said GHA002 fires 346 times at High.
Re-run across the same 14 repositories on their default branches today — 564 workflows, same code path (
RuleCatalogue.All()overWorkflowParser).What changed
Thirty-eight findings went away and none were hidden. Twenty-seven are grants an action in the same job cannot work without — CodeQL uploading results,
actions/staleediting the issues it sorts,create-pull-requestopening the branch it just wrote — each now listed as examined and accepted. Eleven are checkouts whose job goes on togit pushwith the credential the finding told them to remove.The larger change is the one the totals understate: 242 findings left High and Critical for the severity their scope actually carries (184 Medium, 31 Low), and every
pull_request_targetsite moved off Critical because none of the 27 checks out pull-request code.Per-repository counts are refreshed and the totals reconcile: 564 files, 532 with findings, 2,563 findings.
A note in the reproduction steps
The first attempt at this re-measurement reported four parse failures, which would have contradicted the headline claim that nothing in the wild broke the parser.
They were not parse failures.
dotnet/runtimekeeps four.eval.yamlfiles in.github/workflows/evals/, a nested folder GitHub does not treat as workflows, and they define no jobs. The original scan used the contents API, which is not recursive; my enumeration was.The steps now say that directory only, and why, so the next person to reproduce this does not spend the same ten minutes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01N7HjzX5e8Wv7PKENDo2Uj7
Generated by Claude Code