fix(budget): refuse to report a percentage the watchdog cannot substantiate - #133
Merged
Conversation
…ntiate The previous change made the watchdog go green reporting "0 of 3000 (0%)". It could read neither billing nor workflow runs, warned about it, and then reported a comfortable zero as a pass. That is worse than the red it replaced, and worse in the specific way my own commit message had warned against two commits earlier: a budget watchdog that reports no usage because it cannot see usage looks like good news. A false all-clear on a budget alarm is the failure mode the whole thing exists to prevent. Two changes. A partial or empty read is now refused rather than averaged into a percentage. If any run-listing query is rejected, or if zero runs are found across every private repo in a month, the run fails and says what to grant. Zero runs org-wide is not a small number, it is evidence the measurement did not happen. The capable token is now minted best-effort with continue-on-error instead of behind an opt-in flag. create-github-app-token fails the job when asked for a permission the installation lacks, which is why the flag existed -- but allowing that one step to fail achieves the same protection without anyone having to set anything, and an App that does hold the grants is used automatically. It asks for organization-plan: read for the exact billing figure and actions: read for the fallback, and falls back to a minimal token that is only used to file the alert issue. Verified by reproducing the production condition -- a token that lists repos but is refused both billing and runs -- which now exits 1 naming the permissions, where before it exited 0 claiming 0%. Refs #126
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What went wrong
My previous change (#132) made the watchdog go green, reporting:
It could read neither billing nor workflow runs, warned about both, and then reported a comfortable zero as a pass.
That is worse than the red it replaced — and worse in exactly the way my own commit message had warned against two commits earlier: "a budget watchdog that quietly reports 0% when it cannot read the budget is worse than one that is absent, because it looks like good news." I wrote that, then shipped the thing it describes. A false all-clear on a budget alarm is the precise failure the alarm exists to prevent.
The org is at 109% of its allowance while this reported 0%.
Fix 1 — refuse to report what cannot be substantiated
A partial read is no longer averaged into a reassuring percentage.
Fix 2 — best-effort capable token instead of an opt-in flag
create-github-app-tokenfails the job when asked for a permission the installation lacks — that is why theapp-has-billing-permissionflag existed. Butcontinue-on-error: trueon that one step achieves the same protection with nothing to configure, and an App that does hold the grants is used automatically.Token precedence: capable App token →
BUDGET_TOKEN→ minimal token. The flag is removed.Verified against the exact production condition
A shimmed
ghthat 403s both billing and runs — reproducing what actually happened:0 of 3000 (0%), exit 0::error::naming the grants, exit 1actionlintclean.Honest status
The watchdog will stay red until the App gets
actions: read(for the fallback) ororganization-plan: read(for the exact figure), or aBUDGET_TOKENis supplied. My attempt to remove that owner action did not succeed — it only moved which permission is needed.Red-and-honest is the correct state for a deployed alarm that cannot see its subject. Tracked in #126.
Refs #126