feat(budget): price runs from the Actions API when billing is unreachable - #132
Merged
Conversation
…able The watchdog has been failing on every scheduled run since it was deployed: the App token cannot read org billing, so it exited non-zero twice a day. A permanently red alarm is worse than no alarm -- it teaches people to ignore the one signal that is supposed to interrupt them -- and it was red because I shipped it without the access it needs. It now falls back to pricing every run this month from the Actions API, which needs only actions:read. That makes it work today, with no App change. Billing stays the preferred source when reachable, and the source is named in the log, the step summary and the alert issue, so nobody mistakes one for the other. Two things this had to get right: The runs endpoint caps a single query at 1000 results and says nothing when it truncates. fleet-infra alone exceeds that in a month: an unwindowed query returned exactly 1000 where weekly windows returned 1337, silently losing a quarter of the month. Queries are therefore windowed by week. Billing rounds each job up to a whole minute, so the fallback does too, and weights by runner multiplier from the job's labels. Summing raw durations is the mistake that makes a split pipeline look cheap. On accuracy, measured rather than asserted: against billing on the same month the fallback read 3386 to billing's 3263, about 4% high. An earlier version that let the runs endpoint truncate read 5% low. So the error does not have a known sign and this is not a bound -- I had written "lower bound" until the measurement contradicted it. Running slightly high is the safe direction for an alarm; the figure is still not one to quote. Refs #126
This was referenced Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CI failure this fixes
The watchdog has failed on every scheduled run since I deployed it — twice a day,
403on the billing endpoint. That is the actual red in CI, and it is my fault: I shipped it without the access it needs.A permanently red alarm is worse than no alarm. It teaches people to ignore the one signal meant to interrupt them.
Fix
It now falls back to pricing every run this month from the Actions API, which needs only
actions:read. That makes it work today, with no App change and no owner action.Billing stays the preferred source when reachable. The source is named in the log line, the step summary and the alert issue, so the two are never confused:
Two things it had to get right
The runs endpoint silently truncates at 1000.
fleet-infraalone exceeds that in a month:A quarter of the month was vanishing with no indication. Queries are now windowed by week.
Billing rounds each job up to a whole minute, so the fallback does too, and weights by runner multiplier from the job's labels. Summing raw durations is the mistake that makes a split pipeline look cheap — the same rounding that was 80% of this estate's spend.
Accuracy — measured, and it corrected me
I had written this up as a lower bound. My own test falsified that: with truncation fixed it reads high, not low. The error has no known sign — re-run attempts overcount, retention ageing undercounts — so the code and the alert text now say "approximate" and explicitly warn against quoting the number.
Running slightly high is the safe direction for an alarm: it trips a little early rather than a little late.
Verified
Both paths exercised against live data, the fallback under a shimmed
ghthat 403s only the billing endpoint — reproducing the exact CI condition:::error::, exit 1::error::, exit 1Both correctly alarm: the org really is over its August allowance.
actionlintclean; the embedded script AST-parses.Fallback runtime is ~3m40s for 1668 runs against a 15-minute timeout, on a free public runner.
Still worth doing
Granting the App "Organization plan" (read) restores the exact figure and drops the run to seconds. This removes the urgency, not the reason. Tracked in #126.
Refs #126