Skip to content

feat(budget): price runs from the Actions API when billing is unreachable - #132

Merged
ExtraToast merged 1 commit into
mainfrom
feat/budget-actions-api-fallback
Aug 30, 2026
Merged

feat(budget): price runs from the Actions API when billing is unreachable#132
ExtraToast merged 1 commit into
mainfrom
feat/budget-actions-api-fallback

Conversation

@ExtraToast

Copy link
Copy Markdown
Contributor

The CI failure this fixes

The watchdog has failed on every scheduled run since I deployed it — twice a day, 403 on the billing endpoint. That is the actual red in CI, and it is my fault: I shipped it without the access it needs.

A permanently red alarm is worse than no alarm. It teaches people to ignore the one signal meant to interrupt them.

Fix

It now falls back to pricing every run this month from the Actions API, which needs only actions:read. That makes it work today, with no App change and no owner action.

Billing stays the preferred source when reachable. The source is named in the log line, the step summary and the alert issue, so the two are never confused:

::notice::3263 of 3000 ... [source: billing API (exact)]
::notice::about 3386 of 3000 ... [source: Actions API (lower bound)]

Two things it had to get right

The runs endpoint silently truncates at 1000. fleet-infra alone exceeds that in a month:

query runs returned
whole month, one query 1000 (capped, no warning)
same month, weekly windows 1337

A quarter of the month was vanishing with no indication. Queries are now windowed by week.

Billing rounds each job up to a whole minute, so the fallback does too, and weights by runner multiplier from the job's labels. Summing raw durations is the mistake that makes a split pipeline look cheap — the same rounding that was 80% of this estate's spend.

Accuracy — measured, and it corrected me

source figure vs billing
billing API 3263
fallback (windowed) 3386 +4%
fallback (earlier, truncating) 3008 −5%

I had written this up as a lower bound. My own test falsified that: with truncation fixed it reads high, not low. The error has no known sign — re-run attempts overcount, retention ageing undercounts — so the code and the alert text now say "approximate" and explicitly warn against quoting the number.

Running slightly high is the safe direction for an alarm: it trips a little early rather than a little late.

Verified

Both paths exercised against live data, the fallback under a shimmed gh that 403s only the billing endpoint — reproducing the exact CI condition:

  • billing reachable → 3263 (109%), ::error::, exit 1
  • billing 403 → warns, falls back, prices 1668 runs, 3386 (113%), ::error::, exit 1

Both correctly alarm: the org really is over its August allowance. actionlint clean; the embedded script AST-parses.

Fallback runtime is ~3m40s for 1668 runs against a 15-minute timeout, on a free public runner.

Still worth doing

Granting the App "Organization plan" (read) restores the exact figure and drops the run to seconds. This removes the urgency, not the reason. Tracked in #126.

Refs #126

…able

The watchdog has been failing on every scheduled run since it was deployed:
the App token cannot read org billing, so it exited non-zero twice a day. A
permanently red alarm is worse than no alarm -- it teaches people to ignore
the one signal that is supposed to interrupt them -- and it was red because I
shipped it without the access it needs.

It now falls back to pricing every run this month from the Actions API, which
needs only actions:read. That makes it work today, with no App change. Billing
stays the preferred source when reachable, and the source is named in the log,
the step summary and the alert issue, so nobody mistakes one for the other.

Two things this had to get right:

The runs endpoint caps a single query at 1000 results and says nothing when it
truncates. fleet-infra alone exceeds that in a month: an unwindowed query
returned exactly 1000 where weekly windows returned 1337, silently losing a
quarter of the month. Queries are therefore windowed by week.

Billing rounds each job up to a whole minute, so the fallback does too, and
weights by runner multiplier from the job's labels. Summing raw durations is
the mistake that makes a split pipeline look cheap.

On accuracy, measured rather than asserted: against billing on the same month
the fallback read 3386 to billing's 3263, about 4% high. An earlier version
that let the runs endpoint truncate read 5% low. So the error does not have a
known sign and this is not a bound -- I had written "lower bound" until the
measurement contradicted it. Running slightly high is the safe direction for
an alarm; the figure is still not one to quote.

Refs #126
@ExtraToast
ExtraToast merged commit 4d0f476 into main Aug 30, 2026
6 checks passed
@ExtraToast
ExtraToast deleted the feat/budget-actions-api-fallback branch August 30, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant