Billing-aware runner routing: prefer-hosted-while-free policy for private repos (gated on App billing probe)

[kyle-sexton]

Goal

Public repos: always hosted (free). Private repos: consume included hosted minutes FIRST while pool headroom exists and no budget block; otherwise — and on ANY error — route to the self-hosted fleet.

This was believed "not possible." Research (2026-07-26, adversarially re-graded by an independent verifier) refutes that: GitHub ships no native fallback mechanism (confirmed absent — runs-on arrays AND-match; GitHub explicitly declined first-party fallback in community discussion #20019, and the 2026 changelog through Jun was fetched directly to confirm nothing shipped), but this org already owns the substitute seam: select-runner's dynamic runs-on.

Research basis (findings inlined; artifact is session-local memory tier, not linkable)

• Usage API: GET /organizations/{org}/settings/billing/usage (+ /usage/summary) — live-verified HTTP 200 on this org; the legacy /orgs/.../settings/billing/actions endpoint returns 410 and is removed from the OpenAPI spec. Classic PAT admin:org verified live; GitHub App tokens expected to work (OpenAPI enabledForGitHubApps: true) but unverified pending Phase 0; fine-grained PATs officially unsupported; a workflow-scoped GITHUB_TOKEN can NEVER read billing (no such permission exists).
• Budgets API: readable (this org's $0 Actions budget with prevent_further_usage: true is visible) but exposes CONFIG ONLY — no consumed-amount field. Spend must be derived from usage rows.
• Freshness: no official SLA; empirically same-day in one probe — treat as hours-behind, never real-time. Hence headroom margins, not exact cutoffs.
• Cap-hit behavior: docs say hosted jobs are blocked at start ("The job was not started because recent account payments have failed or your spending limit needs to be increased."); a stuck-queued variant was observed locally 2026-07-18 (github-iac ADR 0008). Conflict unresolved — Phase 5 characterizes it. Vendor-docs-only claim; no independent corroboration channel exists.
• Self-hosted remains free: the announced per-minute self-hosted charge was postponed indefinitely.
• Included minutes: Free plan 2000/mo, Team 3000/mo for private repos (verify the org's plan at implementation). ubuntu-slim pool-draw behavior and OS-multiplier mechanics are open gaps — decidable cheaply in the first days after the Aug-1 rollover.
• Unmatched-label queue: jobs on an unroutable label auto-fail at 24h (vendor-docs-only).

Verdict gate

Feasibility is MEDIUM-confidence until Phase 0's empirical probe passes. Everything downstream is gated on it.

Phases

• Phase 0 (GATE): extend the observer GitHub App with billing read; probe the usage endpoint with an App token; record the working permission name (docs never name it). A 200 settles feasibility HIGH.
• Phase 1: scheduled poll (every 2–6h) reads usage + budget state, caches routing state in an org variable (private-pool consumption %, any private paid spend).
• Phase 2: new select-runner policy prefer-hosted-while-free: route hosted only while consumption < ~85% AND no private paid spend; fleet otherwise and on any error (fail toward the fleet). The selector job itself stays fleet-routed — a hosted selector dies with its own cap.
• Phase 3: consumer rollout. Sequencing dependency: gates: private-caller hosted fallback in semantic-pr / do-not-merge-gate / pr-issue-linkage reusables #148's pin-bump rollout completes first.
• Phase 4: Aug-1 rollover tests: ubuntu-slim pool draw; OS-multiplier mechanics. Shared window with Decide: fail-closed publisher recovery shape on fleet outage after the fleet-label flip (carve-out vs sentinel) #241's revisit trigger — one test session covers all three.
• Phase 5: characterize actual cap-hit behavior (fail-at-start vs stuck-queued) and encode the observed truth in publisher expectations.

Genuinely impossible (design around, don't chase)

Native fallback; readable "budget currently blocking" state; real-time usage; GITHUB_TOKEN-only design; graceful behavior for jobs racing the cap inside the polling window.

References

• Decide: fail-closed publisher recovery shape on fleet outage after the fleet-label flip (carve-out vs sentinel) #241 — closed decision (fleet-recovery interim, hosted-selector direction deferred); shares the Aug-1 trigger.
• gates: private-caller hosted fallback in semantic-pr / do-not-merge-gate / pr-issue-linkage reusables #148 — retargeted to rollout completion; independent prerequisite for Phase 3.
• select-runner: zero self-hosted capacity surfaces as a GitHub billing failure, sending the operator to the wrong place #246 — select-runner billing-failure surfacing (adjacent UX).
• tracker binding: config.role_labels is null — canonical roles run on defaults, loud warning every lane pass #247 — tracker binding role_labels (unrelated mechanically, same session).
• github-iac ADR 0008 revisit issue — linked in a follow-up comment once filed.

---

Filed by an AI agent at the operator's direction (attended queue session, operator-ratified 2026-07-26).

[kyle-sexton]

github-iac ADR 0008 revisit issue filed alongside: https://github.com/melodic-software/github-iac/issues/226

AI agent (work-items:attend-queue lane).

[kyle-sexton]

Operator brief (drain orchestration)

Status: blocked on host — Phase 0 gate unpassed; all downstream phases (poll, policy, rollout, Aug-1 tests, cap-hit characterization) remain parked.

Ordered next actions

1. Phase 0 (GATE) — App billing probe. Extend the runner-observer GitHub App (or create a minimal billing-read App per org precedent) with billing/usage read permission. Probe GET /organizations/{org}/settings/billing/usage (and /usage/summary) with an App installation token; record the working permission name (docs do not name it). A 200 elevates feasibility to HIGH and unblocks Phase 1.
2. Org-owner credential work. Classic PAT with admin:org was live-verified in the 2026-07-26 research session, but GitHub App token path is explicitly unverified. Fine-grained PATs are officially unsupported; workflow GITHUB_TOKEN can never read billing. This step requires org-owner / github-iac browser or IaC changes — not agent-automatable.
3. After Phase 0 passes — Phase 1: scheduled poll (2–6h) caches routing state in an org variable (private-pool consumption %, any private paid spend).
4. Phase 2: implement prefer-hosted-while-free in select-runner — hosted while consumption < ~85% AND no private paid spend; fleet on any error (fail toward fleet). Selector job itself must stay fleet-routed.
5. Phase 3: consumer rollout — prerequisite #148 is closed (hosted fallback in semantic-pr / do-not-merge-gate / pr-issue-linkage reusables landed).
6. Phase 4: Aug-1 rollover tests (ubuntu-slim pool draw, OS multipliers) — shares window with #241 revisit trigger (closed; fleet-recovery interim decision recorded).
7. Phase 5: characterize cap-hit behavior (fail-at-start vs stuck-queued) — see github-iac#226 (ADR 0008 revisit).

Agent verified vs human/host only

Agent verified	Human/host only
Native GitHub fallback does not exist (runs-on arrays AND-match; community #20019)	App permission widening or new App creation
Usage API live 200 on org with classic PAT admin:org (2026-07-26 research)	App-token billing probe (Phase 0 deliverable)
Budgets API exposes config only, not consumed amount	Org plan verification for included-minutes math
#148 closed — Phase 3 sequencing prerequisite met	Policy call on prefer-hosted-while-free thresholds (~85%)
#241 closed — interim fleet-recovery shape decided	Aug-1 test session scheduling (operator)
Adjacent UX tracked in #246 (zero capacity → billing false lead)	—
No prefer-hosted-while-free policy or billing poll code in repo yet	—

Links

• Issue body research basis (2026-07-26, operator-ratified)
• Prerequisite: #148 (closed)
• Shared Aug-1 trigger: #241 (closed)
• ADR revisit: github-iac#226
• Adjacent: #246 (select-runner billing false lead)
• Observer App today: CI_RUNNER_OBSERVER_CLIENT_ID org variable; select-runner.yml

[kyle-sexton]

Status (2026-08-12 drain): still host-blocked at Phase 0 — leave open. Operator brief from 2026-08-11 remains current.

Remaining host action: extend runner-observer (or minimal) GitHub App with billing/usage read; probe GET /organizations/{org}/settings/billing/usage with an App installation token; record the working permission name. A 200 unblocks Phase 1. Org-owner / github-iac only — not agent-automatable. Downstream prefer-hosted-while-free work stays parked until that gate passes.

[kyle-sexton]

Frontier consensus (Fable + Sol, 2026-08-12): keep open — see drain consensus table (host / engineering / dated). No autonomous closure.