Skip to content

fix(ci): name an exhausted API quota once, at every read and at the create - #303

Merged
github-actions[bot] merged 1 commit into
mainfrom
fix/backfill-quota-class
Nov 10, 2026
Merged

github-actions[bot] merged 1 commit into
mainfrom
fix/backfill-quota-class

Conversation

@aitrawog

Copy link
Copy Markdown
Contributor

Summary

opencharly/.github#300. The fleet run of scripts/backfill-missing-tags.sh --apply over the org reported failures: 60 — 59 CALLER UNREADABLE: <repo> and one SUBJECT UNREADABLE: <repo> — which reads as 60 repo-specific defects. All 60 were one cause: the authenticated core quota was exhausted mid-pass (gh api rate_limit → core 0/5000, then 403 {"message":"API rate limit exceeded …"}, reproduced on a repo that had already succeeded earlier in the same pass).

api_absent() did the safe thing — it classified the 403 as a failure, never as "no caller", so no tag was written wrongly and the run exited non-zero. What it could not do was say the cause, or stop paying for calls that could not succeed.

What this does (2 files, +242 −35):

  1. One classifier. classify_api_error() answers 404 = absent, a spent quota = refused, anything else = failure. api_absent(), the subject read, the parent read and the create all call it, so the predicate exists once.
  2. One bookkeeping path. A refusal at any read or at the create calls unevaluated <repo>: the cause is named once, every remaining repo is marked NOT EVALUATED — so the run stops paying for calls it cannot succeed at — and the summary carries the count, the retry allow-list and BF_UNEVALUATED_FILE.
  3. The safety behaviour is untouched. Add-only, the per-repo CalVer form, the explained-head skip and the point-read identity checks all behave exactly as before.

Scope — narrower than the PRs this replaces, and that is deliberate

#301 and #302 were auto-closed by the validator at five counting verdicts each (AI_REVIEW_AUTO_CLOSE_AFTER=5). Their tail had one cause that was mine, not the validator's: they also changed a generic API read failure from ABORT to CONTINUE, which #300 never asked for. That turned one new answer into a rc=2 branch at every call site, and each of those was a new behaviour that rightly demanded an arm.

This change removes it. A read that fails for a reason other than 404 or a spent quota aborts exactly as it did before, and the one new answer is the quota class. The 19 arms below cover every refusal site — caller, head, tags listing, second (parent) tags listing, confirming ref, 422-path ref, subject, parent, create — and nothing else changed.

How tested

Gate: this repo's own .github/workflows/workflow-checks.yml owner-script tests — the class that owns a change to scripts/*.sh — run offline against a mock gh, on the committed tree d5d48b6.

The changed script's own suite, all 19 arms:

$ bash scripts/backfill-missing-tags_test.sh
ok   an unexplained head is tagged at that head
ok   an explained head is skipped
ok   a proxy-consumed root module gets Go's tag form
ok   an existing tag means no action
ok   a repo with no caller is skipped
ok   check mode writes nothing
ok   a CHANGELOG tip with an untagged parent is tagged
ok   a successful create is verified by the ref, not by the listing
ok   a lost create is verified by the ref, not by the POST's exit status
ok   a ref at a different commit is a failure
ok   an exhausted quota is one named class, once, and the run stops paying for it
ok   a quota refusal at the head fetch is the same one named class, with a runnable retry line
ok   a quota refusal at the tags listing reports through the same summary and set
ok   a quota refusal at the confirming ref read is reported, never counted as tagged
ok   a quota refusal at the subject read is the same one named class
ok   a quota refusal at the parent read is the same one named class
ok   a quota refusal at the create POST is reported as NOT EVALUATED, not as a create failure
ok   a quota refusal at the SECOND (parent) tags read is reached and reported
ok   a quota refusal at the ref read inside the 422 path is reported, not mislabelled
PASS: the backfill tags only what is untagged and unexplained, in the right form, verifies every create by the ref itself, and names an exhausted quota once at every read AND at the create (19 properties)

The new arms fail on the PRE-CHANGE script — the coverage is not vacuous. Same suite, with git show origin/main:scripts/backfill-missing-tags.sh in place of the changed one:

$ bash $T/scripts/backfill-missing-tags_test.sh
ok   an unexplained head is tagged at that head
ok   an explained head is skipped
ok   a proxy-consumed root module gets Go's tag form
ok   an existing tag means no action
ok   a repo with no caller is skipped
ok   check mode writes nothing
ok   a CHANGELOG tip with an untagged parent is tagged
ok   a successful create is verified by the ref, not by the listing
ok   a lost create is verified by the ref, not by the POST's exit status
ok   a ref at a different commit is a failure
FAIL: the cause must be named exactly once, not once per repo

The sibling owner-script tests, on the same tree (they share scripts/lib-org.sh, which this diff does not touch):

$ bash scripts/reconcile-tag-on-merge-callers_test.sh
  ok   canonical carries an immutable pin
  ok   unpinned @v7 -> mutable (the T3 question)
  ok   pinned pin, old prose -> still pinned
  ok   both defects -> mutable on column 1
  ok   no github-script step -> no-uses
  ok   no caller -> absent
  ok   canonical prose -> identical
  ok   only the pin differs -> prose identical
  ok   old prose -> drifted (the revision question)
  ok   both defects -> drifted on column 2
  ok   no caller -> absent
  ok   empty allow-list passes everything
  ok   a single name keeps only itself
  ok   comma-separated names keep both, in arrival order
  ok   space-separated names keep both, in ARRIVAL order not request order
  ok   an unmatched allow-list selects nothing
  ok   a whitespace-only allow-list passes everything

reconcile-tag-on-merge-callers_test: 17 passed, 0 failed
$ bash scripts/reconcile-tag-on-merge-callers_tag_test.sh
ok   reconcile tags the commit it wrote (v2026.314.0800)
ok   the reconciled line names the tag
  NO COMMIT SHA RETURNED — cannot tag it: fixture
ok   a missing commit sha fails loudly
ok   the source repo is skipped, never written or tagged
ok   an existing tag is left alone
ok   calver_at is per-repo: module=v0.2026314.800 plain=v2026.314.0800
PASS: the reconcile tags every write it makes, guards the source repo, does not re-tag, and uses Go's form for module repos (6 properties)
$ bash scripts/tag-on-merge-assert_test.sh
ok   c1
ok   c2
ok   c3  ->  PR #101 — merge aaaa111111111111111111111111111111111111 (merged 2026-11-10T19:02:08Z) has no tag
ok   c4  ->  PR #101 — merge aaaa111111111111111111111111111111111111 (merged 2026-11-10T19:02:08Z) has no tag
ok   c5
ok   c6
PASS: tag-on-merge untagged-merge detector (8 properties: 4 green arms, 2 red arms naming the merge, 2 structural)
$ bash scripts/assert-org-rulebook_test.sh
assert-org-rulebook_test: OK (a project rulebook passes; an empty prompt fails and names the cause; a prompt that names no project fails; the step forwards vars.AI_REVIEW_PROMPT)
$ python3 scripts/check-action-runtimes.py --self-test
RC=0

Not run here, stated rather than glossed: scripts/retired-sweep-arm_test.sh (it clones the pinned contract over the network). It exercises the retired-name sweep, not this script; CI runs it on this PR.

Tier, and why it is not higher. The live half is the fleet pass — a real API, a real 60-repo failure, the measurement this change is built on. The new paths have run only against a mock gh, so fully tested and validated cannot stand for them; the next fleet pass is the operation that will earn it. The model named in the trailer is the runtime's own for this session.

Rulebook compliance

Rule How this change honours it
R0 /charly-internals:git-workflow loaded before the branch; this repo's AGENTS.md read first (Part II #4)
R1 The anomaly is #300, filed with mechanism, missed control, blast radius and root fix, and claimed before branching. The previous PRs' findings are carried here as a smaller diff rather than as new branches
R2 One classifier (classify_api_error), one bookkeeping path (unevaluated), one summary
R3 No retry, no sleep, no suppression. The run still exits non-zero; it stops issuing calls it knows cannot succeed
R4 The product (the script) is fixed; no doc edited to route around it
R5 No legacy path, shim or alias. The rc=2 continuation the previous PRs introduced is deleted, not left behind
R6 git status read first: only the two scripts/backfill-missing-tags* files changed, both authored by this session; no foreign work touched, no force-push, no bypass
R7 Coverage that fails without the change — the pre-change FAIL pasted above; and the arms assert the ARTEFACT at its boundary (the written set, the call count, the runnable retry line, the absence of any created tag)
R8 The API boundary is mocked in the offline suite, and the mock emits the real 403 body; the real service was exercised live by the fleet pass that found the defect
R9 No charly binary is involved — a bash script in the org-config repo
R10 The change-class gate for scripts/*.sh in this repo is the owner-script test set in workflow-checks.yml, run above on the committed tree with zero failures. Every refusal site has an arm, enumerated by where a refusal can ARISE

Change classification

  • Scope: scripts/backfill-missing-tags.sh and its test. No workflow, no rulebook, no ruleset, no pin.
  • What changes for consumers: same verbs (check / --apply), same output lines, plus an API RATE LIMITED line and summary field, and a non-zero exit when repos were not evaluated. Every other path — including what a generic read failure does — is unchanged.
  • Rollback: revert this commit; an exhausted quota again renders as N generic failures, still without writing a wrong tag.

Filed issues

None new. opencharly/.github#300 is the finding this change closes.

Diff — origin/main...HEAD on d5d48b6

$ git diff --stat origin/main...HEAD
 scripts/backfill-missing-tags.sh      | 127 +++++++++++++++++++++-------
 scripts/backfill-missing-tags_test.sh | 150 ++++++++++++++++++++++++++++++++--
 2 files changed, 242 insertions(+), 35 deletions(-)

Closes #300

Agent: containerd-crabbox-program · session 8b1c01cf-7f79-4a75-bff8-240415a06208
Assisted-by: Claude Code deepseek-v4.1-flash:cloud (analysed on a live system)

…reate

#300. Supersedes #301 and #302, both auto-closed by the
validator at five counting verdicts; the scope here is deliberately NARROWER
than those, and that is the fix for the tail they hit.

WHAT THE TAIL TAUGHT. Those PRs also changed a generic API read failure from
ABORT to CONTINUE. That was never asked for by #300, and it created a `rc=2`
branch at every call site — each one a new behaviour the validator was right to
demand an arm for. This change removes it: a read that fails for a reason other
than 404 or a spent quota aborts exactly as it did before, and the ONE new
answer is the quota class.

Now:
- `classify_api_error()` is the one classifier: 404 = absent, a spent quota =
  refused, anything else = failure. `api_absent()`, the subject read, the parent
  read and the create all call it.
- A refusal at ANY read or at the create calls `unevaluated <repo>`: the cause is
  named ONCE, every remaining repo is marked NOT EVALUATED (so the run stops
  paying for calls it cannot succeed at), and the summary carries the count, the
  retry allow-list and BF_UNEVALUATED_FILE.

Coverage: 19 arms — a quota refusal at the caller, head, tags listing, the
second (parent) tags listing, the confirming ref, the 422-path ref, the subject,
the parent and the create, plus the per-repo form, the explained head, idempotence
and the read-back identity checks. All fail on the pre-change script.

Agent: containerd-crabbox-program
Assisted-by: Claude Code deepseek-v4.1-flash:cloud (analysed on a live system)
@github-actions

Copy link
Copy Markdown
Contributor

Review — PASS

Head SHA: d5d48b653bce795b5e96633730f97c08316c9686 (branch: fix/backfill-quota-class)

Security screen: PASS — T1/T2 clean (no embedded directives; body is descriptive data); T3 no secrets/exfil/new deps/weakened guardrails (a scripts-only diff; the BF_UNEVALUATED_FILE write is a CI capture feature, not egress); T4 not security machinery.
Checklist: All applicable rules PASS or NA. Change class = org-config scripts/*.sh → offline owner-script test set (not charly runtime), so Go/ADE/SDD/DISP/ARCH bed gates are N/A with reason. Body is complete, single squash commit, attribution tier analysed on a live system is the correct ceiling (new quota paths ran only against the mock; live fleet pass informed the analysis, fully tested not claimed). Classifier/bookkeeping are single-sourced (R2), no retry/sleep (R3), rc=2 continuation deleted not left behind (R5), pre-change FAIL pasted proving non-vacuous coverage (R7/R10), and the pasted 19-arm output matches the diff's own echo strings and mock branches — internally consistent.

Non-blocking notes (tool-limited, ground rule 4): the Assisted-by model token deepseek-v4.1-flash:cloud and the contents of referenced #300 cannot be re-derived from this context; no internal contradiction found, so not carried as a finding. Cosmetic: test comment numbering skips # 15. (goes 14→16) while the suite still totals the 19 properties the body claims — no functional impact.

Verdict: PASS


ollama-cloud/deepseek-v4.1-flash — action-review.

View action run

@github-actions
github-actions Bot enabled auto-merge (squash) November 10, 2026 20:05
@github-actions
github-actions Bot merged commit 62db129 into main Nov 10, 2026
4 checks passed
charly-auto-merge Bot added a commit that referenced this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

backfill: an exhausted API quota renders as N repo names, and the not-evaluated set is only log lines

2 participants