Repository navigation
ci: Turbo seeds vs the 10 GB Actions cache pool — measure the pool, then shrink or retire the per-sha seeds #25452
Description
Filing gate: ② ruling-directed sibling card — batch #5 item 1, letter C (#25373 comment 6366946328, maintainer 「同意」 2026-10-23T14:13:20Z). Reader: the domain:devx execution seat, dispatch step. Dedup: search_issues on objectstack-ai/objectstack, "Actions cache pool 10 GB eviction turbo seed" → 0 hits. Session: cse_01LKHDniRQSWjRJyDtacwpEd.
One-sentence problem
Every push to main writes about 12.8 GB of per-sha Turbo seeds to the Actions cache (six Test Core shard seeds of about 2.1 GB each, plus the Build Core, Type Check and other seeds), while the repository's Actions cache pool is 10 GB, so each merge can evict the seeds — and the pnpm store and Next.js caches — that the next PR run restores, and PR builds go cold for reasons the seed design cannot see.
Evidence (file lines at objectstack commit 2b7c331)
- Per-sha seed keys, saved on every push event, including cancelled runs (
if: always() && github.event_name == 'push'):ci.yml:1397-1402(Test Core, per shard),ci.yml:2242-2247,ci.yml:2749-2754(Build Core),lint.yml:5928-5930(typecheck). Keys end in${{ github.ref_name }}-${{ github.sha }}(ci.yml:606,:1786,:2087,:2320,:2529,:2974;lint.yml:5140,:5753), so no two merges share a key and nothing is overwritten — every merge adds a full set. - Cancelled push run 39923516471 saved all six Test Core shard seeds at about 2.1 GB each (the ci: the push-to-main CI run is cancelled by the next merge 9 times in 13, so the Turbo cache seed that only that run saves goes stale and every PR shard and lint lane rebuilds cold (Build Core 51 s → 4m30s, shard closures Σ 20–31 runner-min per run) #25373 card thread's falsification, comment 6364570166).
- Plain
actions/cache@v6steps (restore and post-save on every event, PRs included) sit in 12 workflow files;ci.yml:2785-2798saves the pnpm store andapps/docs/.next/cacheon PR runs too. PR-scoped entries count toward the same pool. - The Turbo remote cache is ON: on PR ci: assert the Turbo remote cache is on after Build Core, the Test Core closure and the three Type Check builds #25419 (card ci: no job asserts the Turbo remote cache is actually on — TURBO_TOKEN is withheld whenever the signing secret is empty, turbo then prints "Remote caching disabled" and builds cold with nothing red; add the assertion to Build Core and the closure builds #25374) Build Core printed "Remote caching enabled" with 76/76 remote hits, and Type Check · workspace 71/71 (
TURBO_TOKEN/TURBO_TEAM,ci.yml:749-750). For the tasks the remote cache serves, the per-sha.turbo/cacheseed in the Actions pool may be redundant weight. - Pool usage is NOT measured yet.
GET /repos/objectstack-ai/objectstack/actions/cache/usageandGET /repos/objectstack-ai/objectstack/actions/cachesare refused from a cloud seat container (egress proxy 403 on Actions cache paths, 2026-10-23T14:17Z).
Governing text
ci.yml:597-610andlint.yml:5101-5145— the seed discipline comments (main writes, PR and merge_group restore by prefix). This card changes what and how much is written, not the restore order, unless the measurement shows the seeds are redundant with the remote cache.- AGENTS.md lines 371–372 are about the local worktree cache, not CI.
- Ruling 6366946328: "Raising the pool limit is a billing item and comes back to the maintainer."
Change (in order; the dev pins what the measurement supports)
- Measure first. Read
actions/cache/usageand the 100 largest entries ofactions/caches(sorted bysize_in_bytes, desc), from a localghor a one-off read step in aworkflow_dispatchjob withactions: read. Record in the PR: pool total, the share held by*-turbo-*keys, byrefs/pull/*refs, and bypnpm-store/nextjskeys, plus the age of the oldest live Test Core seed. If the pool is well under 10 GB and seeds survive at least 5 merges, stop and say so — the card closes with the reading. - Then shrink, cheapest first, choosing from: (a) drop the
.turbo/cacheActions seed for jobs whose tasks the remote cache already serves (Build Core, Type Check), keeping the seed only where the remote cache misses; (b) save the Test Core seeds from one shard-independent job, or prune.turbo/cacheto the run's own hashes before save, to cut the per-shard size; (c) stop PR runs from saving the Next.js cache (restore-only on PRs, as the Turbo seed already is). - Never raise the pool limit in this card. If after (a)–(c) the pool is still over budget, return the billing option to the maintainer as a needs-user-decision with the measured numbers.
Acceptance
- The PR records the before-measurement (pool total and the four shares above) and the after-measurement taken after ≥ 5 merges to main.
- After: pool total ≤ 8 GB at steady state, and over the next 10 PR runs whose closure matches main, the shard "Build this shard's dependency closure" step median is ≤ 1.7 min and Build Core's build step is ≤ 2.5 min (the speed acceptance moved here from ci: the push-to-main CI run is cancelled by the next merge 9 times in 13, so the Turbo cache seed that only that run saves goes stale and every PR shard and lint lane rebuilds cold (Build Core 51 s → 4m30s, shard closures Σ 20–31 runner-min per run) #25373 by the ruling).
- Or, if step 1 shows the pool is not the cause: a comment with the reading, and the card closes
not_plannedwith the measured cause named for a follow-up card.
Risks
- Dropping a seed the remote cache does not actually cover makes those jobs colder; the dev keeps the remote-cache hit counts from the
assert-turbo-remote-cachelines (ci: assert the Turbo remote cache is on after Build Core, the Test Core closure and the three Type Check builds #25419, once landed) before and after. - Actions cache eviction is least-recently-used; a measurement taken right after a burst of merges can overstate pressure — take two readings at least an hour apart.
Work estimate
Lane domain:devx · 1 PR (.github/workflows/ci.yml, .github/workflows/lint.yml; possibly .github/workflows/*.yml for the Next.js save gate) · Clause-②: no · Metadata-protocol: none · Tier H (governed surface — the PR waits for a GOVERNED_APPROVERS approval before ready). Size M. Serial: lands after #25373's PR (same ci.yml concurrency/save region) and after #25419 (needs its remote-cache assert lines for the before/after reading).
Related: #25373 (ruled sibling), #25374 / PR #25419 (remote cache assert), #5401 (closed).
Maintainer directive (verbatim, untranslated)
同意
— the maintainer, 2026-10-23T14:13:20Z, on director batch #5 item 1 (letter C = land the staged concurrency change as a coverage fix and file a separate card for the cache pool).
Generated by Claude Code
Activity
objectstack-fleet commentedon Oct 23, 2026
Claim: PM loop round 3 · 2026-10-23T18:23Z
Session:
session_01SawdJgiuoz8deqVdM51jAE(card thread, dispatched by seat sessionsession_01LKHDniRQSWjRJyDtacwpEd)Account:
os-elon-muskBranch:
claude/project-thread-53z7nt(the harness-pinned branch for the card thread; it names no issue, so this claim is the identity record)Worktree:
objectstack-issue-25452Domain:
domain:devxSeat:
domain:devx#1(seat post #6023)File surface: .github/workflows/ci.yml, .github/workflows/lint.yml (possibly other workflow files for the Next.js save gate); Tier H (stop on breach; explain in the report)
Container & model:
M,mode:thread,model: default tier,effort: highClause-②: no
Metadata-protocol: none
Responsibility: domain:devx seat 1 (#6023) reviews; Tier H — GOVERNED_APPROVERS approval before ready
Thread-read: none
Serial constraints cleared: cleared — #25454 (concurrency, merged 18:20Z) and #25419 (remote-cache assert, merged 18:10Z) are on main; #25465 (DTS skip) also landed 18:20Z
Dispatched under the maintainer's 「同意」 to the director's ruling batch 1, item 5A (devx dispatches p1–p3). This card is also covered by the maintainer's 「同意，立卡」 for the CI performance audit cards, where the card says so. Dispatch order:
/mnt/project-files/pm-devx-seat/dispatch-25452.md.objectstack-fleet commentedon Oct 23, 2026
ACCEPT (seat review) → PR #25523 ·
domain:devxseat 1 ·session_01LKHDniRQSWjRJyDtacwpEd· 2026-10-23T18:55Zfcae6e3a53: two files.scripts/ci/prune-turbo-cache.mjsis new (+381) with a 25-case self-test, wired into Build Core..github/workflows/ci.yml(+29 / -1) adds a prune step before Test Core's "Save Turbo cache (main only)", and the save now runs only when the prune sayssave=true..turbo/cache, and the bareLinux-turbo-test-restore fallback carries that lineage across shards; PR run 39999049930 shard 1 restored shard 3's 2.50 GB seed. This is card change (b).--summarizerun summary (.turbo/runs/, whichci.ymlwrites on both the package and the slice legs), plus the closure build's summary that the ci: assert the Turbo remote cache is on after Build Core, the Test Core closure and the three Type Check builds #25419 assertion moved to$RUNNER_TEMP/turbo-remote-cache. All three files of every other entry are deleted. A file of unknown shape is kept and counted, so a turbo layout change cannot silently empty the seed. With no run summary (cancelled, or an empty shard) there is no save, and the previous seed stays newest. Every run-time path exits 0, so the shard attestation is unaffected.--self-testat this head is OK (25 cases, 5 batteries). CI is all green: 34 success, 11 skipped.actions/cache/usageand the largest entries) is pending. The maintainer has been asked to run twoghreads locally; this is not a landing blocker. The after-reading (≥ 5 merges, pool ≤ 8 GB at steady state, closure medians) is owed after landing..github/workflows/ci.yml). A GOVERNED_APPROVERS approval is the review record; hotlong and os-zhuang are requested.objectstack-fleet commentedon Oct 24, 2026
After-reading, first signal (devx seat, 2026-10-24 01:05Z; #25523 merged 00:35Z). Two push runs so far; the acceptance asks for ≥ 5.
041a120b61), Test Core 1/6:prune-turbo-cache: … kept 3 task entries named by 2 run summariesanddropped 17001 file(s) the run did not use: 2912.4 MB -> 0.8 MB on disk. It then savedLinux-turbo-test-1-main-041a120b61…. Shard 2 also saved.NOT SAVED -- no readable run summary. Those are empty shards on an affected-only push, which is the designed no-save path, and the previous seed stays newest.Still owed for the acceptance: the pool total at steady state (≤ 8 GB) and the PR shard-closure and Build Core medians over 10 PR runs, after ≥ 5 merges. The seat reads them on the next check.
Generated by Claude Code
objectstack-fleet commentedon Oct 24, 2026
Final after-reading (devx seat, 2026-10-24 08:10Z; #25523 merged 00:35Z, interim 6377899689). Seven push runs to
mainsince landing, all 42 Test Core jobs read.Seeds: met. Every saved seed is under 49 MB, and no prune step failed.
041a120b9c43e19fba9e0c37e0734559ba6912c1393769728801f2bbNOT SAVED -- no readable run summary, the designed path. No shard that ran tests skipped its save.keptline reports0 have no entry.PR timings over 10 PR runs since landing. The runs are 40072679921, 40066454701, 40065311088, 40063381603, 40062740824, 40062506129, 40062326459, 40059926372, 40058336591 and 40058057228. I skipped two
action_requiredruns, one cancelled run and two runs with Build Core skipped.Build this shard's dependency closure: met. The median over 60 shard jobs is 1.23 min (target ≤ 1.7), and the max is 6.68 min. Shards 3–5 carry the long tail at 200–400 s. The slowest shard's median per run is 4.45 min.Build packages (excluding docs): not met as read. The median is 3.28 min (target ≤ 2.5), and the max is 6.32 min.main". All ten PRs change source, and two of them are the release PR, which bumps every version.started_at−created_at) has a median of 2 s and a max of 3 s.Still open: the pool total (target ≤ 8 GB at steady state).
actions/cache/usagereturns 403 from the seat's container, and the two localghreads asked of the maintainer have not arrived. On the arithmetic, about 0.27 GB per full merge leaves the 10 GB pool more than 30 merges of headroom. But the old 2.4 GB entries age out on GitHub's clock, and only the usage read can confirm that.The seat recommends closing #25452 once the pool read is in. If the pool read is still missing, the card can close on the seed evidence above, with the pool figure recorded when it arrives. That choice is the maintainer's.
Generated by Claude Code
hotlong commentedon Oct 24, 2026
Pool reading (the two local
ghreads asked of the maintainer) · 2026-10-24 14:25ZRead by the cloud ops seat at the maintainer's request, from a local
gh:GET /actions/cache/usageand every entry ofGET /actions/caches(31 entries, sorted by size).mainwas at2c523d0100. This is reading 1; the card's risk note asks for a second at least an hour later.Pool total: 10.30 GB in 31 entries. That is over the ≤ 8 GB acceptance and at the 10 GB limit.
cache/usageanswered 11.01 GB in 30 entries; the listing taken seconds later is used below.*-turbo-*keysrefs/pull/*refspnpm-storekeysnextjskeysgh-readonly-queue/*)refs/heads/mainEviction is running. Of the 31 live entries, 29 were created in the last 1.5 h. The 2 older ones (13.4 h) are
pnpm-store-v3andcorepackonmain, which every run reads. The oldest live Test Core seed is 1.3 h old (Linux-turbo-test-6-main-49653f91…). Live Test Core seeds cover 3mainshas, short of the card's "survive at least 5 merges".What #25523 fixed holds. The 7 live Test Core seeds total 0.27 GB, 0.8–48 MB each.
Where the weight is now:
mainrefs alike:ci.yml:2918-2923is a plainactions/cache@v6, which post-saves on every event.main: 2.79 GB.turbo-build-core571 MB,turbo-pack-smoke2 × 557 MB,turbo-typecheck2 × 555 MB.Linux-pnpm-store-v3-d2b76958…is stored 5 times (4 merge-queue refs and 1 PR ref), about 1.47 GB of duplicates, plusspec-shape-gate's own 489 MB store.pnpm-store-v3step is a plainactions/cache@v6in about 12 places acrossci.ymlandlint.yml.gh-readonly-queue/main/pr-*ref is used by its one run.main's seeds out.Against the acceptance:
Per the card's step 3, the billing option comes back to the maintainer only after (a)–(c). (a) and (c) have not been done, and the per-ref pnpm/Next.js saves on
merge_groupand PR events look like the cheapest next cut.Nothing was changed; both calls are reads.
13 remaining items
objectstack-fleet commentedon Oct 28, 2026
devx seat 2 · premise correction · 2026-10-28T00:09Z
Claim 6455607148 listed a "plain
pnpm-store-v3cache" atci.ymlL3219/3311/3355/3613/3852/4174 andgoverned-surface-guard.ymlL247 as undone work for change (c). That was wrong. The seat grepped the cache keys, not theuses:lines. Every one of those steps is alreadyactions/cache/restore@v6onorigin/main(re-read now: ci.yml L3216/3352/3610/3849/4171/4421uses: actions/cache/restore@v6), andlint.ymlroutes every job throughsetup-workspace. The dev measured this in os-dev report 6456906843. So the only step of (c) left was the Next.js cache, which PR #26984 makes restore-only onpull_request/merge_group. The pnpm half of the directive was already done (setup-workspacevia #26083, and the ui jobs' restore-only steps).PR #26984 is in review (a contract review at
CONTRACT_REVIEW_TIERis under way; Tier H). Out of scope, carried to step 3's pool re-reading:os-create-smoke.yml:95andpack-smoke-optin.yml:125post-save a per-sha.turbo/cacheonpull_request, outside directive (c), which names Next.js and pnpm only.Generated by Claude Code
objectstack-fleet commentedon Oct 28, 2026
ACCEPT → PR #26984 comment 6457218903 · 2026-10-28T00:20Z · contract review PASS 6457159321 · os-zhuang approved and armed auto-merge; CI on head
4597816157: 48 success, 11 skipped, 0 failure.Generated by Claude Code
objectstack-fleet commentedon Oct 28, 2026
Landed (partial): step 1, change (c), via PR #26984, merged as
eae9a588e7· 2026-10-28T00:48ZRelease:
session_01QSeKYmTYfaPbLQD4BJfomW(domain:devx#2) · partial landing · landed: the Next.js cache restores on every run and saves only onpush/schedule. The pnpm half was already restore-only onmain(correction 6457011853). · remaining: step 2, change (a), now decision card #26957; step 3, the pool re-reading after ≥ 5 merges, which needs a reader withactions/cache/*access because the seat container gets 403 · state:pm:dispatched→pm:blocked, assignee clearedBlocked-by: #26957
The merged diff matches the reviewed head
4597816157(sorted +/− lines ofci.ymlhash-identical,merge-base..headagainsteae9a588e7^1..eae9a588e7). ACCEPT → PR #26984 comment 6457218903; contract review PASS 6457159321; os-zhuang approved.Restart: when #26957 is ruled, its letter decides step 2 (A: prune the build-core / typecheck / pack-smoke seeds; B: drop them; C: no change until the pool reading). Either way, step 3's pool reading also carries the two per-sha
.turbo/cachepost-saves onpull_request(os-create-smoke.yml:95,pack-smoke-optin.yml:125) noted in the dev's report.Generated by Claude Code
objectstack-fleet commentedon Oct 28, 2026
Pointer for the devx seat · director seat · 2026-10-28T02:11Z
Step 2 of the directive 6385486517 (「再去掉远程缓存已覆盖的 build-core/typecheck/pack-smoke 种子」) is rewritten by ruling #26957 → A (「同意」 to batch #416): prune, do not drop. The three seeds stay; each seed-save step is wired behind
scripts/ci/prune-turbo-cache.mjs --site …as Test Core and the UI jobs are. Its premise fell with ruling #26070 → A (6417142465): PR runs carry no remote-cache token, so these seeds are the only cache a PR build has. Step 3 (the pool reading after ≥ 5 merges) and "⛔ The pool limit is not raised here" stand. Dispatch #26957 as this card's step 2; change (c) is unaffected. ⛔ Not a claim.objectstack-fleet commentedon Oct 28, 2026
Dispatch note · devx seat 2 · 2026-10-28T03:12Z
Step 2 is ruled (#26957 → A, prune not drop, 6458731814) and dispatched on #26957 (claim 6459608872, cloud session
session_01QSeKYooUn7D13pPguPf727). This card stayspm:blockedon #26957 until that PR lands; then step 3, the pool reading, remains.Generated by Claude Code
objectstack-fleet commentedon Oct 28, 2026
Step 2 landed (prune, not drop; ruling 6458731814) · devx seat 2 · 2026-10-28T06:52Z
PR #27067 merged as
6817bf1eae. The first post-prune seed sizes are in #26957 comment 6462677575:turbo-typecheck: 5,885.7 MB → 33.3 MB on disk.turbo-build-core: 55.4 MB, unchanged; it was already clean.turbo-pack-smoke: 148.7 MB → 54.9 MB.Step 3, the pool reading after ≥ 5 merges, is next. It needs an account that can read
actions/cache/*; the seat container gets 403 there. This card stayspm:blockedon that reading.Generated by Claude Code
objectstack-fleet commentedon Oct 29, 2026
triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U
Triage: unblocked ·
pm:blocked→pm:queue· 2026-10-29T03:47Z⛔ Not a claim.
6817bf1e;turbo-typecheckwent from 5,885.7 MB to 33.3 MB), and [Decision] #25452 change (a) assumed the Turbo remote cache serves PR builds, but ruling #26070 → A keeps PRs on the Actions-cache seeds alone: drop, prune or keep the build-core / typecheck / pack-smoke seeds? #26957 is closed. Only step 3, the pool reading, remains..github/workflows/**).Generated by Claude Code
objectstack-fleet commentedon Oct 29, 2026
domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW
Maintainer-action: read the Actions cache pool for objectstack-ai/objectstack —
gh api repos/objectstack-ai/objectstack/actions/cache/usageandgh api --paginate 'repos/objectstack-ai/objectstack/actions/caches?per_page=100' --jq '.actions_caches[] | [.key, .size_in_bytes, .last_accessed_at] | @tsv'— and paste both outputs on this card — done when a comment on this card carries both outputs, read at or after6817bf1eaeplus ≥ 5 merges tomainWhy it is parked with the maintainer · 2026-10-29T21:22Z · ⛔ not a claim
Triage's unblock 6487355419 leaves step 3, the pool reading, as the only remaining step. The lane seat cannot take that reading: re-probed now,
GET /repos/objectstack-ai/objectstack/actions/cache/usageandGET …/actions/cachesboth answer 403 to this seat's token. The relay has no cache-read operation either. Nothing else on the card is open, so dispatching a dev would not help, because the dev's container has the same token scope.The reading decides the card's end (see its Acceptance section). Either the seeds still crowd the 10 GB pool, which means a shrink/retire step; or they no longer do, and the card closes
not_plannedwith the measured cause named. Since the step-2 prune,turbo-typecheckis down from 5,885.7 MB to 33.3 MB (6462691006). Once a reading is posted, any lane seat acts on it.Generated by Claude Code