ci: cap the self-hosted runners' cargo target dir size - #564
Merged
Conversation
The macOS runner has been running out of disk. Nothing in cargo ever
deletes a stale artifact: every dependency bump, toolchain update and
feature-flag combination adds a new set of rlibs, rmeta and incremental
fragments alongside the old ones, and the old ones stay forever. The
jobs that feed target/ check out with `clean: false` on purpose -- a
warm target dir is what keeps test-rust inside its 120-minute budget --
so unbounded growth is the price of that cache, and a ceiling is the
only thing that bounds it.
Add a `cargo-target-gc` composite action that runs cargo-sweep at the
end of every ci.yml job that compiles Rust on a persistent runner, and
wire it into test-rust, test-python, lint, security, coverage and build.
Per-run rather than a nightly cron, for the reason docker-host-gc
already documents: the runner fills during the working day, and a
nightly sweep cannot help a job that dies out of disk at 16:03. A cron
would also have to be pinned to one runner, and four of these jobs use
an unpinned `runs-on: [self-hosted]` and land on macOS unpredictably --
so pinning would miss the jobs actually doing the accumulating.
Notes on the implementation:
- `--maxsize` takes MEGABYTES when given a bare number ("Unit defaults
to MB", verified against cargo-sweep 0.8.0's own help). `--maxsize
100` is therefore 100MB, which on this workspace evicts essentially
the entire warm cache on every run -- the opposite of a cap. The
action passes `100GB` and refuses a unitless value outright, so a
later edit cannot reintroduce that silently.
- Non-fatal by construction. Under `if: always()` a failing sweep
would turn a green build red; `cargo install || true` alone does not
fix that, because the next line still fails. Every command is
guarded and the step reports via ::warning:: instead.
- cargo-sweep is pinned to 0.8.0 and installed only when absent.
>=0.7.0 is required for unit suffixes on --maxsize, >=0.8.0 for Rust
1.85+ compatibility (our MSRV is 1.90).
- Skipped on Windows inside the action. The unpinned jobs can land
there, and ci.yml already documents `shell: bash` being unreliable
on that runner; Windows is not the runner running out of space.
This is a soft cap. cargo-sweep evicts deps/, build/, .fingerprint/ and
incremental/ -- precisely the part that compounds -- but not the linked
outputs at target/<profile>/<bin>, which are hardlinked to their deps/
copy, so removing the dep entry frees nothing while the top-level name
survives. That residue is a small constant per profile, overwritten by
the next build, and does not accumulate. Expect the directory to settle
slightly above the ceiling rather than exactly at it.
~/.cargo/registry also grows without bound but is single-digit GB and
needs a different tool; cargo-sweep does not touch it.
Claude-Session: https://claude.ai/code/session_01Cqj66b1tDdvTPgL37ATMbR
The previous commit claimed the step was non-fatal and it was not. The
first CI run failed the Linting job on it, exit 101 after 40ms.
Omitting `set -e` does not disable errexit here. Actions invokes
`shell: bash` as
/usr/bin/bash --noprofile --norc -e -o pipefail {0}
so errexit is already on at the process level before the script's first
line runs. The first probe -- `cargo sweep --version` on a runner that
does not have cargo-sweep yet -- exits 101, and with errexit plus
pipefail that assignment terminated the script and failed the job. A
step whose entire purpose is to be safely ignorable was the one thing
turning a green build red.
Set `+e` explicitly, and end the script with `exit 0` so no later edit
to the reporting tail can fail the job either.
Verified by running the extracted script under the exact invocation CI
uses (`bash --noprofile --norc -e -o pipefail`) across all four paths:
cargo-sweep absent and un-installable (warns, exits 0), cargo-sweep
present with a real target dir (sweeps, exits 0), unitless maxsize
(errors, exits 0), and a target dir cargo-sweep cannot read (exits 0).
The earlier local runs missed this because they invoked the script as
plain `bash script`, without the flags Actions adds.
Claude-Session: https://claude.ai/code/session_01Cqj66b1tDdvTPgL37ATMbR
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…dir-size # Conflicts: # .github/workflows/ci.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The macOS runner has been running out of disk.
Nothing in cargo ever deletes a stale artifact. Every dependency bump, toolchain update and feature-flag combination adds a new set of
.rlibs,.rmetas and incremental fragments alongside the old ones, and the old ones stay forever. On a GitHub-hosted runner that doesn't matter — the machine is thrown away. On our self-hosted runners it compounds indefinitely, because the jobs that feedtarget/check out withclean: falseon purpose: a warm target dir is what keepstest-rustinside its 120-minute budget. Growth is the price of that cache, and a ceiling is the only thing that bounds it.Change
A
cargo-target-gccomposite action (following the existingdocker-host-gcconvention) that runscargo sweep --maxsizeat the end of every ci.yml job compiling Rust on a persistent runner —test-rust,test-python,lint,security,coverage,build.Per-run rather than a nightly cron, for the reason
docker-host-gcalready documents: the runner fills during the working day, and a nightly sweep can't help a job that dies out of disk at 16:03. A cron would also have to be pinned to one runner, and four of these jobs use an unpinnedruns-on: [self-hosted]and land on macOS unpredictably — so pinning would miss the jobs actually doing the accumulating.--maxsize 100is 100 MB, not 100 GBWorth calling out explicitly, since it inverts the intent. Verified against the pinned binary rather than from docs:
A 100MB ceiling evicts essentially the whole warm cache on every run, so every build would start cold. Confirmed end-to-end on a scratch crate: a
1MBcap cleaned 4.14 MiB out of a 4.4M target dir. The action passes100GBand refuses a unitless value outright, so a later edit can't reintroduce this silently.Other implementation notes
if: always(), a failing sweep would turn a green build red — andcargo install ... || truealone doesn't fix that, because the next line still fails. Every command is guarded; the step reports via::warning::instead.shell: bashasbash --noprofile --norc -e -o pipefail {0}, so errexit is already on before the script's first line — omittingset -edoes not disable it. The first CI run failedLintingon exactly that:cargo sweep --versionon a runner without cargo-sweep exited 101 and took the script down with it. Nowset +eexplicitly, plus a trailingexit 0.>=0.7.0is required for unit suffixes on--maxsize;>=0.8.0for Rust 1.85+ compatibility (our MSRV is 1.90).shell: bashbeing unreliable on that runner. Windows isn't the runner running out of space.du -sh targetbefore/after plusdf -h, so the trend is visible per run.test-fhirpathis left alone (Windows-pinned).Limitations
This is a soft cap. cargo-sweep evicts
deps/,build/,.fingerprint/andincremental/— precisely the part that compounds — but not the linked outputs attarget/<profile>/<bin>. Those are hardlinked to theirdeps/copy (verified: link count 2), so removing the dep entry frees nothing while the top-level name survives. That residue is a small constant per profile, overwritten by the next build, and doesn't accumulate. Expect the directory to settle slightly above 100GB rather than exactly at it.Cap sizing: the macOS runner has a 512GB disk (confirmed by @smunini), so a 100GB ceiling leaves ample headroom. It's an input (
with: maxsize:) if it ever needs changing.~/.cargo/registryalso grows without bound, but it's single-digit GB and needs a different tool — cargo-sweep doesn't touch it. Not addressed here.Testing
yaml.safe_load); the action's script passesbash -n.bash --noprofile --norc -e -o pipefail) across all four paths: cargo-sweep absent and un-installable, cargo-sweep present against a real target dir, unitless maxsize, and a target dir cargo-sweep can't read. All exit 0. (The first round of local testing used plainbash script, which is why it missed the errexit bug above.)--versionoutput confirmed to parse as expected by the version check.https://claude.ai/code/session_01Cqj66b1tDdvTPgL37ATMbR