Skip to content

ci: cap the self-hosted runners' cargo target dir size - #564

Merged
smunini merged 3 commits into
mainfrom
fix/cap-cargo-target-dir-size
Aug 18, 2026
Merged

ci: cap the self-hosted runners' cargo target dir size#564
smunini merged 3 commits into
mainfrom
fix/cap-cargo-target-dir-size

Conversation

@smunini

@smunini smunini commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Problem

The macOS runner has been running out of disk.

Nothing in cargo ever deletes a stale artifact. Every dependency bump, toolchain update and feature-flag combination adds a new set of .rlibs, .rmetas and incremental fragments alongside the old ones, and the old ones stay forever. On a GitHub-hosted runner that doesn't matter — the machine is thrown away. On our self-hosted runners it compounds indefinitely, because the jobs that feed target/ check out with clean: false on purpose: a warm target dir is what keeps test-rust inside its 120-minute budget. Growth is the price of that cache, and a ceiling is the only thing that bounds it.

Change

A cargo-target-gc composite action (following the existing docker-host-gc convention) that runs cargo sweep --maxsize at the end of every ci.yml job compiling Rust on a persistent runner — test-rust, test-python, lint, security, coverage, build.

Per-run rather than a nightly cron, for the reason docker-host-gc already documents: the runner fills during the working day, and a nightly sweep can't help a job that dies out of disk at 16:03. A cron would also have to be pinned to one runner, and four of these jobs use an unpinned runs-on: [self-hosted] and land on macOS unpredictably — so pinning would miss the jobs actually doing the accumulating.

--maxsize 100 is 100 MB, not 100 GB

Worth calling out explicitly, since it inverts the intent. Verified against the pinned binary rather than from docs:

  -m, --maxsize <MAXSIZE>
          Remove oldest artifacts from the target folder until it's smaller than MAXSIZE
          Unit defaults to MB, examples: --maxsize 500, --maxsize 10GB

A 100MB ceiling evicts essentially the whole warm cache on every run, so every build would start cold. Confirmed end-to-end on a scratch crate: a 1MB cap cleaned 4.14 MiB out of a 4.4M target dir. The action passes 100GB and refuses a unitless value outright, so a later edit can't reintroduce this silently.

Other implementation notes

  • Non-fatal by construction. Under if: always(), a failing sweep would turn a green build red — and cargo install ... || true alone doesn't fix that, because the next line still fails. Every command is guarded; the step reports via ::warning:: instead.
    • This needed a second commit to get right. Actions invokes shell: bash as bash --noprofile --norc -e -o pipefail {0}, so errexit is already on before the script's first line — omitting set -e does not disable it. The first CI run failed Linting on exactly that: cargo sweep --version on a runner without cargo-sweep exited 101 and took the script down with it. Now set +e explicitly, plus a trailing exit 0.
  • cargo-sweep pinned to 0.8.0, installed only when absent. >=0.7.0 is required for unit suffixes on --maxsize; >=0.8.0 for Rust 1.85+ compatibility (our MSRV is 1.90).
  • Skipped on Windows inside the action. The unpinned jobs can land there, and ci.yml already documents shell: bash being unreliable on that runner. Windows isn't the runner running out of space.
  • Logs du -sh target before/after plus df -h, so the trend is visible per run.
  • test-fhirpath is left alone (Windows-pinned).

Limitations

This is a soft cap. cargo-sweep evicts deps/, build/, .fingerprint/ and incremental/ — precisely the part that compounds — but not the linked outputs at target/<profile>/<bin>. Those are hardlinked to their deps/ copy (verified: link count 2), so removing the dep entry frees nothing while the top-level name survives. That residue is a small constant per profile, overwritten by the next build, and doesn't accumulate. Expect the directory to settle slightly above 100GB rather than exactly at it.

Cap sizing: the macOS runner has a 512GB disk (confirmed by @smunini), so a 100GB ceiling leaves ample headroom. It's an input (with: maxsize:) if it ever needs changing.

~/.cargo/registry also grows without bound, but it's single-digit GB and needs a different tool — cargo-sweep doesn't touch it. Not addressed here.

Testing

  • Both workflow files parse (yaml.safe_load); the action's script passes bash -n.
  • Ran the extracted script under the exact invocation CI uses (bash --noprofile --norc -e -o pipefail) across all four paths: cargo-sweep absent and un-installable, cargo-sweep present against a real target dir, unitless maxsize, and a target dir cargo-sweep can't read. All exit 0. (The first round of local testing used plain bash script, which is why it missed the errexit bug above.)
  • The pinned cargo-sweep 0.8.0 was installed and its --version output confirmed to parse as expected by the version check.

https://claude.ai/code/session_01Cqj66b1tDdvTPgL37ATMbR

The macOS runner has been running out of disk. Nothing in cargo ever
deletes a stale artifact: every dependency bump, toolchain update and
feature-flag combination adds a new set of rlibs, rmeta and incremental
fragments alongside the old ones, and the old ones stay forever. The
jobs that feed target/ check out with `clean: false` on purpose -- a
warm target dir is what keeps test-rust inside its 120-minute budget --
so unbounded growth is the price of that cache, and a ceiling is the
only thing that bounds it.

Add a `cargo-target-gc` composite action that runs cargo-sweep at the
end of every ci.yml job that compiles Rust on a persistent runner, and
wire it into test-rust, test-python, lint, security, coverage and build.

Per-run rather than a nightly cron, for the reason docker-host-gc
already documents: the runner fills during the working day, and a
nightly sweep cannot help a job that dies out of disk at 16:03. A cron
would also have to be pinned to one runner, and four of these jobs use
an unpinned `runs-on: [self-hosted]` and land on macOS unpredictably --
so pinning would miss the jobs actually doing the accumulating.

Notes on the implementation:

  - `--maxsize` takes MEGABYTES when given a bare number ("Unit defaults
    to MB", verified against cargo-sweep 0.8.0's own help). `--maxsize
    100` is therefore 100MB, which on this workspace evicts essentially
    the entire warm cache on every run -- the opposite of a cap. The
    action passes `100GB` and refuses a unitless value outright, so a
    later edit cannot reintroduce that silently.

  - Non-fatal by construction. Under `if: always()` a failing sweep
    would turn a green build red; `cargo install || true` alone does not
    fix that, because the next line still fails. Every command is
    guarded and the step reports via ::warning:: instead.

  - cargo-sweep is pinned to 0.8.0 and installed only when absent.
    >=0.7.0 is required for unit suffixes on --maxsize, >=0.8.0 for Rust
    1.85+ compatibility (our MSRV is 1.90).

  - Skipped on Windows inside the action. The unpinned jobs can land
    there, and ci.yml already documents `shell: bash` being unreliable
    on that runner; Windows is not the runner running out of space.

This is a soft cap. cargo-sweep evicts deps/, build/, .fingerprint/ and
incremental/ -- precisely the part that compounds -- but not the linked
outputs at target/<profile>/<bin>, which are hardlinked to their deps/
copy, so removing the dep entry frees nothing while the top-level name
survives. That residue is a small constant per profile, overwritten by
the next build, and does not accumulate. Expect the directory to settle
slightly above the ceiling rather than exactly at it.

~/.cargo/registry also grows without bound but is single-digit GB and
needs a different tool; cargo-sweep does not touch it.

Claude-Session: https://claude.ai/code/session_01Cqj66b1tDdvTPgL37ATMbR
The previous commit claimed the step was non-fatal and it was not. The
first CI run failed the Linting job on it, exit 101 after 40ms.

Omitting `set -e` does not disable errexit here. Actions invokes
`shell: bash` as

    /usr/bin/bash --noprofile --norc -e -o pipefail {0}

so errexit is already on at the process level before the script's first
line runs. The first probe -- `cargo sweep --version` on a runner that
does not have cargo-sweep yet -- exits 101, and with errexit plus
pipefail that assignment terminated the script and failed the job. A
step whose entire purpose is to be safely ignorable was the one thing
turning a green build red.

Set `+e` explicitly, and end the script with `exit 0` so no later edit
to the reporting tail can fail the job either.

Verified by running the extracted script under the exact invocation CI
uses (`bash --noprofile --norc -e -o pipefail`) across all four paths:
cargo-sweep absent and un-installable (warns, exits 0), cargo-sweep
present with a real target dir (sweeps, exits 0), unitless maxsize
(errors, exits 0), and a target dir cargo-sweep cannot read (exits 0).
The earlier local runs missed this because they invoked the script as
plain `bash script`, without the flags Actions adds.

Claude-Session: https://claude.ai/code/session_01Cqj66b1tDdvTPgL37ATMbR
@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…dir-size

# Conflicts:
#	.github/workflows/ci.yml
@smunini
smunini merged commit 3b46edc into main Aug 18, 2026
18 checks passed
@smunini
smunini deleted the fix/cap-cargo-target-dir-size branch August 18, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant