Skip to content

fix(developer-rust): bound the pool by free space - #95

Merged
catinspace-au merged 1 commit into
mainfrom
fix/cache-prune-free-space
Sep 24, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/cache-prune-free-space

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

The pooled Rust build cache could still fill its disk. A dedicated 512G cache volume with several concurrent agent build sessions hit 96% while both prune units exited 0.

Why

  • auto set the ceiling at a sixth of the volume (85G), under a working set of around 310G. Each nightly prune evicted live workspaces and the rebuild refilled the disk.
  • The hourly guard only pruned back to that ceiling, and about 59G landed in the hour after one run.
  • Docker's data root on the same volume (about 120G) was bounded by nothing.

What changes

  • Below rust_cache_prune_free_floor (15%, was 20%) the guard drops the least-recently-built workspaces until rust_cache_prune_free_target (25%) is free, past the ceiling if need be. New --free-target.
  • The guard runs every five minutes (OnBootSec/OnUnitActiveSec, launchd StartInterval 300). rust_cache_prune_guard_schedule is gone.
  • No run evicts a workspace a build is using. Cargo holds an exclusive flock on .cargo-build-lock (.cargo-lock in 1.91) in each profile directory for the whole build (src/cargo/core/compiler/layout.rs). The prune takes it non-blocking and holds it through the delete.
  • auto means no ceiling when the pool's st_dev differs from $HOME's. An explicit size still wins.
  • If the pool alone cannot reach the target and Docker's data root is on the same filesystem, the guard runs docker builder prune -f --filter until=72h, then docker image prune -af --filter until=168h. As the user, never sudo, never daemon.json. rust_cache_prune_docker / --no-docker switch it off.
  • hyperi-doctor no longer calls a worktree or submodule checkout "not a git checkout" (.git is a file there). That failed a doctor test in any worktree.

How it was verified

  • tools/ci/run-tests.sh and hyperi-ci check pass: 97 pytest, syntax check, shellcheck.
  • 32 new tests cover the free-target loop, lock skip, st_dev detection and docker gating against a stand-in docker. Eight mutants, one per design point, each fail a test.
  • ruff and ty clean. systemd-analyze verify clean on the rendered guard units.
  • --check on a real dedicated-volume pool detected the separate filesystem, kept a workspace a running cargo check had locked, and listed the docker commands it would run.

What a user sees

  • A guard that fires every five minutes and frees down to 25% when the disk is tight.
  • No auto pool ceiling on a dedicated cache volume.
  • Docker build cache and old unused images pruned only when the pool was not enough, and only on the same disk.

Existing hosts pick it up with ./install.sh --tags rust-cache from an updated checkout.

A dedicated 512G cache volume with several concurrent build sessions still reached 96% with both prune units exiting 0, because the `auto` ceiling (a sixth, 85G) sat under the day's working set and the hourly guard only ever pruned back to that ceiling. The guard now runs every five minutes and, below `rust_cache_prune_free_floor` (15%), evicts the least-recently-built workspaces until `rust_cache_prune_free_target` (25%) is free, past the ceiling if need be. On a pool with a filesystem of its own, `auto` now means no ceiling and free space bounds it. No run evicts a workspace whose cargo build-dir lock is held, and the prune holds that lock through the delete. If the pool alone cannot reach the target and Docker's data root is on the same filesystem, the guard runs `docker builder prune` and then `docker image prune` as the user, never with sudo (`rust_cache_prune_docker`, `--no-docker`). hyperi-doctor also stops reporting a worktree or submodule checkout as not a git checkout, since its `.git` is a file there.
@catinspace-au
catinspace-au marked this pull request as ready for review September 24, 2026 22:59
@catinspace-au
catinspace-au merged commit fd60029 into main Sep 24, 2026
18 checks passed
@catinspace-au
catinspace-au deleted the fix/cache-prune-free-space branch September 24, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant