Skip to content

[rush] Avoid redundant downloads when local processes race for the same build cache entry#5883

Open
iclanton wants to merge 3 commits into
microsoft:mainfrom
iclanton:iclanton/build-cache-download-lock
Open

[rush] Avoid redundant downloads when local processes race for the same build cache entry#5883
iclanton wants to merge 3 commits into
microsoft:mainfrom
iclanton:iclanton/build-cache-download-lock

Conversation

@iclanton

Copy link
Copy Markdown
Member

Summary

Follow-up to #5746. When useDirectFileTransfersForBuildCache is enabled, multiple local Rush processes (e.g. parallel rush build invocations on the same machine or CI agent) restoring the same cache entry would each independently download it from the cloud cache provider, wasting bandwidth for potentially large cache entries.

Details

  • Before downloading, OperationBuildCache now attempts to acquire a LockFile scoped to the cache entry (mirroring the pattern already used for installing the package manager — see InstallHelpers.ensureLocalPackageManagerAsync), so later processes wait for the first one to finish instead of racing.
  • After acquiring the lock (or failing/timing out), it rechecks whether the final cache entry path already exists — if a peer process just finished populating it while we waited, we skip the network round-trip entirely.
  • This is strictly a best-effort optimization: if the lock cannot be acquired (for example, because it's held by a process on a different machine sharing a network cache folder, where LockFile's same-machine stale-holder detection can't see across machines), we simply fall back to downloading independently, exactly as before. Correctness never depends on the lock, since downloads always land in a uniquely-named temp file that is atomically renamed into place (from the prior fix in Add file-based transfer APIs to build cache provider interface and all cache plugins #5746).
  • The lock's resource name is derived by hashing the cache ID (sha1-hex) rather than using it directly, since cache IDs can contain characters (e.g. +, _, /) that LockFile.getLockFilePath's resource name validation rejects, depending on the configured cacheEntryNamePattern. (Rush's own recommended default pattern produces IDs containing both + and _.)

How it was tested

  • Added 3 new unit tests to OperationBuildCache.test.ts covering: lock acquired/released around a normal download, download skipped when another process already populated the entry while waiting for the lock, and fallback to independent download when the lock cannot be acquired.
  • All existing OperationBuildCache tests continue to pass (11/11).
  • Full rush build (typecheck, lint, API Extractor) passes; OperationBuildCache is @internal, so there is no public API surface change.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

iclanton and others added 2 commits July 17, 2026 20:50
…d cache entry

When useDirectFileTransfersForBuildCache is enabled, multiple local Rush
processes (e.g. parallel "rush build" invocations on the same machine or CI
agent) restoring the same cache entry would each independently download it
from the cloud cache provider.

Use LockFile to have later processes wait for the first one to finish
downloading, mirroring the pattern already used for installing the package
manager (see InstallHelpers.ensureLocalPackageManagerAsync). After acquiring
the lock (or failing/timing out), recheck whether the final cache entry path
already exists before downloading, so a process that waited can skip the
network round-trip entirely if a peer already populated it.

This is strictly a best-effort optimization: if the lock cannot be acquired
(for example, it's held by a process on a different machine sharing a
network cache folder, where LockFile's stale-holder detection cannot see
across machines) we fall back to downloading independently, exactly as
before. Correctness never depends on the lock, since downloads always land
in a uniquely-named temp file that is atomically renamed into place.

The lock's resource name is derived by hashing the cache ID (sha1-hex),
since cache IDs can contain characters (e.g. "+", "_", "/") that
LockFile.getLockFilePath's resource name validation rejects, depending on
the configured cacheEntryNamePattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8d807794-b652-4bf8-bc4b-8ee4b2039f92
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8d807794-b652-4bf8-bc4b-8ee4b2039f92
@iclanton iclanton changed the title Avoid redundant downloads when local processes race for the same build cache entry [rush] Avoid redundant downloads when local processes race for the same build cache entry Jul 18, 2026
@iclanton
iclanton enabled auto-merge (squash) July 18, 2026 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

1 participant