Add file-based transfer APIs to build cache provider interface and all cache plugins

[Copilot]

Summary

Adds optional file-based transfer APIs to the build cache provider interface (ICloudBuildCacheProvider) and implements them across all cache plugins (HTTP, Amazon S3, Azure Blob Storage). When enabled via the useDirectFileTransfersForBuildCache experiment flag, cache entries are transferred directly between local files and cloud storage without buffering entire contents in memory, preventing out-of-memory errors for large build outputs.

Details

Core changes:

• ICloudBuildCacheProvider gains two optional methods: tryDownloadCacheEntryToFileAsync and tryUploadCacheEntryFromFileAsync. Providers that don't implement them gracefully fall back to the existing buffer-based APIs.
• OperationBuildCache conditionally uses the file-based path when useDirectFileTransfersForBuildCache is enabled and the provider supports it. Includes cleanup of partial files on failed downloads.
• WebClient is refactored to extract a shared _makeRawRequestAsync core used by both buffer and streaming request paths, with a new fetchStreamAsync method and Content-Encoding decompression support for streaming responses.
• FileSystem in node-core-library gains createReadStream, createWriteStream, and createWriteStreamAsync methods (wrapped in _wrapException for consistent error handling).
• FileSystemBuildCacheProvider is simplified — the stream method is removed since cloud providers now handle file I/O directly.

Plugin implementations:

• HTTP plugin: Downloads via fetchStreamAsync → pipeline() to file. Uploads via createReadStream → fetchStreamAsync. Uses maxAttempts: 1 for uploads (stream consumed after first attempt), with credential fallback skipped for stream bodies.
• Amazon S3 plugin: Downloads via streaming GET with retry support. Uploads hash the tarball on disk first via _hashFileAsync, then stream with the SHA-256 hash included in the AWS Signature V4 request — restoring full payload signing (no more UNSIGNED_PAYLOAD). No retry on uploads.
• Azure Blob Storage plugin: Downloads via SDK-native blobClient.downloadToFile(). Uploads via blockBlobClient.uploadFile(). Parent directory creation ensured before download.

Gating:

• Controlled by useDirectFileTransfersForBuildCache in experiments.json. Defaults to off. Falls back to buffer-based APIs if the cloud provider plugin doesn't implement the file-based methods.

How it was tested

• Unit tests for HttpBuildCacheProvider (14 tests): buffer and file-based GET/SET, 404 cache miss, credential fallback skip for file uploads, write-not-allowed checks, retry behavior, pipeline assertions
• Unit tests for AmazonS3Client (38 tests): buffer and file-based GET/SET, signed payload hash verification (not UNSIGNED-PAYLOAD), download retry on transient 5xx, no-retry on upload, credential validation, pipeline assertions
• All existing tests continue to pass (51 S3 total, 14 HTTP total)
• Verified backwards compatibility: when the experiment flag is off (default), all code paths are identical to the pre-change behavior

Impacted documentation

• experiments.schema.json updated with useDirectFileTransfersForBuildCache description
• common/reviews/api/rush-lib.api.md updated with new API surface
• common/reviews/api/node-core-library.api.md updated with new FileSystem stream methods

[iclanton]

@copilot - add APIs to FileSystem in node-core-library for the fs APIs you're making use of in this PR. Ideally, none of our projects (except for FileSystem itself) should use fs. Everything should use FileSystem. The change to node-core-library will also need a changefile. Take a look at the conventions in FileSystem and follow that pattern.

[Copilot]

@copilot - add APIs to FileSystem in node-core-library for the fs APIs you're making use of in this PR. Ideally, none of our projects (except for FileSystem itself) should use fs. Everything...

Added createReadStream() and createWriteStream() static methods to FileSystem in node-core-library, along with FileSystemReadStream and FileSystemWriteStream type aliases (following the same pattern as FileSystemStats and FolderItem). Updated FileSystemBuildCacheProvider and OperationBuildCache to use the new FileSystem APIs instead of importing from node:fs directly. Change file for @rushstack/node-core-library included as a minor bump. Commit: 468caf7

[iclanton]

@copilot - are there async variants of those fs APIs? Should we expose and use those too?

[Copilot]

@copilot - are there async variants of those fs APIs? Should we expose and use those too?

No, fs.createReadStream() and fs.createWriteStream() don't have async variants — they're already inherently asynchronous. They return stream objects immediately and all the actual I/O happens asynchronously through the stream events/pipeline. The fs/promises module doesn't include them for this reason.

[iclanton]

@copilot - take a look at the other FileSystem APIs. Many have an optional options parameter with useful utilities. For example, the write operations have an option to create the parent folder if it doesn't exist. Consider adding useful options to these new APIs

[Copilot]

@copilot - take a look at the other FileSystem APIs. Many have an optional options parameter with useful utilities. For example, the write operations have an option to create the parent folder if it...

Added IFileSystemCreateWriteStreamOptions with ensureFolderExists to createWriteStream(), following the same pattern as IFileSystemWriteBinaryFileOptions and other write APIs. Updated FileSystemBuildCacheProvider to use the new option instead of manual ensureFolderAsync(). For createReadStream, there aren't useful options from the existing FileSystem conventions (read methods use encoding/convertLineEndings which don't apply to raw streams). Commit: 91161d8

[iclanton]

@copilot - can we add stream support to the other cache plugins?

[Copilot]

@copilot - can we add stream support to the other cache plugins?

Added streaming support to both remaining cache plugins:

Amazon S3 plugin (733ef5c):

• Refactored AmazonS3Client._makeRequestAsync to extract signing into _buildSignedRequest
• Added _makeStreamRequestAsync using WebClient.fetchStreamAsync() with UNSIGNED-PAYLOAD for AWS Sig V4 signing of streaming uploads
• Added getObjectStreamAsync() (with retry support) and uploadObjectStreamAsync() (single attempt, since consumed streams can't be replayed)
• Added tryGetCacheEntryStreamByIdAsync and trySetCacheEntryStreamAsync to AmazonS3BuildCacheProvider

Azure Storage plugin (733ef5c):

• Added tryGetCacheEntryStreamByIdAsync using blobClient.download() with readableStreamBody
• Added trySetCacheEntryStreamAsync using blockBlobClient.uploadStream()
• Added _logBlobError helper for consistent error handling across buffer and stream methods

All three cache plugins (HTTP, S3, Azure) now implement both streaming methods. All existing tests pass (45 S3, 10 Azure).

[iclanton]

@copilot - Can you put usage of this functionality behind a Rush experiment?

[Copilot]

@copilot - Can you put usage of this functionality behind a Rush experiment?

Added a useStreamingBuildCache experiment flag in experiments.json. The streaming code paths in OperationBuildCache now only activate when both the experiment is enabled AND the cloud provider implements the streaming methods. Without the experiment enabled, all cache operations use the existing buffer-based approach. Commit: 2493ed8

[iclanton]

@copilot - This change introduces a lot of duplicated code. I've called out some instances of it. Can you review with that in mind and clean that up?

[Copilot]

Pull request overview

This PR adds optional file-based transfer APIs to the cloud build cache provider interface and wires them through Rush’s build cache flow, enabling streaming uploads/downloads to avoid buffering large cache entries in memory (gated behind the useDirectFileTransfersForBuildCache experiment).

Changes:

• Added optional tryDownloadCacheEntryToFileAsync / tryUploadCacheEntryFromFileAsync to ICloudBuildCacheProvider and integrated conditional usage into OperationBuildCache.
• Refactored WebClient to support streaming responses (fetchStreamAsync) with shared request core and Content-Encoding decompression for streams.
• Implemented/updated providers and tests for HTTP, Amazon S3, and Azure Blob storage to support the new file-based path.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
rush-plugins/rush-http-build-cache-plugin/src/test/HttpBuildCacheProvider.test.ts	Expands unit coverage for buffer vs file-based HTTP cache transfers and retry/credential behaviors.
rush-plugins/rush-http-build-cache-plugin/src/HttpBuildCacheProvider.ts	Adds file-based download/upload APIs and introduces stream-based request path via WebClient.fetchStreamAsync.
rush-plugins/rush-bridge-cache-plugin/src/BridgeCachePlugin.ts	Plumbs the experiment flag into cache plugin options.
rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageBuildCacheProvider.ts	Adds Azure SDK-native file-based download/upload implementations and shared helpers.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/test/AmazonS3Client.test.ts	Adds coverage for file-based GET/PUT behavior, retry expectations, and payload signing hash.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/test/snapshots/AmazonS3Client.test.ts.snap	Snapshot update for added/modified tests.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3Client.ts	Adds streaming file transfers and reinstates payload signing via SHA-256 hash for file uploads.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3BuildCacheProvider.ts	Adds file-based provider APIs and centralizes object-name computation.
libraries/rush-lib/src/utilities/WebClient.ts	Introduces streaming response API and shared raw request core; adds stream decompression support.
libraries/rush-lib/src/schemas/experiments.schema.json	Adds schema entry for useDirectFileTransfersForBuildCache.
libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts	Threads experiment option through cache plugin initialization.
libraries/rush-lib/src/logic/buildCache/test/OperationBuildCache.test.ts	Updates test setup to include the new option.
libraries/rush-lib/src/logic/buildCache/OperationBuildCache.ts	Uses file-based download/upload when enabled and supported, with cleanup of partial downloads.
libraries/rush-lib/src/logic/buildCache/ICloudBuildCacheProvider.ts	Extends interface with optional file-based transfer methods.
libraries/rush-lib/src/logic/buildCache/FileSystemBuildCacheProvider.ts	Refactors local cache path generation and existence checks.
libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts	Plumbs experiments configuration into cache/skip plugin initialization.
libraries/rush-lib/src/api/ExperimentsConfiguration.ts	Adds typed config surface for the new experiment.
common/reviews/api/rush-lib.api.md	Updates API report for new interface and option surface.
common/reviews/api/rush-amazon-s3-build-cache-plugin.api.md	Updates API report for new S3 client methods.
common/changes/@microsoft/rush/copilot-stream-cache-entry-for-http-plugin_2026-04-05-03-56.json	Adds change entry describing the new APIs and gating.

[iclanton]

@copilot - There are merge conflicts now. Can you rebase this branch to resolve them?

[Copilot]

@copilot - There are merge conflicts now. Can you rebase this branch to resolve them?

Resolved the branch conflicts and pushed the update in 31b74e9.

[iclanton]

@copilot - Can you look at the unresolved comments?

[Copilot]

@copilot - Can you look at the unresolved comments?

Addressed the actionable unresolved items in 6867354: streamed request bodies are destroyed on request errors, S3 direct-file uploads now send Content-Length, buffered S3 GET failures preserve the response body in the error, and the file-download interface docs now match the actual false-return contract. Targeted tests passed for @microsoft/rush-lib and @rushstack/rush-amazon-s3-build-cache-plugin.