Skip to content

fix: stream uploads from disk instead of holding binaries in memory#9

Merged
riglar merged 1 commit into
devfrom
fix/streaming-uploads
Jun 12, 2026
Merged

fix: stream uploads from disk instead of holding binaries in memory#9
riglar merged 1 commit into
devfrom
fix/streaming-uploads

Conversation

@riglar

@riglar riglar commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Fourth deferred item from the #3 review: the upload pipeline held the entire binary in memory up to three times simultaneouslyprepareFileForUpload read it into a File (zipping .app dirs in memory first), the Supabase TUS path copied it again via Buffer.from(await file.arrayBuffer()) (ironic for a "resumable/chunked" protocol), and the Backblaze simple path made a third copy. Large iOS zips could OOM constrained CI runners.

Everything now streams from disk via a small UploadSource descriptor ({ diskPath, name, size, contentType }):

  • .app directories are zipped to a temp file through yazl's output stream (removed in a finally) instead of being buffered whole
  • SHA-256 dedup hash is computed from a read stream
  • the TUS upload feeds tus-js-client a createReadStream with uploadSize — it buffers one 6 MB chunk at a time
  • the Backblaze large path now uses the existing readFileChunk disk reads, which were dead code before (the in-memory File always shadowed them)
  • the Backblaze simple path keeps a single transient buffer — bounded, since the server only picks that strategy for small binaries, and S3 pre-signed PUTs reject chunked transfer encoding (ruling out a stream body)

Measured result

Peak RSS uploading a 500 MB binary (/usr/bin/time -l, same machine, same mock API):

maximum resident set size
before 2.18 GB
after 165 MB

Test plan

  • pnpm test 122/122 passing (covers the dedup short-circuit and the full --ignore-sha-check upload attempt)
  • pnpm build and pnpm lint clean; also verified clean under --strict so it composes with fix: enable TypeScript strict mode and type-check tests in CI #7
  • RSS measurement above exercises prepare + hash on a real 500 MB file

🤖 Generated with Claude Code

The upload pipeline materialized the whole binary up to three times:
prepareFileForUpload read it into a File (with .app dirs zipped in
memory first), the TUS path copied it again via Buffer.from(await
file.arrayBuffer()), and the Backblaze simple path made a third copy.
A 500 MB binary peaked at 2.18 GB RSS; large iOS zips could OOM CI
runners.

Everything now streams from disk via a small UploadSource descriptor
({ diskPath, name, size, contentType }):

- .app dirs are zipped to a temp file through yazl's output stream
  (cleaned up in a finally) instead of being buffered
- SHA-256 is computed from a read stream
- the Supabase TUS upload feeds tus-js-client a createReadStream with
  uploadSize — it buffers one 6 MB chunk at a time
- the Backblaze large path uses the existing readFileChunk disk reads
  (previously dead code — the in-memory File always shadowed it)
- the Backblaze simple path keeps one transient buffer, bounded by the
  server only choosing that strategy for small binaries (S3 pre-signed
  PUTs reject chunked encoding, ruling out a stream body)

Peak RSS uploading a 500 MB binary: 2.18 GB before, 165 MB after.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@riglar riglar merged commit 0af9d50 into dev Jun 12, 2026
2 checks passed
@riglar riglar deleted the fix/streaming-uploads branch June 18, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant