Skip to content

Stream sealed frames as upload bodies: O(part) → O(frame) memory#130

Open
ServerSideHannes wants to merge 2 commits into
feat/backend-unsigned-payload-etag-verifyfrom
feat/frame-streaming-bodies
Open

Stream sealed frames as upload bodies: O(part) → O(frame) memory#130
ServerSideHannes wants to merge 2 commits into
feat/backend-unsigned-payload-etag-verifyfrom
feat/frame-streaming-bodies

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Closes #128. Stacked on #129 (merge that first; this PR's base is its branch and will retarget to main on merge).

What

The framed UploadPart path and the streaming-copy pump buffered a complete internal part of ciphertext before uploading it — the single reason proxy pods need 1Gi limits (fleet peaks 640–1020MiB vs the 192MB governed budget). This PR uploads internal parts as FramedStreamBody: an async-iterable body that yields sealed 8MiB AES-GCM frames as encryption produces them, sliced into 1MB pieces so the transport never pins a whole frame.

  • Content-Length is exact (framed_ciphertext_size via __len__) — no chunked transfer encoding; verified against aiohttp's body handling.
  • Ciphertext MD5 accumulates in-stream and feeds the Disable backend payload signing; verify ciphertext MD5 against backend ETag #129 backend-ETag verification — integrity coverage is unchanged.
  • No format changes: same frames, nonces, part numbering, sidecar metadata. Streamed output is byte-identical to the old buffered encryption (pinned by test).

Retry semantics (the one behavioral change)

A streamed body can't be replayed by botocore's internal retry:

  • Copies (where the ~500MB peaks lived): the pump owns retry — on a mid-part failure it reopens the plaintext source at the failed part's offset (ranged re-read) and re-streams, up to 3 attempts. The client-part MD5 is committed per attempt from an md5.copy() snapshot so retried bytes are never double-hashed. Self-healing, no client-visible change.
  • Uploads: multi-frame internal parts stream; a mid-part backend hiccup surfaces as a failed client part that rclone/barman re-send (the retry behavior they already exhibit). Single-frame parts (≤8MiB, e.g. all ClickHouse parts) stay buffered bytes and keep botocore's invisible retry.
  • Short sources / short request bodies now fail loudly (400 / retryable 500) instead of ever writing an object with wrong sizes.

Measured memory (tracemalloc, real handler paths)

path before after reservation
UploadPart, 512MB–1GB client part ~56–417MB ~24MB 32MB (flat 4*FRAME)
Copy pump, any object size up to ~500MB+ ~32MB 48MB (flat 4*FRAME + 2*MAX_BUFFER)

Peaks are size-independent — the clamp machinery (*_governor_clamped_reserve) is now vestigial. Combined with #129 this makes 512Mi pod limits safe by arithmetic (next step after a watched backup cycle).

Tests

  • test_framed_stream_body.py: frame/nonce byte-compatibility with buffered encryption, exact declared length, hash tracking, short-source flagging, single-shot no-replay, botocore param-validation surface.
  • test_copy_internal_retry.py: mid-body failure → reopen at correct offset → roundtrip + correct client ETag (catches double-hashing), deferred-tail prefix rebuild, retry exhaustion, loud short-source failure, governor returns to zero.
  • test_streaming_framed_upload.py: mid-part failure surfaces and records nothing; body shorter than Content-Length → 400.
  • Governor/memory suites recalibrated to the new formulas; reservation-vs-real tracemalloc tests now also assert peaks stay O(frame) (regression to whole-part buffering fails them).
  • Test fakes/mocks updated to consume streaming bodies chunk-wise like the real transport, and MockS3Client asserts yielded bytes match declared Content-Length.

Backup impact & rollout

Happy path: none — backups complete in all paths, objects are byte-identical. Only difference under backend hiccups is the client-visible part retry described above; copies get strictly better. Deploy between backup runs, watch one full daily cycle (Failed counters, PART_PLAINTEXT_MISMATCH, memory peaks). Rollback is a plain image revert. Sequence after #129.

Upload paths buffered a full internal part of ciphertext before handing it to
the backend (up to ~240MB for multi-GB copies; the reason pods need 1Gi).
FramedStreamBody now yields sealed 8MiB AES-GCM frames (in 1MB slices) as
encryption produces them, with exact Content-Length via framed_ciphertext_size
and a running ciphertext MD5 for backend ETag verification.

- UploadPart framed path: multi-frame internal parts stream; single-frame
  parts stay buffered bytes so botocore can replay them. A mid-part backend
  failure surfaces as a failed client part (rclone/barman re-send it).
- Copy pump: internal parts stream with pump-owned retry — on failure the
  plaintext source is reopened at the part's offset (ranged re-read) and the
  client-part MD5 is committed per attempt from a copy() snapshot, so retried
  bytes are never double-hashed. Short sources now fail loudly instead of
  writing wrong sizes.
- Governor formulas recalibrated (tracemalloc): streaming_upload_peak flat
  4*FRAME (32MB, real ~24MB) for multi-frame parts; copy_chunk_peak flat
  4*FRAME + 2*MAX_BUFFER (48MB, real ~32MB). Clamp machinery is now vestigial.

Closes #128
…maining governor tests

Scylla manifest sidecars overstate total_plaintext_size relative to the stored
segments; the pump's strict Content-Length expectation must come from the
segments' real extent or every inflated-metadata copy aborts as a short source.
Also update the four remaining tests pinned to the old peak formulas.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant