Stream sealed frames as upload bodies: O(part) → O(frame) memory#130
Open
ServerSideHannes wants to merge 2 commits into
Open
Stream sealed frames as upload bodies: O(part) → O(frame) memory#130ServerSideHannes wants to merge 2 commits into
ServerSideHannes wants to merge 2 commits into
Conversation
Upload paths buffered a full internal part of ciphertext before handing it to the backend (up to ~240MB for multi-GB copies; the reason pods need 1Gi). FramedStreamBody now yields sealed 8MiB AES-GCM frames (in 1MB slices) as encryption produces them, with exact Content-Length via framed_ciphertext_size and a running ciphertext MD5 for backend ETag verification. - UploadPart framed path: multi-frame internal parts stream; single-frame parts stay buffered bytes so botocore can replay them. A mid-part backend failure surfaces as a failed client part (rclone/barman re-send it). - Copy pump: internal parts stream with pump-owned retry — on failure the plaintext source is reopened at the part's offset (ranged re-read) and the client-part MD5 is committed per attempt from a copy() snapshot, so retried bytes are never double-hashed. Short sources now fail loudly instead of writing wrong sizes. - Governor formulas recalibrated (tracemalloc): streaming_upload_peak flat 4*FRAME (32MB, real ~24MB) for multi-frame parts; copy_chunk_peak flat 4*FRAME + 2*MAX_BUFFER (48MB, real ~32MB). Clamp machinery is now vestigial. Closes #128
…maining governor tests Scylla manifest sidecars overstate total_plaintext_size relative to the stored segments; the pump's strict Content-Length expectation must come from the segments' real extent or every inflated-metadata copy aborts as a short source. Also update the four remaining tests pinned to the old peak formulas.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #128. Stacked on #129 (merge that first; this PR's base is its branch and will retarget to main on merge).
What
The framed UploadPart path and the streaming-copy pump buffered a complete internal part of ciphertext before uploading it — the single reason proxy pods need 1Gi limits (fleet peaks 640–1020MiB vs the 192MB governed budget). This PR uploads internal parts as
FramedStreamBody: an async-iterable body that yields sealed 8MiB AES-GCM frames as encryption produces them, sliced into 1MB pieces so the transport never pins a whole frame.framed_ciphertext_sizevia__len__) — no chunked transfer encoding; verified against aiohttp's body handling.Retry semantics (the one behavioral change)
A streamed body can't be replayed by botocore's internal retry:
md5.copy()snapshot so retried bytes are never double-hashed. Self-healing, no client-visible change.Measured memory (tracemalloc, real handler paths)
4*FRAME)4*FRAME + 2*MAX_BUFFER)Peaks are size-independent — the clamp machinery (
*_governor_clamped_reserve) is now vestigial. Combined with #129 this makes 512Mi pod limits safe by arithmetic (next step after a watched backup cycle).Tests
test_framed_stream_body.py: frame/nonce byte-compatibility with buffered encryption, exact declared length, hash tracking, short-source flagging, single-shot no-replay, botocore param-validation surface.test_copy_internal_retry.py: mid-body failure → reopen at correct offset → roundtrip + correct client ETag (catches double-hashing), deferred-tail prefix rebuild, retry exhaustion, loud short-source failure, governor returns to zero.test_streaming_framed_upload.py: mid-part failure surfaces and records nothing; body shorter than Content-Length → 400.MockS3Clientasserts yielded bytes match declared Content-Length.Backup impact & rollout
Happy path: none — backups complete in all paths, objects are byte-identical. Only difference under backend hiccups is the client-visible part retry described above; copies get strictly better. Deploy between backup runs, watch one full daily cycle (Failed counters,
PART_PLAINTEXT_MISMATCH, memory peaks). Rollback is a plain image revert. Sequence after #129.