storage: use s3 multipart upload for files > 100 MiB#3356
Merged
Conversation
github-actions
Bot
added
the
S-waiting-on-review
syphar
force-pushed
the
multipart-upload
branch
2 times, most recently
from
f88e14e to
34614e9
Compare
Member
Author
|
cc @GuillaumeGomez do you have time for a review here? |
syphar
commented
Jun 3, 2026
GuillaumeGomez
approved these changes
Jun 3, 2026
Member
|
Looks good to me, thanks! |
github-actions
Bot
added
S-waiting-on-deploy
syphar
removed
the
S-waiting-on-deploy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
the current method we're using (
put_object) works fine most of the time, but has a limitation:it only allows files < 5 GiB.
Especially with the reduced default targets this is more of an edge case, but it is blocking me from my repackaging efforts, where I'm trying to convert old releases into archives, so I can drop the non-archive logic from the codebase. We have a handful of crates where the zipfile with the html files will become > 5 GiB.
AWS recommends using multipart uploads for anything > 100 MiB, so let's do that generally. It also enables us to do parallel uploads.
I'm also dropping our custom retry logic, in favor of the SDK internal retries that we configure too.