feat: direct upload with GC-aware addPieces batching - #71
Open
SgtPooki wants to merge 5 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
New
uploadcommand implementing #70: download source CIDs, pack multi-root CARs (default 1000 MiB, under the SDK's 1016 MiB piece cap), stream each straight to the primary provider via synapse-sdkstore(), have secondaries pull from the primary, and flush parked pieces through onecommit()per provider when the batch reaches 40, when the GC-window guess nears expiry, or when the source drains. The per-provider window estimate persists in the migration DB, is lowered on a detected GC, and never raised; the flush margin comes from commit confirmations observed in the same run. No base-fee gate — the provider submits and pays foraddPieces. Staged CARs are evicted once every copy is committed.The provider-pull ingress paths (
redirect-serve,serve --ingress,pdp-submit) stay available for self-hosting operators but now require--legacy-pull.How to verify
Calibnet runs against providers 4 and 2: a 45-piece run flushed
[batch-full]at exactly 40 then[drained]the remaining 5 on both providers; two 999.99 MiB pieces stored, pulled, and committed; CARs evicted only after both copies landed. A real GC collection cannot be forced on demand, so detection and window-lowering are covered bygc-window.test.tsanddirect-upload-flow.test.tsinstead.