Skip to content

feat: direct upload with GC-aware addPieces batching - #71

Open
SgtPooki wants to merge 5 commits into
mainfrom
feat/direct-upload
Open

feat: direct upload with GC-aware addPieces batching#71
SgtPooki wants to merge 5 commits into
mainfrom
feat/direct-upload

Conversation

@SgtPooki

@SgtPooki SgtPooki commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What changed

New upload command implementing #70: download source CIDs, pack multi-root CARs (default 1000 MiB, under the SDK's 1016 MiB piece cap), stream each straight to the primary provider via synapse-sdk store(), have secondaries pull from the primary, and flush parked pieces through one commit() per provider when the batch reaches 40, when the GC-window guess nears expiry, or when the source drains. The per-provider window estimate persists in the migration DB, is lowered on a detected GC, and never raised; the flush margin comes from commit confirmations observed in the same run. No base-fee gate — the provider submits and pays for addPieces. Staged CARs are evicted once every copy is committed.

The provider-pull ingress paths (redirect-serve, serve --ingress, pdp-submit) stay available for self-hosting operators but now require --legacy-pull.

How to verify

Calibnet runs against providers 4 and 2: a 45-piece run flushed [batch-full] at exactly 40 then [drained] the remaining 5 on both providers; two 999.99 MiB pieces stored, pulled, and committed; CARs evicted only after both copies landed. A real GC collection cannot be forced on demand, so detection and window-lowering are covered by gc-window.test.ts and direct-upload-flow.test.ts instead.

@FilOzzy FilOzzy added this to FOC Aug 3, 2026
@github-project-automation github-project-automation Bot moved this to 📌 Triage in FOC Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 📌 Triage

Development

Successfully merging this pull request may close these issues.

2 participants