Recovering uploads

[cbertelli]

Uploading big files on slow connections asks for a long delay, but is also prone to disconnections.
Maybe it's not possible to recover an upload, even if performed in chunks. Maybe this limit can be circumvented if not solved.

By the way, errors show that Internxt is heavily dependent on s3 OVH infrastructure. The business model works on resource brokerage with limited or no investments in hardware. But who does? It may be useful do understand who does this in Europe.

[CrispStrobe]

yes their storage nodes are presigned-URL S3 at OVH. which might make actual resuming possible... looking into that...

[CrispStrobe]

What the official clients do

The upload protocol is plain S3 multipart behind Internxt's bridge:

1. POST /v2/buckets/{id}/files/start?multiparts=N → returns one presigned S3 PUT URL per part (OVH storage) + an S3 UploadId. Multipart is only allowed for files ≥ 100 MB (InvalidMultiPartValueError in bridge usecase.ts).
2. Client PUTs each encrypted part, collecting the S3 ETag response headers.
3. POST /v2/buckets/{id}/files/finish with {index, shards: [{hash, uuid, UploadId, parts: [{PartNumber, ETag}]}]}.

That start/finish pair is the entire v2 upload API — there is no endpoint to re-issue presigned URLs for an existing UploadId.

Coping with bad connections, per client:

• drive-web: 30 MB parts, 6 parallel, no per-part retry — one failed part aborts everything, and its "retry" restarts the whole upload (max 3 attempts, retryUpload() in src/app/network/upload.ts). Its pause/resume is in-memory only; a page reload loses everything.
• inxt-js (used by the official CLI and desktop): 15 MB parts, 10 parallel, no per-part retry at all; the CLI retries the whole file twice.
• drive-mobile: the only official client with per-part retry — exactly one retry per part.

No official client can resume an interrupted upload after a disconnect or restart. They all start over from byte 0.

Why resume should be possible anyway

Three facts from the source:

1. Encryption is deterministic. Files are AES-256-CTR encrypted as one continuous stream; key and IV derive entirely from (mnemonic, bucketId, index), where index is 32 random bytes chosen at upload start and later sent in finish. Persist the index → byte-identical ciphertext on re-encryption.
2. finish has no deadline. The bridge's completeUpload() just looks up the uploads record by uuid; the Mongo UploadSchema (service-storage-models) has no TTL index and there is no time check.
3. Presigned URLs stay valid for a while. The exact TTL is set in Internxt's closed-source storage-node service; expiry shows up as HTTP 403 ("Request has expired" in drive-web's error mapping).

So a client that persists {index, uuid, UploadId, urls, completed ETags} can — after a crash, disconnect, or reboot — re-derive the same key/IV, re-encrypt locally (cheap, CPU-only), skip the parts that already have ETags, PUT only the missing ones, and call finish. Until the URLs expire.

What internxt-python does now (implemented)

Two layers, both landed:

1. In-session part repair. A part that exhausts its per-PUT retries (3, exponential backoff) no longer aborts the upload. The encrypt/hash producer streams to the end, then a repair pass re-encrypts just the failed parts (AES-CTR is seekable — the keystream can be positioned at any part offset) and re-PUTs them in 2 extra rounds spaced 15 s apart. Brief outages survive without losing anything.

2. Resume across restarts. Multipart uploads (≥ 100 MiB) checkpoint their state to upload_checkpoints/ as each part's ETag arrives. If the process dies or repair gives up, re-running the same upload command resumes it: the file is re-encrypted locally with the checkpointed index (recomputing the shard hash — no network cost), already-uploaded parts are skipped, and only the missing parts travel over the wire. Checkpoints are invalidated if the file changed (size/mtime), and on HTTP 403 (URLs expired) or a rejected finish (bridge purged the record) the CLI falls back cleanly to a fresh upload. Opt out with --no-resume.

Practical effect on a slow line: an overnight 5 GB upload that dies at 80% now costs a local re-encrypt (minutes of CPU) plus the remaining 20% of the transfer, instead of the full 5 GB again.

Caveats: files < 100 MiB are a single PUT (server forbids multipart below that), so they still restart — tolerable at that size. And resume only works within the presigned-URL lifetime, which Internxt doesn't document; the 24 h checkpoint cutoff is a guess, and expired URLs degrade gracefully to a restart.

==> can you please test if it works for you?

[cbertelli]

Thank you for the detailed explanation and for the new code. I'll try it with fair connection. For a bad one, I must rearrange the same setup as this afternoon. I will do it tomorrow. Sorry for the delay.