From 6dbe9de60dc6fcfc17ed6386e67e101a2248fbc8 Mon Sep 17 00:00:00 2001 From: CrispStrobe Date: Mon, 29 Jun 2026 15:08:13 +0200 Subject: [PATCH] docs(PLAN): spec within-file chunk concurrency (fresh-agent-executable) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Parallel multipart part uploads (Step A) + ranged downloads (Step B), ported in intent from the filen bounded-chunk-concurrency work. File/ function anchors, the AES-CTR sequential-keystream constraint, byte-budget gating, and a unit+live test matrix — enough for a fresh agent to execute. --- PLAN.md | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/PLAN.md b/PLAN.md index c8b0d55..86927bb 100644 --- a/PLAN.md +++ b/PLAN.md @@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to: - Sync engine, file versioning, GUI, cross-account migration - Workspaces (`workspaces-list/use/unset`) + +--- + +# Performance: within-file chunk concurrency (added 2026-06-29) + +File-level (batch) concurrency ALREADY exists: `cli.py` uses a +`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/ +`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the +transfer of a SINGLE large file. Two opportunities, ported from the +filen-python / filen-dart "bounded chunk concurrency" work (read those repos' +PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first). + +Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption +is inherently sequential — like a running hash), and **S3** storage (multipart +parts on upload; a single presigned object on download). + +## Step A — parallel multipart part uploads ⬜ TODO +`services/drive.py` → `_perform_network_upload` (~line 185): the sequential +`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part +(`encryptor.update`), updates `sha`, and PUTs it via +`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`) +before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart; +smaller files are a single PUT — keep them sequential. + +Approach (mirror filen Step 1): +- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN + ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then + hands `(part_index, urls[part_index], ciphertext)` to a bounded + `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag / + `DriveService` attr). +- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size + the list and assign BY INDEX (parts finish out of order). +- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))` + before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live. +- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber, + then `api.finish_upload(...)` unchanged. + +CONSTRAINTS — do NOT just flip the loop: +1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer + (keystream + content hash are order-dependent). Only the PUT is parallel. +2. `parts_manifest` ordered by PartNumber regardless of completion order. +3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough). +4. On part failure, surface after join; let the existing checkpoint/resume path + (`create_upload_checkpoint`, `services/drive.py:898`) handle restart. + +Win is the single-large-file case (batch is already file-parallel). + +## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag) +`services/drive.py` → `download_file` (~line 1616): today one +`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL +(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to +disk. S3 presigned GETs honor HTTP `Range`. + +Approach: +- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)` + (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool. +- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or + `os.pwrite`); pre-size with `f.truncate(file_size)`. +- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is + `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic, + bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned + offset. Then each range decrypts independently. Verify the existing content-hash. + +CONSTRAINTS: +1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned. +2. Bound bytes-in-flight (`_mem_acquire`). +3. Fall back to the sequential single-stream path on 200-not-206 or small files. + +## Tests (unit hermetic + live) +Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered; +seekable-CTR decrypt reproduces plaintext; small files stay sequential. +Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up): +- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline +- [ ] interrupted multipart upload resumes (checkpoint) and completes +- [ ] ranged download of a large file is byte-exact + faster than single-stream