Skip to content

perf(file_storage): off-loop S3 spool I/O, one client per multipart upload, browse index - #837

Merged
antosubash merged 8 commits into
mainfrom
fix/s13-perf-file-storage
Nov 5, 2026
Merged

antosubash merged 8 commits into
mainfrom
fix/s13-perf-file-storage

Conversation

@antosubash

Copy link
Copy Markdown
Owner

Performance fix for the file_storage S3 upload path and the file browse page (local performance review, 2026-11-05).

  • Spool I/O off the event loop. Server-side S3 uploads buffer through a SpooledTemporaryFile. Work stays inline while the buffer is in memory, and moves to a worker thread once it rolls to disk or a write would push it past 2 MiB.
    • Offloaded calls are shielded, so cancellation waits for in-flight I/O before the file is closed.
  • One S3 client per multipart upload, instead of one per call.
    • Create, complete and abort now share private client-taking helpers on S3MultipartMixin, so the public methods and the upload path cannot drift. Complete sorts the parts; abort tolerates NoSuchUpload.
  • Browse index. New (tenant_id, is_deleted, created_at DESC, id) index for the default newest-first file list, added in migration f6c2d4e8b109.

Measured locally against a fake S3, with no network. Method and caveats are in docs/perf/2026-11-05-file-storage.md.

  • Event-loop lag p99 during 32–64 MiB uploads went from roughly 29–55 ms to 11–14 ms.
  • Elapsed upload time is within about 10–35% of main.
  • A 16 MiB multipart upload opens 2 clients instead of 10.
  • The browse first page over 100k rows on SQLite went from 38 ms to 0.08 ms.

Pipeline:

  • /optimize ran once and produced 1 commit.
  • A performance pass restored throughput after the first version offloaded every chunk (2–6x slower).
  • Two review rounds fixed three things:
    • _io could spin when its inner task was cancelled;
    • an empty tail chunk;
    • the multipart helper drift described above.

Local CI:

  • make lint ✓, make test-js ✓ and make migrations-roundtrip-pg ✓ (on a fresh database).
  • make test-py (8 workers): all passed except test_clean_target_is_not_in_repo, the known local /tmp/.git placeholder failure.

Note: f6c2d4e8b109, #832's e805a1b2c3d4 and #833's c5d8e2f1a903 all revise e731a1b2c3d4. Whichever merges later re-points its down_revision.

https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-11-05T17:10:13.531779Z cb78e1c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Deploying simple-module-python with  Cloudflare Pages  Cloudflare Pages

Latest commit: d403d4f
Status: ✅  Deploy successful!
Preview URL: https://34dec9eb.simple-module-python.pages.dev
Branch Preview URL: https://fix-s13-perf-file-storage.simple-module-python.pages.dev

View logs

# Conflicts:
#	CHANGELOG.md
#	docs/database/migrations.md
#	host/tests/test_migration_graph.py
@antosubash
antosubash merged commit 876dbaa into main Nov 5, 2026
17 checks passed
@antosubash
antosubash deleted the fix/s13-perf-file-storage branch November 5, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant