feat(storage): S3-compatible object storage on Compute + Postgres by wmadden-electric · Pull Request #69 · prisma/compose

wmadden-electric · 2026-07-13T12:48:41Z

`storage` — S3-compatible object storage on Compute + Postgres

An emulated object-storage module built from framework primitives: a Compute
service speaking the S3 wire protocol, backed by a module-provisioned
Prisma Postgres (bytea), with SigV4 credentials minted at deploy. Ships
as @prisma/compose-prisma-cloud/storage, mirroring the cron shared module.

The contract is the S3 wire protocol itself, so the backing (Postgres now, a
native primitive later) swaps without consumer changes. A consumer declares an
s3() dependency, load()s a typed S3Config, and builds its own
@aws-sdk/client-s3 — the module never leaks its implementation.

Wire scope (the contract)

See CONTRACT-SCOPE.md
and the README:

PutObject, GetObject (inclusive-end Range → 206), HeadObject,
DeleteObject (idempotent), ListObjectsV2 (prefix + continuation-token),
and presigned GET/PUT.
Path-style addressing, SigV4 (AWS4-HMAC-SHA256, signed-payload sha256 or
UNSIGNED-PAYLOAD, any region), opaque quoted-sha256 ETags, correct 404s.
Out of scope by design: multipart, copy, batch delete, conditional requests,
flexible checksums / aws-chunked (rejected 501 — clients set
requestChecksumCalculation: 'WHEN_REQUIRED'), bucket CRUD, ACLs, versioning.

Design

Envelope: objects up to ~16 MiB; single-row bytea; ranged reads via SQL
substring (no full-object detoast).
Credentials minted once at deploy by an s3-credentials resource
(idempotent via Alchemy state), flowing through dependency edges to the
service and the exposed store contract; consumers get the four-field
S3Config binding (url, bucket, accessKeyId, secretAccessKey) and
build their own client (ADR-0015). Creds are delivered via the binding and
never leave the deployment — an external outputs-read path is a recorded
platform ask.
Layering: nothing Prisma-Cloud-specific leaks into @prisma/compose; the
module composes compute + postgres behind a typed boundary
(ADR-0013/0015/0016). Two new pack primitives: the s3-credentials resource
and an s3-store service type whose lowering extends compute's deploy
outputs with the bucket + minted creds.

Verification

✅ SigV4 engine + six ops — unit-proven against @aws-sdk/client-s3-signed
requests and streams-server's own signer shape.
✅ Local end-to-end against real Postgres (16 MiB round-trip, ranged reads,
list pagination) — DoD: runs locally as the dev stand-in.
✅ Deployed to real Prisma Cloud — an @aws-sdk/client-s3 smoke passed
against the deployed store for every in-scope op (PUT/GET/ranged-GET/HEAD/
DELETE/LIST/presign), then destroyed clean.
✅ examples/storage — a clean example app (a blob store/serve service)
that consumes the module via s3() and deploys to Prisma Cloud; a live
store→retrieve round-trip verified against the deployed app.
✅ README — contract scope, envelope, wiring example, local-dev usage.

Note: this PR proves the storage module. The brief originally named
streams-server as the acceptance consumer; that end-to-end proof was descoped in
favour of the plain example app above (streams-server remains a valid future
check).

Tracker: TML-3019.

🤖 Generated with Claude Code

D1 of S5 (storage module): the S3-compatible resource-kind contract that every later dispatch (protocol engine, Postgres store, module, lowerings) implements against, mirroring postgresContract/postgres() exactly. Also commits CONTRACT-SCOPE.md, the wire-protocol scope doc that doubles as the platform ask for a native S3-compatible primitive. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

D2 (part 1 of the protocol engine). `sigv4.ts` verifies AWS4-HMAC-SHA256 in both the Authorization-header and presigned-query forms: it reconstructs the canonical request, derives the signing key, and compares in constant time. The payload hash is taken from what the client signed (x-amz-content-sha256, real or UNSIGNED-PAYLOAD; UNSIGNED-PAYLOAD for presign), so both signed-payload and unsigned-streaming puts verify without re-hashing the body. Region/service come from the credential scope (any region — streams signs `auto`). `store.ts` is the minimal pluggable ObjectStore the D3 Postgres store will implement. Neither is re-exported from the authoring barrel — this is runtime engine code (node:crypto) kept out of the consumer bundle. Tests prove the streams-server signer shape (a faithful replica of its R2 client's canonical-request construction verifies; tamper and wrong-secret reject) and real aws-sdk presigned URLs (GET/PUT verify; expired and tampered reject). aws-sdk added as a devDependency only. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

D2 (part 2). `handler.ts` is the pure Request→Response engine: SigV4 verification first (403 on failure), then path-style routing where the bucket in the path is the store namespace (any name accepted). It serves the six ops — PUT (200 + ETag), GET (200/206 with Range parsing, inclusive and open-ended, + Content-Range), HEAD, DELETE (204, idempotent), and ListObjectsV2 as S3 XML that both aws-sdk and streams' regex parser read. Content-type defaults to application/octet-stream. Not re-exported from the authoring barrel — runtime engine code. `memory-store.ts` is the in-memory ObjectStore reference (sha256 ETags, ranged reads, prefix/token/maxKeys pagination) the D3 Postgres store will match; test-only. Op tests drive a real @aws-sdk/client-s3 over a Bun.serve wrapping the handler — the true wire path, no mocking — covering every op, ranged and open-ended GET, list pagination across pages, 404s, delete idempotency, and presigned GET/PUT round-trips. A store-level suite pins the ObjectStore contract for D3. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

A flexible-checksum PUT frames the body as aws-chunked (content-encoding: aws-chunked, x-amz-content-sha256: STREAMING-…-TRAILER). The seed signature verifies — the verifier reads that sentinel as the payload hash, which is correct — so handlePut would otherwise store the chunk framing as the object bytes and a later GET would return garbage. Decoding aws-chunked is out of scope (checksums are a non-goal), but silent corruption must not ship: detect the streaming markers and return 501 with a clear message before reading the body. Documented in CONTRACT-SCOPE.md; aws-sdk consumers set requestChecksumCalculation: 'WHEN_REQUIRED'. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

D3. pg-store.ts implements the ObjectStore over Postgres bytea (spec § 3): one objects table applied idempotently at init behind a bounded cold-start retry (FT-5226 — a freshly provisioned Postgres rejects the first connect). Ranged reads use SQL substring(bytes from N for L) so a range never detoasts the whole object; list paginates with key like prefix||'%' and key > token order by key limit max+1 (the extra row yields isTruncated + nextContinuationToken); put upserts and computes the quoted-sha256 ETag, matching the in-memory reference. Bun's SQL client with FT-5219 posture (max: 1, idleTimeout). storage-server.ts boots the D2 handler on Bun.serve (binds 0.0.0.0) with the FT-5219 process guards so an idle-close logs instead of crash-looping. Neither is re-exported from the authoring barrel — runtime engine code, kept out of the consumer bundle (dist stays runtime-token-free). The integration test (DoD 5) spins a throwaway local Postgres via the state-store harness pattern, builds the store + server, and drives a real @aws-sdk/client-s3 across every op — exact-bytes round-trip (incl. a ~16 MiB object at the streams segment cap), ranged reads that slice a TOASTed value, list pagination, 404s, delete idempotency, and presigned GET/PUT. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

F3: pg-store list() used `key like ${prefix} || '%'`, so `_`/`%` in a prefix acted as LIKE wildcards and over-matched. Switch to starts_with(key, prefix) (Postgres 11+) for S3's literal-prefix semantics; add an integration test proving prefix "a_" matches "a_b"/"a_c" but not "axb". F4: toBytes returned empty bytes if bytea ever decoded as a non-Uint8Array — the silent-corruption shape F2 closed. Throw a TypeError instead so a driver surprise fails loudly rather than serving wrong bytes. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

…D4a) Two pack capabilities the storage module (D4b) will provision. s3-credentials: a `credentials`-kind contract + dual-form factory (mirrors postgres()) — `s3Credentials({ name })` is the resource identity, `s3Credentials()` the dependency whose binding is CredentialsConfig (secretAccessKey secret-flagged). Its mint is a custom Alchemy resource whose provider `reconcile` returns the persisted `output` (the prior key pair) unchanged and generates a fresh pair only on first create — so an unchanged module no-ops on redeploy. The pair uses the Web Crypto global (no node: import, honoring the package's runtime-coupling invariant) and lives in Alchemy state, the same boundary as the postgres Connection. s3-store: `s3StoreService(def)` is compute's runnable with the routing type overridden to 's3-store' (nothing at runtime keys off type — the serializer keys off address + param owner/name). Its service lowering delegates provision/serialize/package to compute's and extends serialize+deploy outputs with the four consumer-visible S3Config fields: `bucket` from the service param, `accessKeyId`/`secretAccessKey` from the wired s3-credentials resource (reachable in the built Config as inputs.credentials). A consumer wiring the `store` port into an s3() slot resolves those by name. Both registered in the Target table (nodes + provider) and exported from the extension barrel. Tests prove mint idempotency (a redeploy reuses the stored pair) and that s3-store deploy outputs carry all four S3Config field names. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

… (F5) s3-store's serialize now throws a clear error when the `credentials` dependency (accessKeyId + secretAccessKey) or the `bucket` param is missing, instead of surfacing undefined outputs. An unwired store would otherwise deploy and 403 every request (no verifiable credentials) or have no key namespace — this documents the D4a<->D4b naming contract and turns that silent failure into a deploy-time error. Lowering test covers both throws. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

Makes @internal/storage a deployable, consumable module, mirroring cron. - storage-service.ts: `storageService({ bucket })` builds the s3-store service (postgres db + minted credentials deps, a bucket param, the store port on s3Contract), and default-exports a bare node the deploy bootstrap runs. - storage-entrypoint.ts: the runtime boot — load() the deps, config() the params, createPgStore(db.url), startStorageServer(...). Where D2's engine and D3's Postgres store finally meet the framework's load()/config(). - storage-module.ts: `storage()` owns db + credentials + the service wired to both and exposes a single `store` port; a consumer wires ref.store into an s3() slot. - Barrel exports `storage`/`storageService` (authoring); the runtime engine (handler/sigv4/pg-store/storage-server/entrypoint) stays out, so index.mjs carries no bun/node: token. package.json/tsdown add the storage-service + storage-entrypoint dist entries (bun kept external, ADR-0008). - Public @prisma/compose-prisma-cloud/storage subpath (src/storage.ts + ./storage and ./storage/storage-entrypoint exports + tsdown passes), mirroring /cron. Load test proves the wired graph (db->service, credentials->service, and a consumer's s3() slot resolving to the service's store port); a type test proves the store port wires into an s3() slot and a wrong-kind port does not. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

pkg-pr-new · 2026-07-13T12:50:02Z

Open in StackBlitz

npm i https://pkg.pr.new/@prisma/compose@69

npm i https://pkg.pr.new/@prisma/compose-prisma-cloud@69

commit: e64e486

Proves the storage module deploys to real Prisma Cloud and speaks S3 to a real @aws-sdk/client-s3 client (brief DoD 2 + 4). Per design-notes decision 10 (minted creds never surface externally), the proof is an in-deployment smoke consumer: examples/storage provisions storage() plus a `smoke` compute service whose `blob` slot wires to the module's `store` port, so load() hands it the full S3Config (url, bucket, minted creds) with no external creds read. GET / runs the full op suite against the deployed store and returns JSON; the harness reads only the smoke URL (Management API, post-promote — PRO-200) and curls it. Live-deployed, all nine in-scope ops passed against the deployed module (put-get, ranged incl. open-ended, head, delete-idempotency, list pagination, 404s, presigned put+get), then destroyed and verified. Two supporting changes: - fix the storage service node's build.module to emit `service.mjs` (not storage-service.mjs): @prisma/compose/node's assemble() re-bundles build.module and requires the output basename to be `service.*`. The cron-scheduler naming this mirrored was never live-deployed, so this only surfaced at D5's deploy. - add the `/testing` local stand-in surface (createPgStore + startStorageServer, re-exported publicly as @prisma/compose-prisma-cloud/storage/testing, spec § 7) so the example validates the same op suite locally against a throwaway Postgres before the cloud run. Also gitignore the generated .prisma-compose/. The op suite (src/smoke/ops.ts) is shared: proven locally against the stand-in and run in-deployment against the deployed store. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

… app The D5 example read as a smoke-test harness (a consumer that returned pass/fail JSON). Reshape it into a small, natural app that genuinely uses object storage, mirroring examples/cron's cleanliness: a blob store/serve HTTP service backed by the storage module. - src/blobs/app.ts: createBlobApp(S3Config) — a Request→Response handler that builds an @aws-sdk/client-s3 client (path-style, WHEN_REQUIRED checksums) and serves PUT/GET/DELETE /blobs/:key (GET honors Range → 206) + GET /blobs (list, optional ?prefix=). The same handler runs behind Bun.serve in the deployed service and in the integration test with no server. - src/blobs/service.ts: the compute service with an s3() `store` slot. - src/blobs/server.ts: load() the S3Config, Bun.serve(createBlobApp(store)). - module.ts wires storage()'s store port into the blobs app's slot. - tests/blobs.integration.test.ts: drives the app against the storage module's local stand-in (/storage/testing over a throwaway Postgres) and asserts store→retrieve round-trips (put/get, ranged get, prefix list, delete→404). CI-gated like the other integration tests. Drops the unused presigner dep. Deployed live to Prisma Cloud and verified end to end against the deployed store: PUT a blob, GET it back (bytes matched), list, delete→404; then destroyed clean. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

Adds the storage module's README (what it is, the in/out-of-scope contract with a pointer to CONTRACT-SCOPE.md, the ~16 MiB envelope, a real wiring example — provision(storage()) + a consumer's s3() slot building an aws-sdk client with requestChecksumCalculation: 'WHEN_REQUIRED', local-dev via /storage/testing, and the minted-credentials-via-binding note). Adds the @prisma/compose-prisma-cloud/storage row to core-model.md's entry-point map. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

- F6: the storage() module test now passes a custom bucket and asserts it reaches the s3-store service's `bucket` param default (not just the name-derived node ids). - F9: examples/storage's pg-harness comments said "smoke" after the smoke->blobs example reshape; updated to the blob-store integration test. - Drop the stale "authoritative streams proof is D6" claim from sigv4.test.ts (streams was descoped) — the streams-server wire shape is covered at the unit level here. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

wmadden · 2026-07-13T16:16:29Z

+async function withColdStartRetry<T>(
+  op: () => Promise<T>,
+  attempts = 12,
+  delayMs = 5000,
+): Promise<T> {
+  let lastError: unknown;
+  for (let attempt = 1; attempt <= attempts; attempt++) {
+    try {
+      return await op();
+    } catch (error) {
+      if (!isTransient(error)) throw error;
+      lastError = error;
+      if (attempt < attempts) await sleep(delayMs);
+    }
+  }
+  throw lastError;
+}


Reuse this from the last implementation

Done in e64e486. Moved the pure retry helpers (isTransientConnectionError / withConnectionRetry / retryTransientConnect) into @internal/foundation/connection-retry — the shared 0-foundation layer — and re-exported them from pg-connection.ts so the deploy lowerings are unchanged. pg-store.ts now calls the shared retryTransientConnect; deleted its local copy. Runtime bundle stays clean (only bun + node:crypto external — foundation helper is pure, inlined). Gate green: storage 47/47, target 128/128 (via the re-export), repo build + depcruise + cast-ratchet clean.

wmadden · 2026-07-13T16:16:50Z

do we really need to handroll all this?

Fair question — hand-rolling crypto deserves the scrutiny. The catch is that this is the verification side, the awkward half of SigV4:

For signing you would never hand-roll — that is exactly what the @aws-sdk/client-s3 client does in examples/storage and the tests.

For a server verifying an incoming signature there is no standard drop-in. S3-compatible servers (MinIO, LocalStack, s3proxy) either hand-roll it or re-sign-and-compare with the signer lib.

The one realistic way to cut the crypto here is @smithy/signature-v4 (the official signer): re-sign the incoming request with the known secret and compare, instead of building the canonical request + deriving the key by hand. That removes ~half this file — but adds @smithy/signature-v4 (+ @smithy/protocol-http + a sha256 impl) as runtime deps to the storage service, which today has zero deps beyond node:crypto. It still needs the parse / presign / expiry glue, and since it is the auth boundary I would re-run the aws-sdk + streams-server-signer fixtures (and re-deploy) after.

My lean is to keep it hand-rolled: zero-dep, unit-tested against both aws-sdk-signed and streams-server-signed requests, and live-proven against the deployed store. But happy to do the @smithy/signature-v4 swap if you would rather trade ~100 lines for the dependency — say the word and I will.

Keeping it hand-rolled + zero-dep, per your call. Doing a dedicated readability/brevity pass over this file (and the rest of the module engine) now — it should be a lot leaner and easier to read, since the fixtures pin the behavior.

Readability pass landed in 8216671. The header and presign paths were each re-doing the whole canonical-request + key-derivation + compare; they now delegate to one verifySignature core, so the crypto lives in one place. Plus a couple of smaller dedups (GET/HEAD headers, the ranged/whole get() row mapper) and trimmed the doc comments. All the sigv4 + integration fixtures stay green (behavior unchanged).

Honest caveat: line count is basically flat (~219). Zero-dep SigV4 verification has a floor around here — canonical request, key derivation, header + presign parsing, constant-time compare are all genuinely needed. It reads a lot better now (single-source), but it is not materially shorter. If shorter mattered more than zero-dep, the library swap was the only lever — but you called zero-dep and I think that is right for a boundary like this.

pg-store.ts hand-rolled its own FT-5226 cold-start retry (withColdStartRetry + isTransient + a transient-fragment list), duplicating the target extension's pg-connection.ts. Move the pure connection-resilience helpers (isTransientConnectionError, withConnectionRetry, retryTransientConnect) into @internal/foundation/connection-retry — the 0-foundation layer both packages can import — so there is one implementation. pg-connection.ts re-exports them (its consumers — prisma-next, pg-warm-resource, prisma-next-migrate, tests — are unchanged), and pg-store.ts imports retryTransientConnect from foundation. The helper is pure (no effect/alchemy/pg/bun), so the storage runtime bundle stays clean: storage-entrypoint.mjs and /storage/testing keep only `bun` + `node:crypto` external, with the retry helper inlined. The shared predicate covers the storage cold-start case (the "upstream database" PPG-edge reject, FT-5226). foundation gains @types/node (dev-only) for setTimeout. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

Behavior-preserving readability pass (the auth fixtures + integration tests pin correctness — all still green): - sigv4.ts: collapse verifyHeader/verifyPresigned's duplicated signing tail into one verifySignature core (canonical request → signing key → constant- time compare), with the access-key check folded in; each form is now a small "parse this auth form" function (presign keeps its expiry check inline). De-densify the query-sort comparator into a named cmp (byte order unchanged). Trim the header doc to the load-bearing payload-hash note. - pg-store.ts: alias the ranged substring result as `bytes` and map both the whole-object and ranged get() branches through one toGetResult builder. - handler.ts: share the GET/HEAD object-metadata headers via metaHeaders (content-length stays a param — slice length for GET, total for HEAD); trim the isStreamingPut comment. Every guard preserved: constant-time compare + its length/non-empty check, the streaming-PUT 501, the fail-closed asserts, toBytes throw. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

wmadden-electric added 10 commits July 13, 2026 11:04

docs(storage): drop private streams internal paths from scope doc

2ae86ba

Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>

wmadden-electric added 4 commits July 13, 2026 16:35

wmadden-electric marked this pull request as ready for review July 13, 2026 16:04

wmadden approved these changes Jul 13, 2026

View reviewed changes

wmadden-electric added 2 commits July 13, 2026 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(storage): S3-compatible object storage on Compute + Postgres#69

feat(storage): S3-compatible object storage on Compute + Postgres#69
wmadden-electric wants to merge 16 commits into
mainfrom
claude/storage-s3-scope-spike-cd59b6

wmadden-electric commented Jul 13, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

wmadden Jul 13, 2026

Uh oh!

wmadden-electric Jul 13, 2026

Uh oh!

wmadden Jul 13, 2026

Uh oh!

wmadden-electric Jul 13, 2026

Uh oh!

wmadden-electric Jul 13, 2026

Uh oh!

wmadden-electric Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wmadden-electric commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

storage — S3-compatible object storage on Compute + Postgres

Wire scope (the contract)

Design

Verification

Uh oh!

pkg-pr-new Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wmadden Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

wmadden-electric Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

wmadden Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

wmadden-electric Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

wmadden-electric Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

wmadden-electric Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wmadden-electric commented Jul 13, 2026 •

edited

Loading

`storage` — S3-compatible object storage on Compute + Postgres

pkg-pr-new Bot commented Jul 13, 2026 •

edited

Loading