feat(replica): embed read replicas in Cloudflare as per-app Durable Objects#2640
feat(replica): embed read replicas in Cloudflare as per-app Durable Objects#2640riderx wants to merge 7 commits into
Conversation
📝 WalkthroughWalkthroughAdds a Cloudflare Durable Object-based edge read replica system for mirrored tables, a new ChangesEdge replica read path
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_35ce8555-60bc-45ea-8e36-46c5f84ec9cf) |
Merging this PR will not alter performance
Comparing Footnotes
|
RobinWitch
left a comment
There was a problem hiding this comment.
Rechecked the current head (d0f34cb). The freshness heartbeat can mark the D1 replica as safe while unapplied outbox rows are still waiting.
drainOutbox() writes last_applied_at = now() after every D1 batch, and the reader's assertFresh() only checks that timestamp. If the outbox backlog is larger than one alarm can drain (MAX_BATCHES_PER_ALARM * OUTBOX_BATCH_SIZE), the worker exits with rows still pending but a fresh heartbeat, so /updates will read stale D1 data instead of falling back to Postgres. The same happens right after seeding: seedStep() sets both seeded_at and last_applied_at before the accumulated seed-window outbox rows have been replayed, then waits until the next poll to start draining them.
This breaks the “never worse than today” fallback guarantee. A channel/version change made during seed, or any sustained backlog, can be invisible to devices while X-Database-Source: edge_replica is returned as fresh. Please keep the replica unavailable until the seed-window backlog is drained, and make freshness depend on outbox catch-up (for example, no pending rows older than the max lag, or a source commit/outbox watermark) rather than only on the worker's wall-clock poll time. A regression should leave more pending outbox rows than one alarm drains, or enqueue a channel change during seed, and assert the reader falls back instead of serving D1.
|
One more seed-safety issue, separate from the heartbeat/backlog problem above: the initial seed needs a main/outbox-source watermark before it can safely read from an async replica.
Please either seed from the main/outbox source, or record/wait for a WAL/LSN/catch-up watermark proving the seed replica has replayed through the outbox-install point before keyset seeding begins. |
…ication Replace the external Google Cloud SQL read replicas with a D1 database living inside Cloudflare, fed by a transactional outbox on Supabase and fanned out to every region by D1 read replication (Sessions API). - migration: replicate_outbox table + row triggers on the 10 replicated tables, pg_cron safety purge; poll load on main DB is one indexed query every few seconds, device reads never touch Postgres - capgo_replicator worker: single Durable Object draining the outbox with DELETE ... FOR UPDATE SKIP LOCKED (exactly-once, ordered, resumable) and a resumable keyset-paginated seed that reads from the existing replica - edge_replica reader: D1 Sessions hot-path queries mirroring the exact Postgres shapes (app owner + plan validation, channel/override/rollout, manifest aggregation), gated by EDGE_REPLICA_MODE with heartbeat staleness guard and automatic per-request fallback to Postgres - plugin wrangler: commented DB_REPLICA binding for per-region rollout Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rop D1
D1 has a per-database throughput ceiling we already hit with DB_STOREAPPS,
so replace the single shared D1 with one SQLite-backed Durable Object per
${region}:${app_id} — reads scale horizontally with the number of active
apps and each app only competes with itself.
- migration: outbox now carries routing keys (app_id / owner_org, resolved
in-trigger for manifest and stripe_info) and covers the 7 hot-path tables
- ReplicaRouter DO: drains the outbox into a SQLite journal (exactly-once),
fans out to registered replicas by app/org with per-target cursors,
retries then invalidates flaky replicas, refreshes freshness leases,
prunes the journal
- AppReplica DO: lazy self-seed from the read replica on first read in a
region (keyset-paginated, buffers pushes mid-seed), idempotent apply,
lease-gated reads, idle self-eviction after 7 days
- reader: two region-local RPCs per update check, same shapes as the
Postgres path, automatic per-request fallback when a replica is warming
up, stale, or erroring; rollout decisions stay worker-side (Cache API)
- validated: query builders + reader mapping run against real SQLite
(bun:sqlite), migration dry-run applied on local Supabase, both workers
bundle via wrangler --dry-run
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d0f34cb to
7b67872
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_6a84dc6e-6c7c-433b-a911-0d3916d7a607) |
…res fallback The old Cloud SQL system will be destroyed once the embedded replica proves itself, so the new path must never lean on Postgres: - /updates stays exactly as today (pure Postgres) — untouched control group - new /updates_v2 reads only from the per-app replica DOs; when a replica is not ready (warming, lease expired, error) it answers a plain "no update" (edge_replica_not_ready, kind up_to_date) so devices simply retry on their next check while the replica seeds in the background - reader throws EdgeReplicaNotReadyError instead of falling back; the only Postgres client on the v2 path is a lazy read-write handle to the main DB (Hyperdrive DIRECT, survives decommission) for rare notification helpers - load-balance /updates vs /updates_v2 in production, compare, then decommission Cloud SQL Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Stale comment
Risk: high. Not approving: Cursor Bugbot skipped on this run and reported 3 unresolved high-severity findings on the current head, and this hot-path infra change exceeds the low-risk approval threshold. Assigned RobinWitch and WcaleNieWolny for human review.
Sent by Cursor Approval Agent: Pull Request Approver External
- seed watermark (RobinWitch, Bugbot): when the seed source is an async
replica, wait until its subscription has replayed past the outbox
source's pg_current_wal_lsn() before snapshotting; if it does not catch
up, snapshot from the outbox source instead — the snapshot can no longer
miss rows the outbox already skipped
- seed protocol hardening (Bugbot): unregister before clearing the pending
buffer so buffered pushes can never be dropped after registration; an
unseeded replica with no seed running answers { unregister } instead of
acking pushes into the void, and a failed seed unregisters itself
- org moves (Bugbot): applyEntries now calls the new updateOwnerOrg router
RPC which updates the routing key without resetting the push cursor
- backlog-aware freshness (RobinWitch): drainOutbox reports whether the
outbox is fully consumed; while behind, the router loops immediately,
pushes stop extending leases and heartbeats pause, so replicas expire
into "no update" instead of serving data that silently lags the outbox
- CodeQL: admin endpoints return a generic 500, details stay in logs
- SonarQube: readonly members, .at(-1), single push, extracted nested
ternary, object-safe stringification, export ... from re-export
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
All review findings are addressed in bd6ec0f: @RobinWitch — seed watermark: fixed. @RobinWitch — heartbeat/backlog freshness (your review referenced the old D1 head, but the same class of issue existed in the DO rework): fixed at the source of truth. Cursor Bugbot:
CodeQL — admin endpoints return a generic SonarQube — all 12 issues fixed (readonly members, Verified: typecheck, oxlint, unit tests, reader SQL validation against real SQLite, and 🤖 Generated with Claude Code |
There was a problem hiding this comment.
Stale comment
Risk: high. Not approving: Cursor Bugbot is still pending after the polling window, and the current head has 3 unresolved high-severity Bugbot findings on the edge-replica hot path. Human review is required; RobinWitch and WcaleNieWolny are already assigned.
Sent by Cursor Approval Agent: Pull Request Approver External
There was a problem hiding this comment.
Stale comment
Risk: high. Not approving: Cursor Bugbot did not reach a terminal state within the 8-minute polling window on this sync, and prior Bugbot runs reported 3 unresolved high-severity findings on this hot-path replication change. Human review is still required; RobinWitch and WcaleNieWolny are already assigned.
Sent by Cursor Approval Agent: Pull Request Approver
| if (pending === 0) { | ||
| await outboxClient.end().catch(() => undefined) | ||
| return seedClient | ||
| } |
There was a problem hiding this comment.
Seed skips lag verification
High Severity
The openSeedClient method incorrectly interprets zero rows from pg_stat_subscription as a seed replica being caught up. For physical read replicas, such as HYPERDRIVE_SEED in EU production, this table is empty, preventing detection of lag against the outbox source. This bypasses the fallback to the outbox, causing the replica to seed with stale or incomplete data.
Reviewed by Cursor Bugbot for commit bd6ec0f. Configure here.
| if (ownerOrg) | ||
| this.setMeta('owner_org', ownerOrg) | ||
| this.setMeta('seeded', '1') | ||
| this.setMeta('lease_until', String(Date.now() + leaseMs)) |
There was a problem hiding this comment.
Post-seed lease ignores outbox backlog
Medium Severity
After seeding, lease_until is set to a full EDGE_REPLICA_LEASE_SECONDS window even when the router still has outbox backlog and sends pushes with extendLease: false. isFresh() then keeps serving /updates_v2 reads instead of the documented unavailable/no-update behavior while the journal trails live Postgres.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit bd6ec0f. Configure here.
CI forbids console.* under supabase/functions; route DO logging through cloudlog/cloudlogErr with serializeError like the rest of the backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Stale comment
Risk: high. Not approving: Cursor Bugbot skipped on this run, and the latest completed Bugbot review still has 1 unresolved high-severity finding on the edge-replica seed path plus 1 unresolved medium finding; this hot-path infra change also exceeds the low-risk approval threshold. RobinWitch and WcaleNieWolny are already assigned.
Sent by Cursor Approval Agent: Pull Request Approver External
There was a problem hiding this comment.
Stale comment
Risk: high. Not approving: Cursor Bugbot completed with status skipping on this sync, and the current head has 5 unresolved Bugbot findings (4 high-severity) on the edge-replica hot path. Human review is still required; RobinWitch and WcaleNieWolny are already assigned.
Sent by Cursor Approval Agent: Pull Request Approver
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@supabase/functions/_backend/replicator/app_replica.ts`:
- Around line 226-242: Seed failures currently clear immediately in
scheduleSeed(), so repeated unavailable() reads can continuously retry seed()
and storm Postgres. Add a persisted failure marker such as seed_failed_at in
replica_meta and check it at the start of scheduleSeed() to skip reseeding until
a backoff window has elapsed. Update the seeding flow around scheduleSeed(),
seed(), and the routerStub().unregister path so failures register the cooldown
while still leaving the replica unregistered until a successful seed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3e9b2932-d662-4a9c-bc9f-c7ff3fa8354d
📒 Files selected for processing (17)
.gitignorecloudflare_workers/plugin/index.tscloudflare_workers/plugin/wrangler.jsonccloudflare_workers/replicator/README.mdcloudflare_workers/replicator/index.tscloudflare_workers/replicator/wrangler.jsoncpackage.jsonsupabase/functions/_backend/plugins/updates.tssupabase/functions/_backend/replicator/app_replica.tssupabase/functions/_backend/replicator/replicator.tssupabase/functions/_backend/utils/cloudflare.tssupabase/functions/_backend/utils/edge_replica.tssupabase/functions/_backend/utils/edge_replica_schema.tssupabase/functions/_backend/utils/pg.tssupabase/functions/_backend/utils/update.tssupabase/migrations/20260707150000_edge_replica_outbox.sqltests/edge-replica.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
… round - heartbeats only extend leases of targets fully caught up with the journal (cursor >= head); lagging targets refresh through successful pushes only, so a lease can never mask undelivered rows (Bugbot) - age-based journal pruning first invalidates any target still behind the aged window so it reseeds instead of silently missing changes (Bugbot) - org moves now reseed the replica entirely instead of patching routing keys: the new org's rows may not exist in the journal at all, and the updateOwnerOrg RPC raced the cursor advance (Bugbot); a move detected while replaying the seed window re-runs the seed with the fresh owner - failed seeds set a 60s backoff so hot apps with a broken seed source answer "no update" without hammering Postgres (CodeRabbit) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Second round of findings fixed in 72ab3e6: Bugbot — heartbeat extends lease while lagging: the maintenance heartbeat now only targets replicas whose cursor is at the journal head ( Bugbot — journal prunes unconsumed rows: before age-pruning, any registered target whose cursor is still behind the aged window is unregistered and invalidated, so it reseeds on its next read instead of silently diverging forever. (In practice a target can't lag an hour — 3 failed pushes ≈ 15s already unregister it — but the guard makes it airtight.) Bugbot — org rows skipped on move: rethought. Patching the routing key ( CodeRabbit — seed failure backoff: a failed seed sets Verified: typecheck, oxlint, unit tests, SQLite validation of the reader path, wrangler dry-run bundles, and the CI console-guard grep. 🤖 Generated with Claude Code |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
There are 5 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.
| if (lastRead > 0 && Date.now() - lastRead > IDLE_EVICT_MS) { | ||
| await this.wipe() | ||
| return { unregister: true } | ||
| } |
There was a problem hiding this comment.
Router push idle evicts active replicas
Medium Severity
applyBatch wipes the replica and unregisters when last_read_at is older than seven days, but only device read RPCs update last_read_at. Router pushes and heartbeats that keep data fresh still trigger eviction, forcing reseeds and edge_replica_not_ready for apps with infrequent device check-ins.
Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.
| this.scheduleSeed(movedToOrgApp) | ||
| return { unregister: true } | ||
| } | ||
| return { ok: true } |
There was a problem hiding this comment.
Orphaned pending batches after seed
High Severity
Journal pushes that arrive after seedOnce replays pending_batches but before seedingPromise clears are appended to pending_batches and acknowledged by the router, yet nothing drains that table once seeded is set. Those changes never reach SQLite while reads can already pass isFresh(), so /updates_v2 may serve stale channel or plan data until a full reseed.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.
| head, | ||
| Date.now(), | ||
| target.name, | ||
| ) |
There was a problem hiding this comment.
Router cursor acks buffered seed pushes
Medium Severity
While a seed runs, applyBatch only buffers journal entries and returns success, yet pushTarget still advances the target cursor to the journal head. If seeding fails before replay, pending_batches is cleared on retry and those journal IDs are never applied to SQLite for that registration.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.
|
There was a problem hiding this comment.
Risk: high. Not approving: Cursor Bugbot completed with status skipping on this sync, and the current head has 5 unresolved Bugbot findings on the edge-replica hot path. Human review is still required; RobinWitch and WcaleNieWolny are already assigned.
Sent by Cursor Approval Agent: Pull Request Approver
There was a problem hiding this comment.
Risk: high. Not approving: Cursor Bugbot completed with status skipping on this sync, and the current head has 5 unresolved Bugbot findings on the edge-replica hot path; this infra change also exceeds the low-risk approval threshold. RobinWitch and WcaleNieWolny are already assigned.
Sent by Cursor Approval Agent: Pull Request Approver External
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
supabase/functions/_backend/replicator/app_replica.ts (2)
303-315: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftTreat an empty
pg_stat_subscriptionas unknown for seed sources that may be physical replicas. The configured seed source can be a read replica, so this query can report0even when replay is still behind; fall back to the outbox source, or check a real replay LSN before returning the seed client.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@supabase/functions/_backend/replicator/app_replica.ts` around lines 303 - 315, Treat the seed-source readiness check in `AppReplica` as unsafe when `pg_stat_subscription` is empty, since a read replica can report zero rows even while replay is still behind. Update the logic around the `seedClient.query(...)` pending check to distinguish “no subscription rows” from “caught up,” and only return the seed client when you have a real replay/LSN confirmation. Otherwise fall back to the outbox source path that logs the lagging seed source and snapshots from the outbox.
287-296: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftCapture the watermark after registering the target. A write that lands between the current
pg_current_wal_lsn()read androuter.register()can already be in the router journal, butregister()sets the replica cursor tojournalHead(), so that row is never pushed. If the seed snapshot is still only caught up to the older watermark, the write can be missed entirely. Register first, then wait the seed source past a post-registration watermark or snapshot from the outbox source.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@supabase/functions/_backend/replicator/app_replica.ts` around lines 287 - 296, The seed replication flow in openSeedClient() is capturing the watermark too early, which can miss writes that land before router.register() but after the initial pg_current_wal_lsn() read. Update the ordering so the target is registered first, then obtain a post-registration watermark or wait the seed source past it from the outbox source, ensuring the replica cursor and seed snapshot stay aligned. Use the existing openSeedClient(), router.register(), and journalHead() flow to place the watermark/snapshot logic after registration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@supabase/functions/_backend/replicator/app_replica.ts`:
- Around line 187-188: The move/reseed logic in app_replica.ts does not handle
transitions where owner_org is null, so org→null or null→org changes can bypass
reseeding or leave stale metadata. Update the move detection around scheduleSeed
and the owner_org meta writes so changes are detected whenever the stored value
differs, not only when both sides are truthy, and ensure replica_meta is cleared
or rewritten appropriately in the reseed path. Check the related move handling
blocks near scheduleSeed and the metadata update code used for owner_org to keep
null transitions consistent.
In `@supabase/functions/_backend/replicator/replicator.ts`:
- Around line 449-455: The stranded-replica cleanup in `Replicator` is deleting
target rows before `replicaStub(target.name).invalidate()` has actually
completed, which lets pruning advance too early. Update the cleanup flow around
the stranded targets loop so the target row and journal window are only removed
after invalidation is confirmed, or otherwise make the pruning path block/retry
when `invalidate()` fails; keep the existing `ctx.waitUntil`/`replicaStub` logic
tied to successful invalidation rather than immediate deletion.
---
Outside diff comments:
In `@supabase/functions/_backend/replicator/app_replica.ts`:
- Around line 303-315: Treat the seed-source readiness check in `AppReplica` as
unsafe when `pg_stat_subscription` is empty, since a read replica can report
zero rows even while replay is still behind. Update the logic around the
`seedClient.query(...)` pending check to distinguish “no subscription rows” from
“caught up,” and only return the seed client when you have a real replay/LSN
confirmation. Otherwise fall back to the outbox source path that logs the
lagging seed source and snapshots from the outbox.
- Around line 287-296: The seed replication flow in openSeedClient() is
capturing the watermark too early, which can miss writes that land before
router.register() but after the initial pg_current_wal_lsn() read. Update the
ordering so the target is registered first, then obtain a post-registration
watermark or wait the seed source past it from the outbox source, ensuring the
replica cursor and seed snapshot stay aligned. Use the existing
openSeedClient(), router.register(), and journalHead() flow to place the
watermark/snapshot logic after registration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 88e3e853-6abf-496d-a64d-2380cc108235
📒 Files selected for processing (2)
supabase/functions/_backend/replicator/app_replica.tssupabase/functions/_backend/replicator/replicator.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
| this.ctx.storage.sql.exec(`DELETE FROM replica_meta WHERE key IN ('seeded', 'lease_until')`) | ||
| this.scheduleSeed(movedToOrgApp) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Handle org moves to or from NULL owner_org.
owner_org is read as string | null, but move detection requires both values to be truthy, and reseed only writes the meta when ownerOrg is truthy. Moves from org→null or null→org can skip the reseed path or leave stale owner_org metadata.
Proposed fix
- this.ctx.storage.sql.exec(`DELETE FROM replica_meta WHERE key IN ('seeded', 'lease_until')`)
+ this.ctx.storage.sql.exec(`DELETE FROM replica_meta WHERE key IN ('seeded', 'lease_until', 'owner_org')`)- const ownerOrg = typeof entry.row.owner_org === 'string' ? entry.row.owner_org : null
- const storedOrg = this.getMeta('owner_org')
- if (entry.table === 'apps' && ownerOrg && storedOrg && ownerOrg !== storedOrg && typeof entry.row.app_id === 'string')
- movedToOrgApp = entry.row.app_id
+ if (entry.table === 'apps' && typeof entry.row.app_id === 'string' && Object.prototype.hasOwnProperty.call(entry.row, 'owner_org')) {
+ const ownerOrg = typeof entry.row.owner_org === 'string' ? entry.row.owner_org : null
+ const storedOrg = this.getMeta('owner_org')
+ if (ownerOrg !== storedOrg)
+ movedToOrgApp = entry.row.app_id
+ }- if (ownerOrg)
- this.setMeta('owner_org', ownerOrg)
+ if (ownerOrg)
+ this.setMeta('owner_org', ownerOrg)
+ else
+ this.ctx.storage.sql.exec('DELETE FROM replica_meta WHERE key = ?', 'owner_org')Also applies to: 213-216, 392-393
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@supabase/functions/_backend/replicator/app_replica.ts` around lines 187 -
188, The move/reseed logic in app_replica.ts does not handle transitions where
owner_org is null, so org→null or null→org changes can bypass reseeding or leave
stale metadata. Update the move detection around scheduleSeed and the owner_org
meta writes so changes are detected whenever the stored value differs, not only
when both sides are truthy, and ensure replica_meta is cleared or rewritten
appropriately in the reseed path. Check the related move handling blocks near
scheduleSeed and the metadata update code used for owner_org to keep null
transitions consistent.
| const stranded = this.ctx.storage.sql | ||
| .exec('SELECT name FROM targets WHERE cursor < ?', agedMax.m) | ||
| .toArray() as { name: string }[] | ||
| for (const target of stranded) { | ||
| this.ctx.storage.sql.exec('DELETE FROM targets WHERE name = ?', target.name) | ||
| this.ctx.waitUntil(this.replicaStub(target.name).invalidate().catch(() => undefined)) | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Do not prune rows until stranded replicas are actually invalidated.
Line 453 deletes the target and Line 454 only schedules invalidate() with waitUntil; Line 462 can then prune the journal immediately. If invalidation fails or races a read, the AppReplica can keep serving under its existing lease while unregistered and missing pruned rows. Keep the target/journal window until invalidation succeeds, or make pruning fail/retry when invalidation cannot be confirmed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@supabase/functions/_backend/replicator/replicator.ts` around lines 449 - 455,
The stranded-replica cleanup in `Replicator` is deleting target rows before
`replicaStub(target.name).invalidate()` has actually completed, which lets
pruning advance too early. Update the cleanup flow around the stranded targets
loop so the target row and journal window are only removed after invalidation is
confirmed, or otherwise make the pruning path block/retry when `invalidate()`
fails; keep the existing `ctx.waitUntil`/`replicaStub` logic tied to successful
invalidation rather than immediate deletion.







What
Moves the read-replica layer for update checks inside Cloudflare, replacing the external Google Cloud SQL fleet (1 subscriber + 9 regional replicas + Hyperdrive hops) with per-app SQLite replicas running as Durable Objects, exposed on a parallel endpoint
/updates_v2so old and new can be load-balanced in production until the old system is decommissioned.D1 was rejected on purpose: a single shared database has a single throughput ceiling (we already hit it with
DB_STOREAPPS). One DO per${region}:${app_id}spreads reads across as many isolates as there are active apps — each app only competes with itself, and hot apps are further split 9 ways by region.Architecture
No Postgres fallback — by design
The old system gets destroyed after the transition, so
/updates_v2never reads Postgres. If a replica is not ready (warming up, lease expired, router down), the device gets a plain "no update" (edge_replica_not_ready, kindup_to_date) and retries on its next check while the replica seeds itself in the background. Worst case is a delayed update — never broken devices, never read traffic landing on the main DB./updatesis completely untouched (pure Postgres) and acts as the control group during the load-balanced transition. Both endpoints return identical response shapes.Reliability
postgres_changesis fire-and-forget over WebSocket — dropped connections lose rows, no replay cursor, cannot seed. The outbox is transactional and resumable after any downtime.POST /invalidate-allforces a fleet-wide reseed after schema changes. Replicas idle 7 days self-evict and wipe.Cost / load
Rollout
wrangler secret put REPLICATOR_SECRET ... && bun run deploy:cloudflare:replicator:prodAPP_REPLICAbinding +"EDGE_REPLICA_MODE": "on", deploy. Without the flag,/updates_v2just answers "no update" — deploying is always safe./updatesto/updates_v2, watchX-Database-Source: edge_replicavsedge_replica_not_readyand response parity, ramp to 100%.Merging this PR changes nothing by itself:
/updatesis untouched and/updates_v2is inert until the binding + flag are enabled.Verified
wrangler deploy --dry-run.bun run typecheck,oxlint, full unit suite green.🤖 Generated with Claude Code
Note
High Risk
Large new replication and update hot-path surface; incorrect parity or lease/outbox handling could delay OTA responses or mis-route plan/channel data, though
/updatesremains the control path until rollout.Overview
Introduces a Cloudflare edge read-replica stack to eventually replace regional Cloud SQL + Hyperdrive for device update checks, while leaving
/updatesunchanged.Data path: A new migration adds
public.replicate_outboxand triggers on seven hot tables; acapgo_replicatorworker runsReplicaRouter(polls/deletes outbox into a journal, fans out to targets) andAppReplicaDOs (per${region}:${app_id}SQLite, lazy seed, lease-based freshness, push apply).Read path: Plugin workers gain
/updates_v2and optionalAPP_REPLICA+EDGE_REPLICA_MODE=on.updateV2/EdgeReplicaReadermirror Postgres update shapes via DO RPCs with no read fallback—not-ready replicas yieldedge_replica_not_ready(treated as up-to-date).updateWithPGaccepts an optional edge reader for owner/channel lookups.Rollout: Replicator deploy scripts and commented DO bindings in regional plugin
wrangler.jsonc; feature stays inert until migration, replicator deploy, and per-region flag enablement.Reviewed by Cursor Bugbot for commit 72ab3e6. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
/updates_v2endpoint for edge-replica-backed updates alongside existing/updates.