Skip to content

feat(replica): embed read replicas in Cloudflare as per-app Durable Objects#2640

Open
riderx wants to merge 7 commits into
mainfrom
feat/edge-replica-d1
Open

feat(replica): embed read replicas in Cloudflare as per-app Durable Objects#2640
riderx wants to merge 7 commits into
mainfrom
feat/edge-replica-d1

Conversation

@riderx

@riderx riderx commented Jul 7, 2026

Copy link
Copy Markdown
Member

What

Moves the read-replica layer for update checks inside Cloudflare, replacing the external Google Cloud SQL fleet (1 subscriber + 9 regional replicas + Hyperdrive hops) with per-app SQLite replicas running as Durable Objects, exposed on a parallel endpoint /updates_v2 so old and new can be load-balanced in production until the old system is decommissioned.

D1 was rejected on purpose: a single shared database has a single throughput ceiling (we already hit it with DB_STOREAPPS). One DO per ${region}:${app_id} spreads reads across as many isolates as there are active apps — each app only competes with itself, and hot apps are further split 9 ways by region.

Architecture

Supabase Postgres (main)
  │  row triggers on the 7 hot-path tables (apps, orgs, stripe_info,
  │  channels, app_versions, channel_devices, manifest)
  ▼
public.replicate_outbox          ordered change log + routing keys
  │  DELETE .. FOR UPDATE SKIP LOCKED, poll every 5s
  ▼
ReplicaRouter DO                 journal + registry (the "1 replica linked
  │                              to Supabase")
  │  push: app rows → that app's replicas, org rows → all replicas of org
  ▼
AppReplica DO per region:app     SQLite slice of one app (+ its org rows)
  ▲  lazy self-seed from the read replica on first read in a region
  │
capgo_plugin /updates_v2         2 region-local RPCs (~1ms) per check

No Postgres fallback — by design

The old system gets destroyed after the transition, so /updates_v2 never reads Postgres. If a replica is not ready (warming up, lease expired, router down), the device gets a plain "no update" (edge_replica_not_ready, kind up_to_date) and retries on its next check while the replica seeds itself in the background. Worst case is a delayed update — never broken devices, never read traffic landing on the main DB.

/updates is completely untouched (pure Postgres) and acts as the control group during the load-balanced transition. Both endpoints return identical response shapes.

Reliability

  • Exactly-once feed: outbox rows are copied into the router journal before the Postgres delete commits; per-target cursors advance only on a successful push; replicas apply idempotent upserts in outbox order.
  • Why not Supabase Realtime: postgres_changes is fire-and-forget over WebSocket — dropped connections lose rows, no replay cursor, cannot seed. The outbox is transactional and resumable after any downtime.
  • Freshness = lease (15 min, refreshed by every push/heartbeat; typical data lag ≈ 5s poll vs the 180s tolerated on Cloud SQL today). Reads past the lease answer "no update", not stale data.
  • Flaky replicas: 3 failed pushes → invalidate + reseed on next read. POST /invalidate-all forces a fleet-wide reseed after schema changes. Replicas idle 7 days self-evict and wipe.

Cost / load

  • Main DB load: tiny trigger appends on writes + one indexed poll query every 5s. Device reads never touch Postgres.
  • Replicas are created lazily by traffic, so cost scales with active apps, not the catalog.
  • DO pricing: ~$0.15/M requests (2 per check) + SQLite rows read spread across thousands of DOs (no shared ceiling) + $0.20/GB-mo storage. Tens of dollars/month vs the Cloud SQL fleet, with in-colo read latency.

Rollout

  1. Apply the migration (outbox + triggers).
  2. wrangler secret put REPLICATOR_SECRET ... && bun run deploy:cloudflare:replicator:prod
  3. Per plugin region: uncomment the APP_REPLICA binding + "EDGE_REPLICA_MODE": "on", deploy. Without the flag, /updates_v2 just answers "no update" — deploying is always safe.
  4. Load-balance a share of device traffic from /updates to /updates_v2, watch X-Database-Source: edge_replica vs edge_replica_not_ready and response parity, ramp to 100%.
  5. Decommission Cloud SQL; keep one small replica (or the pooler) as the per-app seed source.

Merging this PR changes nothing by itself: /updates is untouched and /updates_v2 is inert until the binding + flag are enabled.

Verified

  • Query builders + reader mapping validated against real SQLite (plan validation incl. trial/exceeded flags, channel/device-override/rollout/builtin paths, manifest aggregation, not-ready → no-update path).
  • Migration applied + rolled back on local Supabase.
  • Both workers bundle via wrangler deploy --dry-run.
  • bun run typecheck, oxlint, full unit suite green.

🤖 Generated with Claude Code


Note

High Risk
Large new replication and update hot-path surface; incorrect parity or lease/outbox handling could delay OTA responses or mis-route plan/channel data, though /updates remains the control path until rollout.

Overview
Introduces a Cloudflare edge read-replica stack to eventually replace regional Cloud SQL + Hyperdrive for device update checks, while leaving /updates unchanged.

Data path: A new migration adds public.replicate_outbox and triggers on seven hot tables; a capgo_replicator worker runs ReplicaRouter (polls/deletes outbox into a journal, fans out to targets) and AppReplica DOs (per ${region}:${app_id} SQLite, lazy seed, lease-based freshness, push apply).

Read path: Plugin workers gain /updates_v2 and optional APP_REPLICA + EDGE_REPLICA_MODE=on. updateV2 / EdgeReplicaReader mirror Postgres update shapes via DO RPCs with no read fallback—not-ready replicas yield edge_replica_not_ready (treated as up-to-date). updateWithPG accepts an optional edge reader for owner/channel lookups.

Rollout: Replicator deploy scripts and commented DO bindings in regional plugin wrangler.jsonc; feature stays inert until migration, replicator deploy, and per-region flag enablement.

Reviewed by Cursor Bugbot for commit 72ab3e6. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • New Features
    • Added a new /updates_v2 endpoint for edge-replica-backed updates alongside existing /updates.
    • Enabled region-aware embedded read-replica support with fail-fast “no update” behavior during warmup/lease expiry.
  • Bug Fixes
    • Improved replica availability handling: when the edge replica is not ready or stale, requests now return “no update” for safer retries.
  • Documentation
    • Added detailed setup, rollout, and operational guidance for the edge-replica system, including admin/status flows.
  • Chores
    • Updated ignore rules for generated replica artifacts.

@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a Cloudflare Durable Object-based edge read replica system for mirrored tables, a new /updates_v2 read path, and the supporting outbox, replica, routing, deployment, and test changes needed to run it.

Changes

Edge replica read path

Layer / File(s) Summary
Outbox migration capturing mirrored table changes
supabase/migrations/20260707150000_edge_replica_outbox.sql
Creates replicate_outbox, adds the capture trigger function, wires table triggers, and schedules outbox cleanup.
Edge replica schema and query builders
supabase/functions/_backend/utils/edge_replica_schema.ts, tests/edge-replica.unit.test.ts
Defines the shared replica schema, SQL builders, RPC DTOs, hot-path queries, and unit tests for conversions and generated SQL.
ReplicaRouter durable object and worker
supabase/functions/_backend/replicator/replicator.ts, cloudflare_workers/replicator/index.ts, cloudflare_workers/replicator/wrangler.jsonc
Implements outbox draining, fan-out, maintenance, admin endpoints, worker entrypoints, and Cloudflare worker configuration.
AppReplica durable object
supabase/functions/_backend/replicator/app_replica.ts
Implements per-app replica seeding, batch application, lease handling, and read RPCs.
EdgeReplicaReader and updates_v2 handler
supabase/functions/_backend/utils/edge_replica.ts, supabase/functions/_backend/utils/update.ts, supabase/functions/_backend/utils/cloudflare.ts, supabase/functions/_backend/utils/pg.ts, supabase/functions/_backend/plugins/updates.ts, cloudflare_workers/plugin/index.ts
Adds edge-replica read helpers, update-v2 routing, updated bindings/types, and the new plugin route.
Deployment config and docs
cloudflare_workers/replicator/README.md, cloudflare_workers/plugin/wrangler.jsonc, package.json, .gitignore
Adds rollout documentation, commented region configs, worker scripts, and a new ignore entry.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

  • Cap-go/capgo#2053: The new /updates_v2 path selects rollout targets with getRolloutDecision, which is related to the rollout-aware update logic in that PR.
  • Cap-go/capgo#2478: The same plugin worker wiring is extended here with /updates_v2, and that PR also changes middleware affecting plugin requests.

Suggested labels: codex

Suggested reviewers: Dalanir, WcaleNieWolny

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: embedding per-app Cloudflare Durable Object read replicas.
Description check ✅ Passed The description is detailed and covers the change, rollout, and verification; only the template section names are not followed exactly.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Jul 7, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_35ce8555-60bc-45ea-8e36-46c5f84ec9cf)

Comment thread supabase/functions/_backend/replicator/replicator.ts Fixed
@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing feat/edge-replica-d1 (72ab3e6) with main (8213b4f)2

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (454a345) during the generation of this report, so 8213b4f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@RobinWitch RobinWitch left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the current head (d0f34cb). The freshness heartbeat can mark the D1 replica as safe while unapplied outbox rows are still waiting.

drainOutbox() writes last_applied_at = now() after every D1 batch, and the reader's assertFresh() only checks that timestamp. If the outbox backlog is larger than one alarm can drain (MAX_BATCHES_PER_ALARM * OUTBOX_BATCH_SIZE), the worker exits with rows still pending but a fresh heartbeat, so /updates will read stale D1 data instead of falling back to Postgres. The same happens right after seeding: seedStep() sets both seeded_at and last_applied_at before the accumulated seed-window outbox rows have been replayed, then waits until the next poll to start draining them.

This breaks the “never worse than today” fallback guarantee. A channel/version change made during seed, or any sustained backlog, can be invisible to devices while X-Database-Source: edge_replica is returned as fresh. Please keep the replica unavailable until the seed-window backlog is drained, and make freshness depend on outbox catch-up (for example, no pending rows older than the max lag, or a source commit/outbox watermark) rather than only on the worker's wall-clock poll time. A regression should leave more pending outbox rows than one alarm drains, or enqueue a channel change during seed, and assert the reader falls back instead of serving D1.

@RobinWitch

Copy link
Copy Markdown

One more seed-safety issue, separate from the heartbeat/backlog problem above: the initial seed needs a main/outbox-source watermark before it can safely read from an async replica.

HYPERDRIVE_SEED is documented/configured as the existing EU read replica, but replicate_outbox only captures writes after this migration installs its triggers. If that seed replica is behind main when /seed starts, rows committed before the outbox trigger existed but not yet replayed to the seed replica are neither copied by the keyset seed nor present in replicate_outbox. Once the seed cursor advances past those rows, D1 can permanently miss apps/channels/versions/manifests until a full reseed.

Please either seed from the main/outbox source, or record/wait for a WAL/LSN/catch-up watermark proving the seed replica has replayed through the outbox-install point before keyset seeding begins.

riderx and others added 2 commits July 7, 2026 15:06
…ication

Replace the external Google Cloud SQL read replicas with a D1 database
living inside Cloudflare, fed by a transactional outbox on Supabase and
fanned out to every region by D1 read replication (Sessions API).

- migration: replicate_outbox table + row triggers on the 10 replicated
  tables, pg_cron safety purge; poll load on main DB is one indexed query
  every few seconds, device reads never touch Postgres
- capgo_replicator worker: single Durable Object draining the outbox with
  DELETE ... FOR UPDATE SKIP LOCKED (exactly-once, ordered, resumable) and
  a resumable keyset-paginated seed that reads from the existing replica
- edge_replica reader: D1 Sessions hot-path queries mirroring the exact
  Postgres shapes (app owner + plan validation, channel/override/rollout,
  manifest aggregation), gated by EDGE_REPLICA_MODE with heartbeat
  staleness guard and automatic per-request fallback to Postgres
- plugin wrangler: commented DB_REPLICA binding for per-region rollout

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rop D1

D1 has a per-database throughput ceiling we already hit with DB_STOREAPPS,
so replace the single shared D1 with one SQLite-backed Durable Object per
${region}:${app_id} — reads scale horizontally with the number of active
apps and each app only competes with itself.

- migration: outbox now carries routing keys (app_id / owner_org, resolved
  in-trigger for manifest and stripe_info) and covers the 7 hot-path tables
- ReplicaRouter DO: drains the outbox into a SQLite journal (exactly-once),
  fans out to registered replicas by app/org with per-target cursors,
  retries then invalidates flaky replicas, refreshes freshness leases,
  prunes the journal
- AppReplica DO: lazy self-seed from the read replica on first read in a
  region (keyset-paginated, buffers pushes mid-seed), idempotent apply,
  lease-gated reads, idle self-eviction after 7 days
- reader: two region-local RPCs per update check, same shapes as the
  Postgres path, automatic per-request fallback when a replica is warming
  up, stale, or erroring; rollout decisions stay worker-side (Cache API)
- validated: query builders + reader mapping run against real SQLite
  (bun:sqlite), migration dry-run applied on local Supabase, both workers
  bundle via wrangler --dry-run

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@riderx riderx force-pushed the feat/edge-replica-d1 branch from d0f34cb to 7b67872 Compare July 7, 2026 16:40
@cursor

cursor Bot commented Jul 7, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_6a84dc6e-6c7c-433b-a911-0d3916d7a607)

@riderx riderx changed the title feat(replica): embed the read replica in Cloudflare (D1 + outbox), drop external Cloud SQL hop feat(replica): embed read replicas in Cloudflare as per-app Durable Objects Jul 7, 2026
…res fallback

The old Cloud SQL system will be destroyed once the embedded replica proves
itself, so the new path must never lean on Postgres:

- /updates stays exactly as today (pure Postgres) — untouched control group
- new /updates_v2 reads only from the per-app replica DOs; when a replica is
  not ready (warming, lease expired, error) it answers a plain "no update"
  (edge_replica_not_ready, kind up_to_date) so devices simply retry on their
  next check while the replica seeds in the background
- reader throws EdgeReplicaNotReadyError instead of falling back; the only
  Postgres client on the v2 path is a lazy read-write handle to the main DB
  (Hyperdrive DIRECT, survives decommission) for rare notification helpers
- load-balance /updates vs /updates_v2 in production, compare, then
  decommission Cloud SQL

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread supabase/functions/_backend/replicator/app_replica.ts
Comment thread supabase/functions/_backend/replicator/app_replica.ts Outdated
Comment thread supabase/functions/_backend/replicator/app_replica.ts
@cursor cursor Bot requested review from RobinWitch and WcaleNieWolny July 9, 2026 08:29

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot skipped on this run and reported 3 unresolved high-severity findings on the current head, and this hot-path infra change exceeds the low-risk approval threshold. Assigned RobinWitch and WcaleNieWolny for human review.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver External

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot skipped on this run and reported 3 unresolved high-severity findings on the current head. Assigned RobinWitch and WcaleNieWolny for human review.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot skipped on this run and reported 3 unresolved high-severity findings on the current head. Human review is required; reviewer assignment was not available for this repository.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver

- seed watermark (RobinWitch, Bugbot): when the seed source is an async
  replica, wait until its subscription has replayed past the outbox
  source's pg_current_wal_lsn() before snapshotting; if it does not catch
  up, snapshot from the outbox source instead — the snapshot can no longer
  miss rows the outbox already skipped
- seed protocol hardening (Bugbot): unregister before clearing the pending
  buffer so buffered pushes can never be dropped after registration; an
  unseeded replica with no seed running answers { unregister } instead of
  acking pushes into the void, and a failed seed unregisters itself
- org moves (Bugbot): applyEntries now calls the new updateOwnerOrg router
  RPC which updates the routing key without resetting the push cursor
- backlog-aware freshness (RobinWitch): drainOutbox reports whether the
  outbox is fully consumed; while behind, the router loops immediately,
  pushes stop extending leases and heartbeats pause, so replicas expire
  into "no update" instead of serving data that silently lags the outbox
- CodeQL: admin endpoints return a generic 500, details stay in logs
- SonarQube: readonly members, .at(-1), single push, extracted nested
  ternary, object-safe stringification, export ... from re-export

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@riderx

riderx commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

All review findings are addressed in bd6ec0f:

@RobinWitch — seed watermark: fixed. AppReplica.openSeedClient() now takes pg_current_wal_lsn() on the outbox source first, then polls the seed replica's pg_stat_subscription.latest_end_lsn until it has replayed past that watermark before any keyset read (a non-subscriber seed source passes trivially). If it doesn't catch up within 10s, the snapshot is taken from the outbox source directly. Rows committed before trigger install / before registration can no longer be missed by both the snapshot and the pushes.

@RobinWitch — heartbeat/backlog freshness (your review referenced the old D1 head, but the same class of issue existed in the DO rework): fixed at the source of truth. drainOutbox() now reports whether the outbox was fully consumed; while a backlog remains the router loops immediately (250ms), pushes are sent with extendLease: false and heartbeats pause — so a lease can only ever assert "caught up as of grant time". A sustained backlog lets leases expire and /updates_v2 answers plain "no update" (there is no Postgres fallback anymore by design — the old system will be destroyed). The seed-window concern is covered by the reworked seed protocol: register → snapshot → replay buffered pushes → only then mark seeded.

Cursor Bugbot:

  • Seed lacks outbox watermark — LSN watermark above; additionally the org-move path no longer goes through register (which resets the cursor): a new updateOwnerOrg router RPC updates the routing key while leaving the push cursor untouched.
  • Unseeded replica acks journal pushes — an unseeded replica with no seed in flight now answers { unregister: true } instead of { ok: true }, and a failed seed unregisters itself, so the router can never advance a cursor past rows that were dropped.
  • Pending batches cleared after register — seed order is now: unregister (no pushes can arrive) → clear buffer → watermark → register → snapshot → replay buffer. The buffer cleared at the start can only contain leftovers from a previous failed attempt.

CodeQL — admin endpoints return a generic internal error 500; details stay in the DO logs.

SonarQube — all 12 issues fixed (readonly members, .at(-1), single push, extracted nested ternary, object-safe stringification, export … from, complexity reduced by splitting seed() into openSeedClient/snapshotTable).

Verified: typecheck, oxlint, unit tests, reader SQL validation against real SQLite, and wrangler deploy --dry-run for both workers.

🤖 Generated with Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot is still pending after the polling window, and the current head has 3 unresolved high-severity Bugbot findings on the edge-replica hot path. Human review is required; RobinWitch and WcaleNieWolny are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver External

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot did not reach a terminal state within the 8-minute polling window on this sync, and prior Bugbot runs reported 3 unresolved high-severity findings on this hot-path replication change. Human review is still required; RobinWitch and WcaleNieWolny are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver

if (pending === 0) {
await outboxClient.end().catch(() => undefined)
return seedClient
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seed skips lag verification

High Severity

The openSeedClient method incorrectly interprets zero rows from pg_stat_subscription as a seed replica being caught up. For physical read replicas, such as HYPERDRIVE_SEED in EU production, this table is empty, preventing detection of lag against the outbox source. This bypasses the fallback to the outbox, causing the replica to seed with stale or incomplete data.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit bd6ec0f. Configure here.

if (ownerOrg)
this.setMeta('owner_org', ownerOrg)
this.setMeta('seeded', '1')
this.setMeta('lease_until', String(Date.now() + leaseMs))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-seed lease ignores outbox backlog

Medium Severity

After seeding, lease_until is set to a full EDGE_REPLICA_LEASE_SECONDS window even when the router still has outbox backlog and sends pushes with extendLease: false. isFresh() then keeps serving /updates_v2 reads instead of the documented unavailable/no-update behavior while the journal trails live Postgres.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit bd6ec0f. Configure here.

@coderabbitai coderabbitai Bot added the codex label Jul 9, 2026
CI forbids console.* under supabase/functions; route DO logging through
cloudlog/cloudlogErr with serializeError like the rest of the backend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread supabase/functions/_backend/replicator/replicator.ts
Comment thread supabase/functions/_backend/replicator/replicator.ts
Comment thread supabase/functions/_backend/replicator/replicator.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot skipped on this run, and the latest completed Bugbot review still has 1 unresolved high-severity finding on the edge-replica seed path plus 1 unresolved medium finding; this hot-path infra change also exceeds the low-risk approval threshold. RobinWitch and WcaleNieWolny are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver External

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approving: Cursor Bugbot completed with status skipping on this sync, and the current head has 5 unresolved Bugbot findings (4 high-severity) on the edge-replica hot path. Human review is still required; RobinWitch and WcaleNieWolny are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@supabase/functions/_backend/replicator/app_replica.ts`:
- Around line 226-242: Seed failures currently clear immediately in
scheduleSeed(), so repeated unavailable() reads can continuously retry seed()
and storm Postgres. Add a persisted failure marker such as seed_failed_at in
replica_meta and check it at the start of scheduleSeed() to skip reseeding until
a backoff window has elapsed. Update the seeding flow around scheduleSeed(),
seed(), and the routerStub().unregister path so failures register the cooldown
while still leaving the replica unregistered until a successful seed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3e9b2932-d662-4a9c-bc9f-c7ff3fa8354d

📥 Commits

Reviewing files that changed from the base of the PR and between 1143e77 and f4654a4.

📒 Files selected for processing (17)
  • .gitignore
  • cloudflare_workers/plugin/index.ts
  • cloudflare_workers/plugin/wrangler.jsonc
  • cloudflare_workers/replicator/README.md
  • cloudflare_workers/replicator/index.ts
  • cloudflare_workers/replicator/wrangler.jsonc
  • package.json
  • supabase/functions/_backend/plugins/updates.ts
  • supabase/functions/_backend/replicator/app_replica.ts
  • supabase/functions/_backend/replicator/replicator.ts
  • supabase/functions/_backend/utils/cloudflare.ts
  • supabase/functions/_backend/utils/edge_replica.ts
  • supabase/functions/_backend/utils/edge_replica_schema.ts
  • supabase/functions/_backend/utils/pg.ts
  • supabase/functions/_backend/utils/update.ts
  • supabase/migrations/20260707150000_edge_replica_outbox.sql
  • tests/edge-replica.unit.test.ts
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • Cap-go/capacitor-updater (manual)

Comment thread supabase/functions/_backend/replicator/app_replica.ts
… round

- heartbeats only extend leases of targets fully caught up with the
  journal (cursor >= head); lagging targets refresh through successful
  pushes only, so a lease can never mask undelivered rows (Bugbot)
- age-based journal pruning first invalidates any target still behind the
  aged window so it reseeds instead of silently missing changes (Bugbot)
- org moves now reseed the replica entirely instead of patching routing
  keys: the new org's rows may not exist in the journal at all, and the
  updateOwnerOrg RPC raced the cursor advance (Bugbot); a move detected
  while replaying the seed window re-runs the seed with the fresh owner
- failed seeds set a 60s backoff so hot apps with a broken seed source
  answer "no update" without hammering Postgres (CodeRabbit)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@riderx

riderx commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Second round of findings fixed in 72ab3e6:

Bugbot — heartbeat extends lease while lagging: the maintenance heartbeat now only targets replicas whose cursor is at the journal head (cursor >= journal_head). A lagging target can only get its lease refreshed by a successful data push, so a lease always means "caught up as of grant".

Bugbot — journal prunes unconsumed rows: before age-pruning, any registered target whose cursor is still behind the aged window is unregistered and invalidated, so it reseeds on its next read instead of silently diverging forever. (In practice a target can't lag an hour — 3 failed pushes ≈ 15s already unregister it — but the guard makes it airtight.)

Bugbot — org rows skipped on move: rethought. Patching the routing key (updateOwnerOrg) was both racy against the cursor advance and insufficient — after a move the replica also needs the new org's current orgs/stripe_info rows, which may never appear in the journal. An org move now triggers a full reseed of that replica (unregister → snapshot with the new owner → re-register); reads answer "no update" during the reseed. A move detected while replaying the seed-window buffer re-runs the seed once with the fresh owner. Org moves are rare and per-app seeds are small, so this is the simple, airtight path. updateOwnerOrg is removed.

CodeRabbit — seed failure backoff: a failed seed sets seed_backoff_until (+60s) in the replica meta; not-ready reads inside the window answer "no update" without opening new Postgres connections.

Verified: typecheck, oxlint, unit tests, SQLite validation of the reader path, wrangler dry-run bundles, and the CI console-guard grep.

🤖 Generated with Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

There are 5 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.

if (lastRead > 0 && Date.now() - lastRead > IDLE_EVICT_MS) {
await this.wipe()
return { unregister: true }
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Router push idle evicts active replicas

Medium Severity

applyBatch wipes the replica and unregisters when last_read_at is older than seven days, but only device read RPCs update last_read_at. Router pushes and heartbeats that keep data fresh still trigger eviction, forcing reseeds and edge_replica_not_ready for apps with infrequent device check-ins.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.

this.scheduleSeed(movedToOrgApp)
return { unregister: true }
}
return { ok: true }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orphaned pending batches after seed

High Severity

Journal pushes that arrive after seedOnce replays pending_batches but before seedingPromise clears are appended to pending_batches and acknowledged by the router, yet nothing drains that table once seeded is set. Those changes never reach SQLite while reads can already pass isFresh(), so /updates_v2 may serve stale channel or plan data until a full reseed.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.

head,
Date.now(),
target.name,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Router cursor acks buffered seed pushes

Medium Severity

While a seed runs, applyBatch only buffers journal entries and returns success, yet pushTarget still advances the target cursor to the journal head. If seeding fails before replay, pending_batches is cleared on retry and those journal IDs are never applied to SQLite for that registration.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 72ab3e6. Configure here.

@sonarqubecloud

sonarqubecloud Bot commented Jul 9, 2026

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: high. Not approving: Cursor Bugbot completed with status skipping on this sync, and the current head has 5 unresolved Bugbot findings on the edge-replica hot path. Human review is still required; RobinWitch and WcaleNieWolny are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: high. Not approving: Cursor Bugbot completed with status skipping on this sync, and the current head has 5 unresolved Bugbot findings on the edge-replica hot path; this infra change also exceeds the low-risk approval threshold. RobinWitch and WcaleNieWolny are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Approver External

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
supabase/functions/_backend/replicator/app_replica.ts (2)

303-315: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Treat an empty pg_stat_subscription as unknown for seed sources that may be physical replicas. The configured seed source can be a read replica, so this query can report 0 even when replay is still behind; fall back to the outbox source, or check a real replay LSN before returning the seed client.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@supabase/functions/_backend/replicator/app_replica.ts` around lines 303 -
315, Treat the seed-source readiness check in `AppReplica` as unsafe when
`pg_stat_subscription` is empty, since a read replica can report zero rows even
while replay is still behind. Update the logic around the
`seedClient.query(...)` pending check to distinguish “no subscription rows” from
“caught up,” and only return the seed client when you have a real replay/LSN
confirmation. Otherwise fall back to the outbox source path that logs the
lagging seed source and snapshots from the outbox.

287-296: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Capture the watermark after registering the target. A write that lands between the current pg_current_wal_lsn() read and router.register() can already be in the router journal, but register() sets the replica cursor to journalHead(), so that row is never pushed. If the seed snapshot is still only caught up to the older watermark, the write can be missed entirely. Register first, then wait the seed source past a post-registration watermark or snapshot from the outbox source.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@supabase/functions/_backend/replicator/app_replica.ts` around lines 287 -
296, The seed replication flow in openSeedClient() is capturing the watermark
too early, which can miss writes that land before router.register() but after
the initial pg_current_wal_lsn() read. Update the ordering so the target is
registered first, then obtain a post-registration watermark or wait the seed
source past it from the outbox source, ensuring the replica cursor and seed
snapshot stay aligned. Use the existing openSeedClient(), router.register(), and
journalHead() flow to place the watermark/snapshot logic after registration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@supabase/functions/_backend/replicator/app_replica.ts`:
- Around line 187-188: The move/reseed logic in app_replica.ts does not handle
transitions where owner_org is null, so org→null or null→org changes can bypass
reseeding or leave stale metadata. Update the move detection around scheduleSeed
and the owner_org meta writes so changes are detected whenever the stored value
differs, not only when both sides are truthy, and ensure replica_meta is cleared
or rewritten appropriately in the reseed path. Check the related move handling
blocks near scheduleSeed and the metadata update code used for owner_org to keep
null transitions consistent.

In `@supabase/functions/_backend/replicator/replicator.ts`:
- Around line 449-455: The stranded-replica cleanup in `Replicator` is deleting
target rows before `replicaStub(target.name).invalidate()` has actually
completed, which lets pruning advance too early. Update the cleanup flow around
the stranded targets loop so the target row and journal window are only removed
after invalidation is confirmed, or otherwise make the pruning path block/retry
when `invalidate()` fails; keep the existing `ctx.waitUntil`/`replicaStub` logic
tied to successful invalidation rather than immediate deletion.

---

Outside diff comments:
In `@supabase/functions/_backend/replicator/app_replica.ts`:
- Around line 303-315: Treat the seed-source readiness check in `AppReplica` as
unsafe when `pg_stat_subscription` is empty, since a read replica can report
zero rows even while replay is still behind. Update the logic around the
`seedClient.query(...)` pending check to distinguish “no subscription rows” from
“caught up,” and only return the seed client when you have a real replay/LSN
confirmation. Otherwise fall back to the outbox source path that logs the
lagging seed source and snapshots from the outbox.
- Around line 287-296: The seed replication flow in openSeedClient() is
capturing the watermark too early, which can miss writes that land before
router.register() but after the initial pg_current_wal_lsn() read. Update the
ordering so the target is registered first, then obtain a post-registration
watermark or wait the seed source past it from the outbox source, ensuring the
replica cursor and seed snapshot stay aligned. Use the existing
openSeedClient(), router.register(), and journalHead() flow to place the
watermark/snapshot logic after registration.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 88e3e853-6abf-496d-a64d-2380cc108235

📥 Commits

Reviewing files that changed from the base of the PR and between f4654a4 and 72ab3e6.

📒 Files selected for processing (2)
  • supabase/functions/_backend/replicator/app_replica.ts
  • supabase/functions/_backend/replicator/replicator.ts
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • Cap-go/capacitor-updater (manual)

Comment on lines +187 to +188
this.ctx.storage.sql.exec(`DELETE FROM replica_meta WHERE key IN ('seeded', 'lease_until')`)
this.scheduleSeed(movedToOrgApp)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Handle org moves to or from NULL owner_org.

owner_org is read as string | null, but move detection requires both values to be truthy, and reseed only writes the meta when ownerOrg is truthy. Moves from org→null or null→org can skip the reseed path or leave stale owner_org metadata.

Proposed fix
-      this.ctx.storage.sql.exec(`DELETE FROM replica_meta WHERE key IN ('seeded', 'lease_until')`)
+      this.ctx.storage.sql.exec(`DELETE FROM replica_meta WHERE key IN ('seeded', 'lease_until', 'owner_org')`)
-      const ownerOrg = typeof entry.row.owner_org === 'string' ? entry.row.owner_org : null
-      const storedOrg = this.getMeta('owner_org')
-      if (entry.table === 'apps' && ownerOrg && storedOrg && ownerOrg !== storedOrg && typeof entry.row.app_id === 'string')
-        movedToOrgApp = entry.row.app_id
+      if (entry.table === 'apps' && typeof entry.row.app_id === 'string' && Object.prototype.hasOwnProperty.call(entry.row, 'owner_org')) {
+        const ownerOrg = typeof entry.row.owner_org === 'string' ? entry.row.owner_org : null
+        const storedOrg = this.getMeta('owner_org')
+        if (ownerOrg !== storedOrg)
+          movedToOrgApp = entry.row.app_id
+      }
-        if (ownerOrg)
-          this.setMeta('owner_org', ownerOrg)
+        if (ownerOrg)
+          this.setMeta('owner_org', ownerOrg)
+        else
+          this.ctx.storage.sql.exec('DELETE FROM replica_meta WHERE key = ?', 'owner_org')

Also applies to: 213-216, 392-393

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@supabase/functions/_backend/replicator/app_replica.ts` around lines 187 -
188, The move/reseed logic in app_replica.ts does not handle transitions where
owner_org is null, so org→null or null→org changes can bypass reseeding or leave
stale metadata. Update the move detection around scheduleSeed and the owner_org
meta writes so changes are detected whenever the stored value differs, not only
when both sides are truthy, and ensure replica_meta is cleared or rewritten
appropriately in the reseed path. Check the related move handling blocks near
scheduleSeed and the metadata update code used for owner_org to keep null
transitions consistent.

Comment on lines +449 to +455
const stranded = this.ctx.storage.sql
.exec('SELECT name FROM targets WHERE cursor < ?', agedMax.m)
.toArray() as { name: string }[]
for (const target of stranded) {
this.ctx.storage.sql.exec('DELETE FROM targets WHERE name = ?', target.name)
this.ctx.waitUntil(this.replicaStub(target.name).invalidate().catch(() => undefined))
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Do not prune rows until stranded replicas are actually invalidated.

Line 453 deletes the target and Line 454 only schedules invalidate() with waitUntil; Line 462 can then prune the journal immediately. If invalidation fails or races a read, the AppReplica can keep serving under its existing lease while unregistered and missing pruned rows. Keep the target/journal window until invalidation succeeds, or make pruning fail/retry when invalidation cannot be confirmed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@supabase/functions/_backend/replicator/replicator.ts` around lines 449 - 455,
The stranded-replica cleanup in `Replicator` is deleting target rows before
`replicaStub(target.name).invalidate()` has actually completed, which lets
pruning advance too early. Update the cleanup flow around the stranded targets
loop so the target row and journal window are only removed after invalidation is
confirmed, or otherwise make the pruning path block/retry when `invalidate()`
fails; keep the existing `ctx.waitUntil`/`replicaStub` logic tied to successful
invalidation rather than immediate deletion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants