Exchange the concrete changed tables across processes

[asiergmorato]

Fixes #165.

This replaces the earlier counter-file version of this PR with the local-table approach suggested in review, since it is less code and lets us propagate the exact changed tables.

Problem

Databases opened at an absolute path (App Group container) share update notifications through a Darwin signal so writes in one process wake watch queries in others. That signal carries no payload, so every receiver re-emitted a catch-all EXTERNAL_CHANGES_MARKER that matches every table. Two consequences:

• Every process re-ran all of its watch queries on every write, including its own (N x M re-queries).
• It could sustain a loop. A write to table A self-delivers the catch-all, which re-runs a watch over an unrelated table B, whose handler writes A again. A device trace pinned this down: the attachment queue watches a synced table and writes its local attachments table, so with an App Group path each write woke it again roughly every 60ms and pinned the CPU (~348% measured). 998 of 1000 of those writes were immediately preceded by a self-delivered marker, so it was marker-driven, not a timer, and not the crud/upload loop. Without the shared path the same write only dispatches its own table, nothing re-fires, and it settles, which is why only App Group databases were affected.

Approach

Processes exchange the concrete changed tables through a local, non-synced ps_swift_updates(id, author, timestamp, tables) table:

• Each pool picks a random author. Before committing a write it reads the changed tables from the update hooks and records them in the same transaction, then posts the Darwin signal after the commit, so a change and its record are atomic and no second, separately-contended write is needed. The record runs in a SAVEPOINT so a bookkeeping failure never rolls back the user's write.
• Receivers read rows newer than their watermark and written by other authors, and re-emit exactly those tables. Ignoring their own author is what stops a process from waking itself, and the precise tables stop unrelated watches from re-running. EXTERNAL_CHANGES_MARKER remains only as a fallback when precise information may be incomplete (read error, undecodable payload, the table's ids regressed, or a read gap longer than the retention window, e.g. a suspended extension).
• Rows older than the retention window are pruned. Only tables another process can act on are recorded: ps_ is reserved for PowerSync, so anything else propagates, and of the internal tables only ps_data__* / ps_data_local__* (what watch matches) and ps_crud (what the upload loop matches) are kept. Local listeners still see every changed table.

Notable behaviour changes / tradeoffs

• execute(sql:parameters:) now runs its statement in a transaction so the record commits atomically with it. Statements that cannot run in a transaction (PRAGMA, VACUUM, ATTACH, DETACH, transaction control) keep using the plain write lock. This is shared with the GRDB backend (its interceptor is nil, so it just gets a transactional execute).
• The per-write recording and the signal apply to any absolute-path database, whether or not it is actually shared, since sharing cannot be detected from the path alone.
• Raw tables must not use a ps_ prefix, or their changes would be mistaken for internal bookkeeping and not propagate. That prefix is reserved for PowerSync; happy to add schema validation for it if you would prefer.

Open questions

• Should this recording ultimately live in the core update hook rather than the Swift write path? In the core it would be atomic and off the write-completion thread by construction, and would cover raw execute() without the write-path plumbing this needed.
• Is the ps_data__* / ps_data_local__* / ps_crud allowlist the complete set with a cross-process consumer, or is there an internal table I should also propagate?

[asiergmorato]

Code review

Self-review of the initial commit (7fe2382) surfaced 4 issues, all addressed in c75e457.

1. SharedProcessCounters.close() was not idempotent, risking a write through a reused file descriptor. fileDescriptor was a let left >= 0 after Foundation.close, and stop() reaches close() from both the explicit AsyncConnectionPool.close() and deinit, so the second call could flock/pwrite on a descriptor the OS had reassigned to an unrelated file.

powersync-swift/Sources/PowerSync/Implementation/SharedProcessCounters.swift

Lines 103 to 112 in 7fe2382

	func close() {
	lock.lock()
	defer { lock.unlock() }
	guard fileDescriptor >= 0 else { return }
	if let ownIndex, flock(fileDescriptor, LOCK_EX) == 0 {
	Self.writeOwner(fd: fileDescriptor, slot: ownIndex, pid: 0)
	_ = flock(fileDescriptor, LOCK_UN)
	}
	Foundation.close(fileDescriptor)
	}

Fixed: descriptor reset to -1 and slot cleared on close, second call is a no-op.

2. Blocking file I/O on the (typically main) thread. The counter table was opened, flocked, and probed with up to 64 kill() calls inside the synchronous init chain reached from the synchronous PowerSyncDatabase(...) initializer, so a peer holding the lock (a suspended widget) could stall app launch.

powersync-swift/Sources/PowerSync/Implementation/SharedProcessCounters.swift

Lines 43 to 72 in 7fe2382

	/// Opens (creating if needed) the counter table for `databasePath` and claims a slot.
	init(databasePath: String) {
	let path = Self.counterPath(for: databasePath)
	let directory = (path as NSString).deletingLastPathComponent
	if !FileManager.default.fileExists(atPath: directory) {
	try? FileManager.default.createDirectory(atPath: directory, withIntermediateDirectories: true)
	}
	let fd = open(path, O_RDWR | O_CREAT, 0o644)
	guard fd >= 0 else {
	self.fileDescriptor = -1
	self.ownIndex = nil
	return
	}
	self.fileDescriptor = fd
	var claimedIndex: Int?
	var initialOthersSum: UInt64 = 0
	if flock(fd, LOCK_EX) == 0 {
	Self.ensureInitialized(fd: fd)
	Self.reclaimDeadSlots(fd: fd)
	claimedIndex = Self.claimFreeSlot(fd: fd)
	if let claimedIndex {
	initialOthersSum = Self.sumCounters(fd: fd, excluding: claimedIndex)
	}
	_ = flock(fd, LOCK_UN)
	}
	self.ownIndex = claimedIndex
	self.lastOthersSum = initialOthersSum
	}

Fixed: moved to prepare(), called from start() on the background pool-open path; the baseline is still captured before the listener registers.

3. Silent fallback. Falling back to no-suppression (open failure or slot exhaustion) had no log, unlike the sibling Darwin-registration failure in start(), so a return of the storm would be undiagnosable. Fixed by passing the logger in and warning on both fallback paths.

4. Doc over-claimed "a change is never missed". Self-suppression assumes every writer participates in the table; a non-participating writer (fallback mode, or an older SDK version during an app-update window) does not bump a slot, so a suppressing sibling can miss its change until the next observed signal.

powersync-swift/Sources/PowerSync/Implementation/SharedProcessCounters.swift

Lines 19 to 21 in 7fe2382

	/// Everything degrades gracefully: any I/O problem, or exhausting the fixed number of slots,
	/// drops the process into a mode where it never suppresses (the previous behavior), so a
	/// change is never missed.

Fixed: documented the participation assumption honestly instead of over-claiming.

Also considered and dismissed: a PID-reuse slot leak (degrades gracefully to the pre-PR behavior) and a create/delete path canonicalization mismatch for the sidecar file (both paths resolve to the same inode, so no leak).

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[simolus3]

Thanks for following up on this! To reduce the amount of code needed for this feature, I wonder if we could use a local SQLite table instead of a second shared file.

Potentially, we could have a CREATE TABLE ps_swift_updates(author INTEGER, timestamp INTEGER, tables TEXT) on the database. Each PowerSync database generates a random value for author and inserts into ps_swift_updates before posting a Darwin notification. When receiving a notification, a process collects updated tables from other authors and keeps track of the highest rowid it has seen in that table to avoid ever seeing a notification twice. To prevent that table from growing too much, we could delete entries older than a few seconds before writing.

But I need to understand the actual loop in more detail first, since from my understanding even a uploadCrud loop should settle eventually when it runs out of things to do.

[asiergmorato]

You're right that uploadCrud settles on its own once ps_crud is empty and the $local target is resolved. The perpetuation doesn't come from crud piling up. It comes from the self-delivered catch-all marker re-triggering the upload machinery:

1. uploadLoop fires on ps_crud or EXTERNAL_CHANGES_MARKER.
2. Every iteration calls notifyCrudUploadComplete() → watchCompletedCrudUploads() dispatches localEvents(.completedUpload).
3. localEvents is merged into controlArgs, so .completedUpload runs powersyncControl(...), which is a db.writeTransaction. That commit registers affected ps_ tables via the update hooks.
4. dispatchWrites → handleUpdates → changeSignal.post() → the same process receives its own Darwin notification → EXTERNAL_CHANGES_MARKER → back to step 1.

The powersyncControl write only touches system ps_ tables, which no user watch matches, so user watches are woken solely by the catch-all marker, and that marker only exists because the post is delivered back to the sender. Remove the self-delivery and the loop opens: the process's own bookkeeping write no longer comes back as a catch-all that re-arms its own uploadLoop, so uploadLoop is driven only by real ps_crud changes and genuine external markers, and it settles. crudThrottle bounds the crud-completion cadence to ~1s (matching your crud upload: notify completion log); the faster watch cadence lines up with the extra precise writes and the second (widget) process, but the closure above is the necessary enabler in every case.

On the SQLite table side, it may be worth considering. It removes the second file and the whole mmap/flock/liveness primitive, and it lets us do something the current design cannot: carry the actual changed tables across processes.

[simolus3]

Is there a realistic test / repro in this PR that fails without these changes? I'm still surprised about this behavior and it sounds like there's something else worth fixing to break the loop.

localEvents is merged into controlArgs, so .completedUpload runs powersyncControl(...), which is a db.writeTransaction. That commit registers affected ps_ tables via the update hooks.

It only does that if there's something to write, specifically a pending checkpoint that we couldn't apply before due to outstanding local data (we'd try again after .completedUpload). So that call should either do nothing, or apply a checkpoint which then triggers one more iteration of the notify loop but not more.

So if you have a way to reproduce this, I'd love to step through that with a debugger to understand what's responsible for the continued writes.

[asiergmorato]

Repro / what's actually driving the loop

Got a device trace with per-process logging. You were right that it's not the crud / completedUpload loop (that shows up only ~68 times in a minute and doesn't perpetuate). The perpetuating write is the attachment queue writing its local table (ps_data_local__route_attachments), and the loop is closed by the self-delivered catch-all marker.

The cycle, repeating every ~60 ms, entirely in the main app:

[COMMIT] tables=["ps_data_local__route_attachments"]
[POST] -> notify_post -> [SIGNAL recv]   (self-delivery, same process, same ms)
[RECV] -> EXTERNAL_CHANGES_MARKER
   -> re-runs every watch, including the AttachmentQueue's watchAttachments
   -> the queue writes route_attachments again
[COMMIT] tables=["ps_data_local__route_attachments"]   (repeat)

998 of 1000 route_attachments commits are immediately preceded by a self-delivered marker, so it's marker-driven, not a timer.

Why it only happens with an App Group path: watchAttachments watches a synced table, while the queue writes the local route_attachments table. Single-process, that write dispatches precisely ["ps_data_local__route_attachments"], which doesn't match the attachment watch, so nothing re-fires. With the App Group path the same write self-delivers the catch-all EXTERNAL_CHANGES_MARKER, which matches every table, re-fires watchAttachments, and the queue writes again. So it isn't one component spinning; it's the catch-all marker crossing table boundaries, connecting a writer of table A to a watcher of an unrelated table B whose handler writes A. (It also re-runs unrelated watches, like our goals query, on every cycle.)

The ps_swift_updates approach

We prototyped your suggestion and it fixes this well in the real app: https://github.com/Chubby-Studio/powersync-swift/tree/experiment/ps-swift-updates-log

• Receivers read rows id > watermark AND author != self and re-emit the precise tables, so EXTERNAL_CHANGES_MARKER becomes only a gap/failure fallback. A process never wakes itself (the author filter replaces the counter file), and a route_attachments write never wakes a watch over a different table, so both the loop and the N x M amplification go away.
• The row is written inside the same transaction as the write (a before-commit hook on the write path), so it commits atomically with the data: no lost-change window and no separate contendible write on the completion thread.

One question for you

The trace also shows a lot of internal-only writes during normal sync (ps_buckets, ps_oplog, ps_tx, ps_stream_subscriptions, ...), each currently recording a row and posting a signal for no cross-process consumer (watches match user tables, the upload loop matches ps_crud). We could skip recording/posting when the changed tables are purely internal bookkeeping, but that needs the authoritative list of internal tables that never have a cross-process consumer (a prefix allowlist won't work because of raw tables). Which tables would you consider safe to never propagate cross-process?

If you prefer this direction, we'll switch this PR over to it.

[simolus3]

but that needs the authoritative list of internal tables that never have a cross-process consumer (a prefix allowlist won't work because of raw tables).

I actually think a prefix is likely fine if you make it a denylist, skip ps_ tables that aren't ps_data__ or ps_data_local__-prefixed or ps_crud. Raw tables should not have a ps_ prefix, that can reasonably be reserved for us.

Even though it adds a bit more complexity in lines of code from a quick look, I think the benefits of being able to track exact tables and not having to roll our own cross-process scheme kind of make me lean towards the approach based on the local table.

[asiergmorato]

Updated this PR to the local-table approach we discussed (force-pushed over the counter-file version, now a single commit). It exchanges the concrete changed tables through a local ps_swift_updates(id, author, timestamp, tables) table:

• Each pool has a random author; the changed tables are recorded inside the same transaction as the write, and the Darwin signal is posted after commit, so the record is atomic with the write.
• Receivers read rows newer than their watermark from other authors and re-emit the precise tables, so a process never wakes itself and unrelated watches don't re-run. EXTERNAL_CHANGES_MARKER stays only as a fallback (read error, undecodable payload, id regression, or a read gap past the retention window).
• Applied your prefix suggestion: only ps_data__* / ps_data_local__* / ps_crud and non-ps_ tables are recorded/signalled, so internal sync bookkeeping (ps_buckets, ps_oplog, ...) no longer generates cross-process chatter. Local listeners still get everything.

Full write-up is in the updated description. Two things I would value your take on:

1. Whether this recording should ultimately live in the core update hook instead of the Swift write path (it would be atomic and off the write-completion thread by construction, and cover raw execute() without the write-path plumbing).
2. Whether ps_data__* / ps_data_local__* / ps_crud is the complete set of internal tables with a cross-process consumer.

The diagnostic trace that pinned the loop (attachment queue + self-delivered catch-all) is summarized in the description under Problem.