feat(cdc): native SQLite CDC source (db.cdc.sqlite)#351
Open
skhaz wants to merge 1 commit into
Open
Conversation
Why: The runtime could stream row changes from Postgres (db.cdc.postgres) but had no equivalent for SQLite. SQLite has no logical-replication slot, so the native mechanism is the preupdate hook, which observes row changes on the connection the runtime writes through. What: Adds a db.cdc.sqlite registry kind backed by a supervised Source that installs SQLite preupdate + commit/rollback hooks on the target db.sql.sqlite pool's writer connection and emits insert/update/delete (plus a gap-free snapshot bootstrap) through the existing, engine-agnostic cdc Lua module (cdc.stream/list_sources/source). How: - service/sql: a build-tagged hook seam (sqlite_preupdate_hook) registers a ConnectHook-enabled driver and installs/clears hooks on a raw *sqlite3.SQLiteConn; the factory selects the driver transparently. - service/cdc/sqlite: the Source buffers preupdate rows per transaction and flushes atomically on commit via a bounded handoff to a drain goroutine. Column names/affinity resolve over a dedicated read-only connection, and checkpoints write through a separate plain-driver connection, so the writer-blocking commit hook can never deadlock against schema resolution or checkpoint writes. A laggard subscriber is closed rather than allowed to stall the writer. Durable snapshot/offset state lives in wippy_cdc_offsets in the source DB. - api/service/cdc: db.cdc.sqlite kind, SQLiteConfig, and a composite inspector/streamer so the Lua module observes both engines. - boot: kind-specific listeners (db.cdc.postgres + db.cdc.sqlite) feed the composite. Capture is in-process and live-only: changes made while the runtime is down, or by an external writer, are not captured. The checkpoint exists for snapshot-gating and idempotent dedupe, not replay. Building requires the sqlite_preupdate_hook tag (added to the Makefile); without it the source fails loudly instead of silently capturing nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The runtime streams Postgres row changes via
db.cdc.postgres(#323) but had no SQLite equivalent. SQLite has no logical-replication slot, so the native mechanism is the preupdate hook, which observes row changes on the connection the runtime writes through.What
Adds a
db.cdc.sqliteregistry kind backed by a supervisedSourcethat installs SQLite preupdate + commit/rollback hooks on the targetdb.sql.sqlitepool's writer connection and emitsinsert/update/delete(plus a gap-freesnapshotbootstrap) through the existing, engine-agnosticcdcLua module (cdc.stream/list_sources/source).service/sql: build-tagged hook seam (sqlite_preupdate_hook) registering aConnectHook-enabled driver; the factory selects it transparently (1-line change).service/cdc/sqlite: preupdate rows buffered per-transaction, flushed atomically on commit via a bounded handoff to a drain goroutine. Column names/affinity resolve over a dedicated read-only connection and checkpoints write through a separate plain-driver connection, so the writer-blocking commit hook can never deadlock against schema resolution or checkpoints. A laggard subscriber is closed loudly rather than allowed to stall the writer. Durable snapshot/offset state lives inwippy_cdc_offsetsin the source DB.api/service/cdc:db.cdc.sqlitekind,SQLiteConfig, and a composite inspector/streamer so the Lua module observes both engines.boot: kind-specific listeners (db.cdc.postgres+db.cdc.sqlite) feed the composite.Limitation (by design)
Capture is in-process and live-only: changes made while the runtime is down, or by an external process writing the file, are not captured (unlike a Postgres slot that replays). The checkpoint exists for snapshot-gating and idempotent dedupe, not replay. This is the trade for zero schema intrusion and lowest overhead.
Setup
Build with the new tag (already wired into the Makefile):
make build-wippy-local. Withoutsqlite_preupdate_hookthe source fails loudly instead of silently capturing nothing.Registry usage:
Lua:
local s = cdc.stream("app:cdc"); local c = s:channel():receive()->{ op, table, before, after, source, lsn }.Testing
golangci-lint(tagsrace,sqlite_preupdate_hook): 0 issues.-race): decode/affinity, subscribers, manager, composite, config; and against a real WAL SQLite file: insert/update/delete, value fidelity (blob/text/NULL), rollback-discard, snapshot bootstrap, restart-keeps-checkpoint, table allowlist, and laggard-subscriber-does-not-stall-writes.mode=roon WAL).cdc.stream-> Lua channel):