feat(providers): DuckDB as an embedded file engine, with the boundary the engine enforces (#424) - #516
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
cevheri
force-pushed
the
feat/duckdb-provider
branch
from
August 27, 2026 15:26
d5a1de0 to
7cb6513
Compare
… the engine enforces (#424) Phase 6 of #424. DuckDB is the seventeenth type-id and the sixteenth external engine: a server-local `.duckdb` file or `:memory:`, through the official `@duckdb/node-api` 1.5.5-r.4 (DuckDB v1.5.5), loaded lazily inside `connect()` so no other engine pays for the 68 MB of platform bindings. Every finding below was MEASURED against a live embedded engine, not read from duckdb.org, and several of them refute what the documentation implies. **The read-only boundary is the engine's, not a word list.** `access_mode: 'READ_ONLY'` refuses writes to the attached database and is NOT a filesystem sandbox: `COPY ... TO`, `EXPORT DATABASE`, `INSTALL`, `LOAD`, `read_text`, `read_blob`, `glob`, `sniff_csv` and `read_csv_auto('/etc/hostname')` all measurably escape a genuinely read-only handle. A name denylist was written first and then broken three ways, each reproduced end to end through the real provider past the upstream statement guard: - `SELECT * FROM "read_text"('/etc/hostname')` returned the file, because `findCodeWord` skips quoted-identifier spans - correct for a keyword, wrong for a function name, since DuckDB resolves `"read_text"` exactly like `read_text`. - `SELECT * FROM '/tmp/x.csv'` has no word to see at all: the replacement scan turns a bare path into `read_csv_auto` by extension. - `json_execute_serialized_sql` carries a whole statement inside a string literal. So the read-only handle now opens with `enable_external_access: 'false'` beside `access_mode`, which refuses every form engine-side with `Permission Error: Cannot access file "..." - file system operations are disabled by configuration`, and cannot be turned back on at runtime. The SQL guard stays as defence in depth and says so; a test derives the file-reaching function list from `duckdb_functions()` so a future DuckDB shipping a new reader fails CI rather than silently reopening the hole. Measured facts that shaped the code: - **`getRowObjects()` is unusable** - it throws on `JSON.stringify` ("Do not know how to serialize a BigInt"). Everything goes through `getRowObjectsJson()`. BIGINT, HUGEINT and DECIMAL arrive as strings, so `DECIMAL(38,2)` survives to the grid exactly. - **A write is recognised by the ANSWER'S SHAPE, never by a leading keyword.** DuckDB's FROM-first syntax, `SUMMARIZE`, `PIVOT` and `CALL` all produce rows, and a statement's first word cannot tell you which. `SELECT 1 AS Count` therefore had to keep its grid while `INSERT` reports `rowsChanged` - both are pinned. - **`duckdb_schemas().internal` is TRUE for `main`** even in a user database, so the obvious `NOT internal` filter silently drops the default schema. Every catalog read is bounded by `database_name = current_database()` instead. - **`duckdb_tables().estimated_size` is a row-group ESTIMATE, not a count**: after `DELETE FROM big WHERE id < 19000000` on 20,000,000 rows it answered 1,076,480 where `count(*)` answered 1,000,000, and a CHECKPOINT left it there. The tree counts exactly instead - one UNION ALL over 41 tables including that one took 8.8 ms, because DuckDB answers `count(*)` out of row-group metadata. - **Per-table bytes are real, not estimated**: distinct persistent `block_id` from `PRAGMA storage_info` times `block_size`. Verified non-double-counting - 12 one-row tables occupied 12 distinct blocks with none shared. An in-memory or un-CHECKPOINTed table publishes NOTHING rather than a zero. - **`pragma_database_size()` returns human strings** ("2.0 MiB", "0 bytes") for every size but `block_size`, so they are parsed; `memory_limit` is host-dependent and is never asserted. - **A second PROCESS is refused even read-only** (`IO Error: Could not set lock on file`), which is stricter than duckdb.org's own summary and is why `singleWriterFile: true` is declared. Same-process second handles are permitted, and a same-process `READ_ONLY` reopen is genuinely read-only - which is what lets the agent's handle exist beside the writer's. - **`interrupt()` exists** on the connection prototype, so `cancelQuery` is implemented rather than deferred. - **`EXPLAIN (ANALYZE, FORMAT JSON)` answers `{"result": "error"}` AND executes the statement** (a probe table went 0 -> 1 rows). Only the estimating `EXPLAIN (FORMAT JSON)` is ever composed, permanently, whatever a later version does to that JSON. - **No slow-query log and no session list**: `duckdb_queries()` and `duckdb_connections() ` do not exist. Both panels answer empty with DuckDB's own words rather than a fabricated zero. `REINDEX`, `PRAGMA integrity_check` and `PRAGMA optimize` do not exist either, so of the six maintenance operations only `vacuum`, `analyze` and `optimize` (mapped to `CHECKPOINT`, global only) are offered - each with a spec, so #U9 cannot repeat. Agent support is real rather than declared: DuckDB reached `AGENT_EXECUTION_ENGINES` and needed the four catalog composers to match, or `inspect_schema`, `profile_table` and `inspect_plan` would have refused on every run while `POST /api/agent/runs` accepted the workflows. `duckdb_constraints()` publishes both sides of a foreign key as LISTS and DuckDB has no `WITH ORDINALITY`, so a composite key is paired by one shared ordinal - B8's cross-product on this engine's own catalog. `duckdb_indexes()` lists only written indexes, so constraint-backed ones are unioned in, which is B25's hole and the only primary-key information the agent path has. A test now asserts `AGENT_EXECUTION_ENGINES` is a subset of the composers' keys. Verified in Chrome against a live two-schema database: connect, browse (exact counts, the view carrying none), nested `VARCHAR[]` and `STRUCT` round-tripping to the grid, EXPLAIN, monitoring, and both agent modes. Plan mode grounded on the real schema and drafted SQL that then ran unchanged; agent mode executed two statements on its own read-only path in 3 ms and 8 ms and composed a report whose claims cite the artifacts. The browser also found what no gate could: D44, the Tables panel's per-row maintenance button sends `table.tableName` without the `table.schemaName` it renders beside it, so `ANALYZE "main"."events"` refused for a table in `analytics`. That is a shared component reaching all twelve providers that implement `runMaintenance`, so it is filed rather than fixed here; the provider's own contract - a qualified target works, a bare one for another schema is refused rather than guessed - is pinned by a test. D43 files the missing `sessionsEmptyState`, which is why DuckDB's Queries panel explains itself and its Sessions panel cannot. Packaging follows the better-sqlite3 triad: external in both build configs, copied explicitly in the Dockerfile and the standalone payload because tracing misses the `.node`, and the musl bindings pruned before linuxdeploy, which treats a musl ELF as fatal. Both Bun 1.3.14 and Node 24.14.0 load the addon.
cevheri
force-pushed
the
feat/duckdb-provider
branch
from
August 27, 2026 16:14
7cb6513 to
106fdbf
Compare
|
cevheri
added a commit
that referenced
this pull request
Aug 27, 2026
…ds a total (#517) * fix(db,ui,docs): an unknown size is absent, not zero, and a share needs a total Two providers stated both facts in one object. Trino paired `databaseSize: TRINO_UNAVAILABLE_TEXT` with `databaseSizeBytes: 0`, and the search seam paired `SEARCH_UNKNOWN_TEXT` with `sizeBytes ?? 0`, so one overview said "unavailable" in the string and "zero bytes" in the number from a single input. Both now omit the key, which is what the optional field is for, and a real zero still reads as zero. Apache Cassandra had already written the argument down. The search seam also held the last unconditional `activeConnections: 0`. It is absent now, and `getHealth()` stops composing a key it has no source for. Not the last zero of any kind: Trino, Druid, ClickHouse and Couchbase each degrade a refused monitoring read to no rows and then map the absent row to zero, which is the same encoding by a longer route and is filed rather than fixed here. The Storage tab is where the fabrications were visible, and three of its figures divided by a total it did not have. A share now requires `totalSize > 0`, which excludes the absent size and the measured zero alike, so the remainder bar is empty instead of full and the two Quick Stats percentages no longer sit under real megabyte figures. The remainder's BYTES are gated on the contradiction rather than on the share: a 0 total beside per-table reads that answered rendered a negative byte count, while `0 - 0 - 0` is an honest zero and stays one. A refused overview now carries the engine's own sentence, the way two sibling panels do. Verified end to end in Chrome against the SQLite sample: unpatched, the breakdown reads 504 + 288 + 4 = 796 KB with bars at -36.68% / -63.82% / -99.4975%; with the overview's size forced to 0 while the table read answers, the remainder reads N/A, all three bars are empty, and no percentage is drawn anywhere. The shared `formatBytes` refused nothing. Measured live: -1, NaN and both infinities all rendered "NaN undefined", and a petabyte rendered "1 undefined" - a number wearing a missing unit, which scans as a figure rather than as garbage. Non-magnitudes return "N/A", the string three other places in this codebase already use for a size they do not have; the ladder gains PB and EB; and the index is clamped, which makes indexing past the array unreachable rather than moving the boundary up two rungs. Also corrects three claims this repo made about itself. The Storage tab defines its own local byte formatter and never imports the shared one, so the negative remainder rendered "-1536 B" and not "NaN undefined" as mongodb.md, a mongodb.ts comment and the backlog entry all said - a review lane reported exactly that last round and was overruled by a measurement of the wrong function. The search docs' claim that `TableStats` cannot say "unknown" is false for its four optional size fields. And a new comment claiming the last zero anywhere was narrowed to the last unconditional one. * docs(backlog): three entries close, four open in their place D46 and U24 close outright. D44 closes three of its fourteen type-ids - the ones that contradicted themselves inside a single object - and its counts moved for a second reason: DuckDB arrived as a seventeenth type-id in #516 and already writes the key conditionally, so it is a fourth correct provider rather than a fifteenth fabricating one. The title now reads 11 of 17, the remaining-work list names all eleven files, and MongoDB is explicitly off that list because both its paths were already right. Four entries open, each measured rather than inferred: D50 - the search seam zeroes two OPTIONAL `TableStats` size fields for a closed index. Not fixed alongside the rest because omitting them takes the Data figure away from every open index in the cluster, through `tables.every(...)`, which is a visible panel change that wants its own decision rather than a drive-by. D51 - Trino, Druid, ClickHouse and Couchbase each swallow an unavailable monitoring surface into an empty result and then map the absent row to 0. ClickHouse's does it to six fields at once, so one refused read publishes a fully-zeroed overview that reads as measured. Filed as one entry because the fix is one decision applied four times: the degrade step already knows the difference and discards it before the mapper can act. Redis and PostgreSQL are named as NOT measured rather than folded in on resemblance. U25 - `formatBytes` exists three times, and the two component-local copies have no guard: measured, they render "-1 B", "NaN B", "Infinity GB" and an exabyte in gigabytes. U26 - the admin fleet total re-parses formatted strings and has no `tb` branch, so a 1 TB database contributes one byte, and this round's PB and EB rungs add two more spellings that land in the same arm. The real defect is that `FleetHealthItem` carries no numeric channel at all, which is also what makes an honest absence inexpressible there. The section index lines were recomputed from the surviving headings rather than merged. Both new ids had to move: this branch took D47 and D48 before #516 merged and published its own, and the bodies auto-merged without a conflict while only the index line reported one - the same mechanism as #513, caught this time by the guard #514 added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 6 of #424. DuckDB becomes the seventeenth type-id and the sixteenth external engine: a server-local
.duckdbfile or:memory:, through the official@duckdb/node-api1.5.5-r.4 (DuckDB v1.5.5), imported lazily insideconnect()so no other engine pays for the 68 MB of platform bindings.Every claim below was measured against a live embedded engine. Several refute what the documentation implies, and the prime reference is
docs/providers/duckdb.md.The read-only boundary is the engine's, not a word list
access_mode: 'READ_ONLY'refuses writes to the attached database and is not a filesystem sandbox. Measured escaping a genuinely read-only handle:COPY ... TO,EXPORT DATABASE,INSTALL,LOAD,read_text,read_blob,glob,sniff_csv,read_csv_auto('/etc/hostname').A name denylist was written first, then broken three ways — each reproduced end to end through the real provider, past the upstream statement guard:
SELECT * FROM "read_text"('/etc/hostname')findCodeWordskips quoted-identifier spans — correct for a keyword, wrong for a function name, since DuckDB resolves"read_text"exactly likeread_textSELECT * FROM '/tmp/x.csv'read_csv_autoby extension; there is no word to findjson_execute_serialized_sql(...)The read-only handle now opens with
enable_external_access: 'false'besideaccess_mode. Every form above is then refused engine-side (Permission Error: Cannot access file "..." - file system operations are disabled by configuration) and it cannot be re-enabled at runtime. The SQL guard remains as defence in depth and says so in its header; a test derives the file-reaching function list fromduckdb_functions(), so a future DuckDB shipping a new reader fails CI instead of silently reopening the hole.Measured facts that shaped the code
getRowObjects()is unusable — it throws onJSON.stringify("Do not know how to serialize a BigInt"). Everything goes throughgetRowObjectsJson(); BIGINT/HUGEINT/DECIMAL arrive as strings, soDECIMAL(38,2)reaches the grid exactly.SUMMARIZE,PIVOTandCALLall produce rows.SELECT 1 AS Countkeeps its grid;INSERTreportsrowsChanged. Both pinned.duckdb_schemas().internalis TRUE formaineven in a user database — the obviousNOT internalfilter silently drops the default schema. Reads are bounded bydatabase_name = current_database().estimated_sizeis an estimate, not a count. AfterDELETE FROM big WHERE id < 19000000on 20 M rows it answered 1,076,480 wherecount(*)answered 1,000,000, and aCHECKPOINTleft it there. The tree counts exactly instead — oneUNION ALLover 41 tables including that one took 8.8 ms, because DuckDB answerscount(*)from row-group metadata.block_idfromPRAGMA storage_info×block_size, verified non-double-counting (12 one-row tables → 12 distinct blocks, none shared). In-memory or un-CHECKPOINTed tables publish nothing, never a zero.IO Error: Could not set lock on file) — stricter than duckdb.org's summary, and whysingleWriterFile: trueis declared. Same-process handles are permitted, and a same-processREAD_ONLYreopen is genuinely read-only, which is what lets the agent's handle exist beside the writer's.interrupt()exists, socancelQueryis implemented rather than deferred.EXPLAIN (ANALYZE, FORMAT JSON)answers{"result":"error"}and executes the statement (a probe table went 0 → 1 rows). Only the estimating form is ever composed — permanently, whatever a later version does to that JSON.duckdb_queries()andduckdb_connections()do not exist, so neither panel fabricates a zero.REINDEX,PRAGMA integrity_checkandPRAGMA optimizedo not exist either, so onlyvacuum,analyzeandoptimize(→CHECKPOINT, global only) are offered, each with a spec so #U9 cannot repeat.Agent support is real, not merely declared
DuckDB reached
AGENT_EXECUTION_ENGINESand needed matching catalog composers, orinspect_schema,profile_tableandinspect_planwould have refused on every run whilePOST /api/agent/runshappily accepted the workflows.duckdb_constraints()publishes both sides of a foreign key as lists, and DuckDB has noWITH ORDINALITY— so a composite key is paired by one shared ordinal. That is B8's cross-product on this engine's catalog.duckdb_indexes()lists only written indexes, so constraint-backed ones are unioned in — B25's hole, and the only primary-key information the agent path has.AGENT_EXECUTION_ENGINESis a subset of the composers' keys, so the next engine cannot repeat this.Verification
Six local gates green, 100.00 % merged line coverage (45115/45115),
build:lib+attwclean, and the addon loads on both Bun 1.3.14 and Node 24.14.0.Driven in Chrome against a live two-schema database: connect · browse (exact counts, the view carrying none) · nested
VARCHAR[]andSTRUCTround-tripping to the grid · EXPLAIN · monitoring · both agent modes. Plan mode grounded on the real schema and drafted SQL that then ran unchanged; agent mode executed two statements on its own read-only path (3 ms, 8 ms) and composed a report whose claims cite the artifacts.The three highest-value new tests were each proven non-vacuous by reverting their fix and observing red.
Found in the browser, filed rather than fixed
D44 — the Tables panel's per-row maintenance button sends
table.tableNamewithout thetable.schemaNameit renders on the next line, so clicking Analyze onanalytics.eventsproduced:No gate could have caught it — both halves are correct in isolation. It is a shared component reaching all twelve providers that implement
runMaintenance, so fixing it is a twelve-engine verification, not a provider change. The provider's own contract (a qualified target works; a bare one for another schema is refused rather than guessed) is pinned by a test.D43 — there is no
sessionsEmptyState, which is why DuckDB's Queries panel explains itself and its Sessions panel cannot.Packaging
Follows the
better-sqlite3triad: external in both build configs, copied explicitly in the Dockerfile and the standalone payload because tracing misses the.node, and the musl bindings pruned before linuxdeploy, which treats a musl ELF as fatal.Closes nothing on its own; advances #424.