Skip to content

feat: add DuckDB backend - #307

Draft
vincentsarago wants to merge 24 commits into
mainfrom
feat/duckdb-backend
Draft

vincentsarago wants to merge 24 commits into
mainfrom
feat/duckdb-backend

Conversation

@vincentsarago

@vincentsarago vincentsarago commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Note

Claude Code made this PR under the supervision of vincentsarago.

Adds a tipg.duck backend that serves DuckDB tables, views and files (GeoParquet, or any ST_Read format) as OGC Features and Tiles collections. They can be served alone or mixed with PostgreSQL collections in the same catalog. This builds on #304, where each collection holds its own connection.

Stacked on #313, which holds the non-DuckDB changes split out of this PR: PostgresCatalogSettings, the new pg_get_collection_index signature, and the geometry literal hooks. Merge #313 first; this PR's base will then move to main.

How it works

  • Connection (tipg/duck/database.py, connect_to_duckdb / close_duckdb_connection, stored on app.state.duckdb_connection):
    • The primary database is always in-memory.
    • The optional TIPG_DUCKDB_DATABASE file (path or URL) is attached read-only under its file name. Without it, only file sources are served.
    • File sources are registered as views in memory.main.
    • Views and settings must be visible from connection.cursor(), which opens a new connection. So the views are regular (not TEMP) and the settings use SET GLOBAL (geometry_always_xy = true, TimeZone = 'UTC').
  • DuckCollection (tipg/duck/collections.py):
    • It mirrors PgCollection: same validation and query structure, same ItemList / MVT output.
    • It holds a reference to the shared DuckDB connection, like PgCollection.connection_pool.
    • Queries run in a threadpool on a per-call cursor.
  • Catalog (duck_get_collection_index(*, connection, settings=None)):
    • Lists tables and views from duckdb_tables() / duckdb_views() / duckdb_columns(), with primary keys from duckdb_constraints().
    • Maps DuckDB types to the type names Column already understands.
    • Reads the SRID from the geometry type (GEOMETRY('EPSG:26915')).
    • Computes spatial and temporal extents in one scan per table.
    • Reuses TableSettings / TableConfig from tipg.settings.
  • CQL2: cql2's SQL output runs as-is on DuckDB. The only backend-specific part is how geometry literals get the column CRS. DuckCollection implements the abstract hooks added in refactor: PostgresCatalogSettings, simpler pg_get_collection_index and backend-specific geometry literal hooks #313 (_tag_geometry_literal / _reproject_geometry_literal) with ST_SetCRS / ST_Transform(g, 'EPSG:4326', 'EPSG:n'). This matters because DuckDB doesn't raise an error on mixed-CRS comparisons, it silently matches nothing.
  • Vector tiles: built with ST_AsMVTGeom / ST_AsMVT, with an ORDER BY inside the aggregate. Property types ST_AsMVT can't encode are cast: DECIMAL to DOUBLE, and dates, timestamps, lists, etc. to VARCHAR.

Packaging

The new tipg[duckdb] extra installs duckdb>=1.4 and pytz. DuckDB needs pytz to return TIMESTAMP WITH TIME ZONE values to Python. Tested with DuckDB 2.0.0.

Not included

  • No app module: tipg/main.py is unchanged. The new docs page shows a DuckDB-only app and a mixed PG + DuckDB app.
  • No SQL function layers for DuckDB.
  • Spatial extents use a full scan (DuckDB has no ST_EstimatedExtent); it can be disabled with TIPG_DUCKDB_SPATIAL_EXTENT=FALSE.

Also in this PR

plans/duckdb-backend.md is the implementation plan. A final "Implementation notes" section lists where the code differs from it.

Testing

  • tests/duck/: 17 tests against a generated .duckdb file plus a GeoParquet source:
    • catalog and extents, including a projected EPSG:26915 table and a table without a primary key;
    • collection bbox and datetime filters;
    • item filters: ids, bbox, datetime, properties, cql2 (IN, LIKE, S_INTERSECTS, including on the projected table);
    • sortby / properties, CSV / NDJSON / GeoJSON-seq / HTML output, single item (including a 404 for a non-integer id);
    • vector tiles: counts, casts, sort order, empty tiles;
    • an app mixing PostgreSQL and DuckDB collections;
    • an app with file sources only (no database file), and attaching the database through a file:// URL.
  • The full suite passes: 127 tests (110 existing PG + 17 new).
  • pre-commit run --all-files and mypy -p tipg --ignore-missing-imports (CI command) pass.
  • The two code examples in docs/src/advanced/duckdb.md were run against generated data and the test PostGIS database (not committed).
  • Not checked by hand: the HTML tile viewer.

🤖 Generated with Claude Code

github-actions[bot]

This comment was marked as off-topic.

vincentsarago and others added 14 commits November 10, 2026 15:20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mirrors PgCollection: same query structure, validation and output. Queries
run in a threadpool on a per-call cursor of the collection's DuckDB
connection; CQL geometry literals are tagged/reprojected with DuckDB's
ST_SetCRS / ST_Transform; MVT properties DuckDB can't encode are cast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
duck_get_collection_index lists the tables and views of all attached
databases (incl. file sources), normalizes DuckDB types to the names
tipg.collections.Column understands, reads the SRID from the geometry
column type and computes spatial/temporal extents in one scan per table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cover catalog/extents, items filters (ids, bbox, datetime, properties,
cql2 incl. projected CRS), sortby/properties, output formats, item,
vector tiles and an app mixing PostgreSQL and DuckDB collections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DuckDBSettings.database now defaults to None: the database file (path or
URL) is only attached when set, otherwise only file sources are served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the read_only setting: tipg only reads data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion_index keyword-only

connect_to_db / close_db_connection → connect_to_duckdb / close_duckdb_connection,
duck_get_collection_index(*, connection, settings=None); update docs and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vincentsarago
vincentsarago changed the base branch from main to refactor/pg-catalog-settings November 10, 2026 14:23
Base automatically changed from refactor/pg-catalog-settings to main November 10, 2026 14:31
@vincentsarago

Copy link
Copy Markdown
Member Author

Definitely not fast... but it works

Screenshot 2026-11-10 at 15 55 16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant