Skip to content

feat(clickhouse): support DataLake catalog source mappings - #6153

Draft
mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:feat/clickhouse-datalake-catalog-mapping
Draft

mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:feat/clickhouse-datalake-catalog-mapping

Conversation

@mday-io

@mday-io mday-io commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Description

ClickHouse DataLakeCatalog exposes tables through a two-level database.table namespace, while SQLMesh source references can use three-level catalog.schema.table names. The new datalake_catalog_mapping setting maps a source catalog to a ClickHouse database. For example, datalake_catalog_mapping: {lake: iceberg} rewrites parsed lake.analytics.events sources as iceberg."analytics.events" for reads, CTAS, INSERT ... SELECT, and replacement queries. The setting defaults to an empty mapping.

Direct reads preserve unmapped foreign catalogs. With a mapping configured, materialization queries reject unconfigured catalogs. The integration test verifies that CTAS rejects an unknown source before creating its target. Mapping applies to parsed query sources. Raw SQL strings and INSERT targets are unchanged. create_view still rejects a mapped source when a virtual catalog is injected.

Test Plan

  • DOCKER_CONTEXT=colima make clickhouse-datalake-up seeded three rows across two Iceberg snapshots on the local ARM64 stack.
  • SQLMESH_RUN_CLICKHOUSE_DATALAKE=1 pytest -n 0 -m "docker and clickhouse_datalake" tests/core/engine_adapter/integration/test_integration_clickhouse_datalake.py passed 19 cases with no skips. The run passed again after a clean DOCKER_CONTEXT=colima make clickhouse-datalake-down and DOCKER_CONTEXT=colima make clickhouse-datalake-up repeat.
  • An unauthenticated request to the fixture MinIO bucket returned HTTP 403 AccessDenied.
  • make style passed all four hooks.
  • make doc-test passed 29 tests.
  • PYTEST_XDIST_AUTO_NUM_WORKERS=4 make fast-test passed 2,933 tests with 13 skipped.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Signed-off-by: Michael Day <mdaytn@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant