Fixes 32021: Attach SQL to native Unity Catalog lineage edges

[IceS2]

Describe your changes:

Fixes #32021

Native Unity Catalog lineage edges can currently appear without SQL even when the originating statement is available. Attach the latest readable statement for each source–target pair by joining system.access.table_lineage to system.query.history on statement and workspace IDs. Table edges and column mappings continue to come from native lineage.

Enrichment runs in batches of 100 eligible edges across targets, schemas, and catalogs. Missing or redacted statements preserve native lineage; a history access failure disables enrichment for the rest of the run. Document the optional history permissions and SQL availability limits.

Type of change:

• Bug fix

High-level design:

Batch only resolved edges that pass ingestion filters. Bind exact source–target pairs and return at most one SQL row per pair. This bounds enrichment batches and round trips, but does not bound warehouse scan bytes or change the existing native-lineage caches.

Tests:

• Added ingestion/tests/unit/topology/database/test_unitycatalog_lineage_sql.py: 38 cases covering SQL selection, workspace isolation, missing/redacted history, batch boundaries across catalogs and schemas, filters, access failures, external lineage, and SDK backfill/update/preservation of stored SQL.
• Relevant ingestion and SDK suite: 170 passed. The 38 new cases also passed in the deployed Airflow Python 3.12 environment.
• Coverage: 94% (34/36) of changed executable lines in lineage.py; full-module coverage is 68%. queries.py is 100%. The tests execute joins, ranking, and bindings against SQLite with catalog/date syntax adapted; Databricks dialect compilation was checked separately.
• Python formatting, lint, and scoped type checks passed. A mutation that flushed per target failed the batching regression as expected.
• Backend integration and Playwright tests: not applicable; no backend API or UI behavior changes. No automated live-Databricks integration test was added; live connector validation is described below.

Manual testing performed:

1. Created source/target tables in a Databricks SQL warehouse and populated the target with a standalone INSERT ... SELECT. Ingested metadata and native lineage through the local UI.
2. Compared deployed baseline and patched ingestion code, removing the test edge before the baseline rerun. Baseline recreated the native edge and column mappings without SQL; patched ingestion attached SQL.
3. Created 205 targets from five sources using a SQL loop. The ingestion processed 444 total edges without warnings or errors in approximately 85 seconds.
4. Queried the OpenMetadata API: all 444 edges had SQL, including all 410 source–target edges in the bulk fixture. Each bulk target had its expected two sources; those edges retained 615 column mappings.

The bulk fixture stored the same whole BEGIN ... END block on its edges, so it verifies SQL presence and native mappings, not distinct expanded SQL per target. Distinct per-edge assignment is covered by automated tests. Warehouse scan cost was not measured.

UI screen recording / screenshots:

Not applicable; only connector help text changed.

Checklist:

• PR title follows Fixes <issue-number>: <short explanation>.
• Linked issue above.
• Regression tests and verification results included.
• Connector documentation updated.
• Comments: existing code structure expresses the implementation; the test adapter documents its non-obvious constraint.
• JSON Schema migrations: not applicable; no schema changes.
• UI recording: not applicable; no UI behavior changes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved

Attaches SQL statements to native Unity Catalog lineage edges by joining system.access.table_lineage to system.query.history. Enrichment runs in batches of 100 eligible edges, with comprehensive test coverage (38 cases, 94% line coverage) and manual verification on live Databricks warehouses. No issues found.

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[IceS2]

Better approach here: #33141