Fixes #23228: enable column-level lineage between a metric and a table column - #33168
Fixes #23228: enable column-level lineage between a metric and a table column#33168mohityadav766 wants to merge 4 commits into
Conversation
…e column A metric has no columns of its own -- it *is* the leaf a column feeds (Total Sales = sum(Sales.Amount)). Let the metric act as its own single column-lineage endpoint, keyed by its own FQN, so the existing column-lineage pipeline (validation, search index, canvas edges, tracing, impact analysis) carries metric edges with no new concepts. LineageRepository.getChildrenNames previously short-circuited METRIC to an empty set, so validateLineageDetails silently dropped every columnsLineage entry touching a metric: the API answered 200 and stored nothing. The UI never offered the connection because EntityType.METRIC was missing from LINEAGE_COLUMN_NODE_SUPPORTED and had no children resolver. childrenCount stays 0 for metrics so the node keeps its compact label outside the column layer -- no "1 column" footer on every metric node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🔄 Playwright impact map auto-refreshedThis PR touched specs or UI source that changed the source→spec routing map. I regenerated What is this file? It is the auto-generated half of Playwright's PR planner. It routes "if source X changes, run specs Y" by walking spec imports and cross-referencing What if I want to regenerate locally instead? Run this before pushing your next change to skip the bot commit: python3 .github/scripts/generate_playwright_impact_map.py
git add .github/playwright/impact-map.generated.json
git commit --amend --no-edit # or a separate commit |
|
| Count | Rule |
|---|---|
| 1 | sonarjs/cognitive-complexity |
All findings
| Location | Rule | Message | |
|---|---|---|---|
| 🟡 | src/utils/EntityLineageNodeUtils.ts:266:3 |
sonarjs/cognitive-complexity |
Refactor this function to reduce its Cognitive Complexity from 16 to the 15 allowed. |
Fix locally (fast - only checks files changed in this branch):
make ui-checkstyle-changed
|
🚦 Removed from the merge queue —
|



Describe your changes:
Fixes #23228
Today you can link a metric to a table, but not to a column of that table — even though a metric is usually an expression over one column (
Total Sales = SUM(Sales.Amount)). The API accepted such an edge with a200and then silently stored nothing, and the UI never offered the connection at all.Why it was dropped:
LineageRepository.getChildrenNames()short-circuitedMETRICto an empty set with aLOG.info("Metric column level lineage is not supported").validateLineageDetails()filters everycolumnsLineageentry whose endpoint isn't in that set, so the mapping was discarded before it was written. On the UI sideEntityType.METRICwas absent fromLINEAGE_COLUMN_NODE_SUPPORTEDand had no children resolver, so a metric node had nothing to attach a column edge to.What I changed: a metric has no columns of its own — it is the leaf that a column feeds. So the metric now acts as its own single column-lineage endpoint, keyed by its own FQN.
{"fromColumns": ["…sales.amount"], "toColumn": "total_sales"}is exactly what an API client would naturally send, and it's the same shape the backend already uses for non-nested children (a dashboard's charts).Type of change:
High-level design:
Approach. Rather than introduce a parallel notion of "metric lineage", make the metric a first-class participant in the column-lineage machinery that already exists. Three small changes are enough because every downstream consumer is already generic over
columnsLineage:LineageRepository.getChildrenNames()case METRICreturnsCollections.singleton(fqn)instead of an empty setLINEAGE_COLUMN_NODE_SUPPORTEDEntityType.METRICEntityLineageNodeUtilsgetMetricEntityChildren— the node exposes itself as its one connectable childEverything else was already entity-type agnostic and needed no edits:
validateLineageDetails,EsLineageData.columns/ search indexing,ColumnFilterMatcher, the canvas edge geometry inCanvasUtils(it derives positions fromcolumnsInCurrentPages+getEntityChildrenAndLabel), column tracing, the "only show columns with lineage" filter, ELK node sizing, and the column-level Impact Analysis table (buildColumnLevelNodesForColumns, whoseFqn.split(col).pop()already renders a bare metric FQN as the metric name).childrenCountstays0. The resolver returns one item indatabutchildrenCount: 0. That is deliberate:childrenCountdrives the node footer ("N columns" badge + expand button), and a metric genuinely has zero columns. Keeping it at 0 means metric nodes look exactly as they do today in the default graph — the endpoint row only appears when the Column layer or edit mode is on, which is precisely when it's needed. Reporting1would add a "1 Metric" footer to every metric node in every lineage graph, a visual change nobody asked for.Collections.singletonoverSet.of.Set.of(null)throws;singleton(null)doesn't, and asingleton{null}rejects everytoColumn, which is the correct degenerate behaviour. This removes a defensivenullOrEmptybranch that was impossible to reach (references come fromgetEntityReferenceById, which always populates the FQN) and therefore impossible to test.Alternatives rejected.
onConnect,getUpdatedColumnsFromEdge,createColumnEdges,CanvasUtils.getColumnLineageCoordinatesand the tracing sets — five places instead of two, for the same user-visible result.measures/dimensionsas its children. Interesting for dbt/Cube-sourced metrics, but out of scope here and absent on the vast majority of metrics; the issue asks for the metric-as-a-whole link. Easy to layer on later since the resolver is the only place that would change.Backward compatibility / migration. None needed. No schema change (
columnLineagealready takes anyfullyQualifiedEntityName), so nomake generateand no SQL migration. Existing metric edges are untouched; the change only stops discarding data that clients were already permitted to send. Validation is unchanged in strictness —toColumn: "total_sales.not_a_column"is still filtered out (covered by a test).Tests:
Use cases covered
sales.amount(table column) to theTotal Salesmetric and the mapping is persisted rather than silently droppedUnit tests
openmetadata-service/src/test/java/org/openmetadata/service/jdbi3/LineageRepositoryTest.java— 3 new tests:…_TableColumnToMetric_KeepsColumnLineage,…_MetricToTableColumn_KeepsColumnLineage,…_UnknownMetricChild_IsFilteredOutopenmetadata-ui/.../src/utils/EntityLineageUtils.test.tsx— metric resolver returns the node as its own endpoint withchildrenCount: 0openmetadata-ui/.../NodeChildren/NodeChildren.component.test.tsx— metric node renders the endpoint rowmvn -pl openmetadata-service -Dtest=LineageRepositoryTest test→ 22/22 pass. Class-level jacoco is not a meaningful figure here (one test class against a 1040-line repository), so measuring the changed line:LineageRepository.java:1200is 4/4 instructions, 0 branches — 100% covered.Lineage.constants.ts100%,NodeChildren.component.tsx98.66% lines, and every line of the newgetMetricEntityChildren(EntityLineageNodeUtils.ts:127) is covered — the file's residual uncovered ranges are all pre-existing code elsewhere in it.LineageRepositorychange makes 2 of the 3 Java tests fail (expected: <1> but was: <0>); reverting the resolver makes both new Jest tests fail. They fail without the fix, which is the point.Backend integration tests
PUT /api/v1/lineagealready accepted this payload; the fix is in validation behaviour, covered by the unit tests above plus the manual API verification below.Ingestion integration tests
Playwright (UI) tests
openmetadata-ui/.../playwright/e2e/Pages/Lineage/DataAssetLineage.spec.ts—Column lineage between a table column and a metric: adds the edge, reloads to prove it persisted server-side, asserts the column-level Impact Analysis row, then removes itopenmetadata-ui/.../playwright/utils/lineage.ts—metricbranch ingetEntityColumns(a metric is its own column endpoint)metricto thecolumnLevelEntitiesN×N matrix, which would have added ~17 slow tests for code paths the other 8 entity types already cover.Manual testing performed
Full local stack built from this branch (
./docker/run_local_docker.sh -m ui -d mysql -s false -i false -r false), sample data created over the REST API — asalesfact table with anamountcolumn and atotal_salesmetric (Total Sales = SUM(sales.amount)), i.e. the issue's own example.PUT /api/v1/lineagewithcolumnsLineage: [{fromColumns: ["sales_demo.sales_db.public.sales.amount"], toColumn: "total_sales", function: "SUM"}]GET /api/v1/lineage/table/name/sales_demo.sales_db.public.sales→ mapping present in the relationship row (empty before this fix)GET /api/v1/lineage/getLineage?...→ same mapping present in the search index, i.e. what the graph readsamountinto the metric'sTotal Salesrowamount→ both the column row and the metric endpoint row takecustom-node-header-column-tracing, confirming a real column edge in the graph rather than just a node edgesales / amount → total_sales / total_sales, depth 1, withdata-row-key="sales_demo.sales_db.public.sales.amount->total_sales"PUTwithtoColumn: "total_sales.not_a_column"alongside a valid entry →200, and only the valid entry is storedUI screen recording / screenshots:
Column-level lineage graph —

sales.amount→Total Sales, both endpoints traced:Column-level Impact Analysis — the impact row the issue asked for:

Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.columnLineage.toColumnalready accepts anyfullyQualifiedEntityName.🤖 Generated with Claude Code