Skip to content

Fixes #23228: enable column-level lineage between a metric and a table column - #33168

Open
mohityadav766 wants to merge 4 commits into
mainfrom
metric-column-lineage-23228
Open

Fixes #23228: enable column-level lineage between a metric and a table column#33168
mohityadav766 wants to merge 4 commits into
mainfrom
metric-column-lineage-23228

Conversation

@mohityadav766

@mohityadav766 mohityadav766 commented Sep 10, 2026

Copy link
Copy Markdown
Member

Describe your changes:

Fixes #23228

Today you can link a metric to a table, but not to a column of that table — even though a metric is usually an expression over one column (Total Sales = SUM(Sales.Amount)). The API accepted such an edge with a 200 and then silently stored nothing, and the UI never offered the connection at all.

Why it was dropped: LineageRepository.getChildrenNames() short-circuited METRIC to an empty set with a LOG.info("Metric column level lineage is not supported"). validateLineageDetails() filters every columnsLineage entry whose endpoint isn't in that set, so the mapping was discarded before it was written. On the UI side EntityType.METRIC was absent from LINEAGE_COLUMN_NODE_SUPPORTED and had no children resolver, so a metric node had nothing to attach a column edge to.

What I changed: a metric has no columns of its own — it is the leaf that a column feeds. So the metric now acts as its own single column-lineage endpoint, keyed by its own FQN. {"fromColumns": ["…sales.amount"], "toColumn": "total_sales"} is exactly what an API client would naturally send, and it's the same shape the backend already uses for non-nested children (a dashboard's charts).

Type of change:

  • New feature

High-level design:

Approach. Rather than introduce a parallel notion of "metric lineage", make the metric a first-class participant in the column-lineage machinery that already exists. Three small changes are enough because every downstream consumer is already generic over columnsLineage:

Layer Change
LineageRepository.getChildrenNames() case METRIC returns Collections.singleton(fqn) instead of an empty set
LINEAGE_COLUMN_NODE_SUPPORTED add EntityType.METRIC
EntityLineageNodeUtils add getMetricEntityChildren — the node exposes itself as its one connectable child

Everything else was already entity-type agnostic and needed no edits: validateLineageDetails, EsLineageData.columns / search indexing, ColumnFilterMatcher, the canvas edge geometry in CanvasUtils (it derives positions from columnsInCurrentPages + getEntityChildrenAndLabel), column tracing, the "only show columns with lineage" filter, ELK node sizing, and the column-level Impact Analysis table (buildColumnLevelNodesForColumns, whose Fqn.split(col).pop() already renders a bare metric FQN as the metric name).

childrenCount stays 0. The resolver returns one item in data but childrenCount: 0. That is deliberate: childrenCount drives the node footer ("N columns" badge + expand button), and a metric genuinely has zero columns. Keeping it at 0 means metric nodes look exactly as they do today in the default graph — the endpoint row only appears when the Column layer or edit mode is on, which is precisely when it's needed. Reporting 1 would add a "1 Metric" footer to every metric node in every lineage graph, a visual change nobody asked for.

Collections.singleton over Set.of. Set.of(null) throws; singleton(null) doesn't, and a singleton{null} rejects every toColumn, which is the correct degenerate behaviour. This removes a defensive nullOrEmpty branch that was impossible to reach (references come from getEntityReferenceById, which always populates the FQN) and therefore impossible to test.

Alternatives rejected.

  • Allow a column→node drop, with the metric node itself as the edge target. Would need special-casing "node as column endpoint" in onConnect, getUpdatedColumnsFromEdge, createColumnEdges, CanvasUtils.getColumnLineageCoordinates and the tracing sets — five places instead of two, for the same user-visible result.
  • Expose the metric's semantic-layer measures / dimensions as its children. Interesting for dbt/Cube-sourced metrics, but out of scope here and absent on the vast majority of metrics; the issue asks for the metric-as-a-whole link. Easy to layer on later since the resolver is the only place that would change.

Backward compatibility / migration. None needed. No schema change (columnLineage already takes any fullyQualifiedEntityName), so no make generate and no SQL migration. Existing metric edges are untouched; the change only stops discarding data that clients were already permitted to send. Validation is unchanged in strictness — toColumn: "total_sales.not_a_column" is still filtered out (covered by a test).

Tests:

Use cases covered

  • A user links sales.amount (table column) to the Total Sales metric and the mapping is persisted rather than silently dropped
  • The reverse direction — a metric as the upstream of a table column — works symmetrically
  • A metric node renders a connectable endpoint row when the Column Level Lineage layer or edit mode is active, and stays compact otherwise
  • Clicking the source column traces through to the metric endpoint in the graph
  • The edge shows up in column-level Impact Analysis, which is the concrete gap the issue calls out ("I am not able to directly see visually the impact")
  • An invalid child under a metric FQN is still rejected by validation

Unit tests

  • I added unit tests for the new/changed logic.
  • Files updated:
    • openmetadata-service/src/test/java/org/openmetadata/service/jdbi3/LineageRepositoryTest.java — 3 new tests: …_TableColumnToMetric_KeepsColumnLineage, …_MetricToTableColumn_KeepsColumnLineage, …_UnknownMetricChild_IsFilteredOut
    • openmetadata-ui/.../src/utils/EntityLineageUtils.test.tsx — metric resolver returns the node as its own endpoint with childrenCount: 0
    • openmetadata-ui/.../NodeChildren/NodeChildren.component.test.tsx — metric node renders the endpoint row
  • Java: mvn -pl openmetadata-service -Dtest=LineageRepositoryTest test22/22 pass. Class-level jacoco is not a meaningful figure here (one test class against a 1040-line repository), so measuring the changed line: LineageRepository.java:1200 is 4/4 instructions, 0 branches — 100% covered.
  • UI: Lineage.constants.ts 100%, NodeChildren.component.tsx 98.66% lines, and every line of the new getMetricEntityChildren (EntityLineageNodeUtils.ts:127) is covered — the file's residual uncovered ranges are all pre-existing code elsewhere in it.
  • Regression-verified: reverting the LineageRepository change makes 2 of the 3 Java tests fail (expected: <1> but was: <0>); reverting the resolver makes both new Jest tests fail. They fail without the fix, which is the point.
  • No regressions: 26 lineage-related Jest suites / 434 tests pass.

Backend integration tests

  • Not applicable — no new or changed REST endpoint. PUT /api/v1/lineage already accepted this payload; the fix is in validation behaviour, covered by the unit tests above plus the manual API verification below.

Ingestion integration tests

  • Not applicable — no ingestion changes.

Playwright (UI) tests

  • I added a Playwright E2E test for the UI change.
  • Files updated:
    • openmetadata-ui/.../playwright/e2e/Pages/Lineage/DataAssetLineage.spec.tsColumn lineage between a table column and a metric: adds the edge, reloads to prove it persisted server-side, asserts the column-level Impact Analysis row, then removes it
    • openmetadata-ui/.../playwright/utils/lineage.tsmetric branch in getEntityColumns (a metric is its own column endpoint)
  • Deliberately a single targeted spec rather than adding metric to the columnLevelEntities N×N matrix, which would have added ~17 slow tests for code paths the other 8 entity types already cover.

Manual testing performed

Full local stack built from this branch (./docker/run_local_docker.sh -m ui -d mysql -s false -i false -r false), sample data created over the REST API — a sales fact table with an amount column and a total_sales metric (Total Sales = SUM(sales.amount)), i.e. the issue's own example.

  1. PUT /api/v1/lineage with columnsLineage: [{fromColumns: ["sales_demo.sales_db.public.sales.amount"], toColumn: "total_sales", function: "SUM"}]
  2. GET /api/v1/lineage/table/name/sales_demo.sales_db.public.sales → mapping present in the relationship row (empty before this fix)
  3. GET /api/v1/lineage/getLineage?... → same mapping present in the search index, i.e. what the graph reads
  4. Table → Lineage tab → Layers → Column → the edge is drawn from amount into the metric's Total Sales row
  5. Clicked amount → both the column row and the metric endpoint row take custom-node-header-column-tracing, confirming a real column edge in the graph rather than just a node edge
  6. Impact Analysis → Impact On: Column → row sales / amount → total_sales / total_sales, depth 1, with data-row-key="sales_demo.sales_db.public.sales.amount->total_sales"
  7. Negative path: PUT with toColumn: "total_sales.not_a_column" alongside a valid entry → 200, and only the valid entry is stored

Note: steps 1–7 were run against a build carrying the functionally identical Set.of(fqn) form of the one-line change; the later swap to Collections.singleton(fqn) has identical contains semantics for a non-null FQN and is covered by the 22 passing unit tests.

UI screen recording / screenshots:

Column-level lineage graphsales.amountTotal Sales, both endpoints traced:
column-lineage-graph

Column-level Impact Analysis — the impact row the issue asked for:
impact-analysis

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed. — no schema change; columnLineage.toColumn already accepts any fullyQualifiedEntityName.
  • For UI changes: I attached a screen recording and/or screenshots above.
  • I have added tests (unit / integration / Playwright as applicable) and listed them above.
  • The issue properly describes why the new feature is needed, what's the goal, and how we are building it.
  • I have updated the documentation. — N/A: no user-facing docs describe the per-entity-type column-lineage support matrix; behaviour is discoverable in the UI.
  • I have added tests around the new logic.

🤖 Generated with Claude Code

…e column

A metric has no columns of its own -- it *is* the leaf a column feeds
(Total Sales = sum(Sales.Amount)). Let the metric act as its own single
column-lineage endpoint, keyed by its own FQN, so the existing
column-lineage pipeline (validation, search index, canvas edges, tracing,
impact analysis) carries metric edges with no new concepts.

LineageRepository.getChildrenNames previously short-circuited METRIC to an
empty set, so validateLineageDetails silently dropped every columnsLineage
entry touching a metric: the API answered 200 and stored nothing. The UI
never offered the connection because EntityType.METRIC was missing from
LINEAGE_COLUMN_NODE_SUPPORTED and had no children resolver.

childrenCount stays 0 for metrics so the node keeps its compact label
outside the column layer -- no "1 column" footer on every metric node.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added backend safe to test Add this label to run secure Github workflows on PRs labels Sep 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Playwright impact map auto-refreshed

This PR touched specs or UI source that changed the source→spec routing map. I regenerated .github/playwright/impact-map.generated.json and pushed the diff to this branch.

- source entries: 759 → 759
- 0 added, 0 removed, 6 changed spec-list

Entries whose spec list changed:
  openmetadata-ui/src/main/resources/ui/playwright/support/entity/TableClass.ts
  openmetadata-ui/src/main/resources/ui/playwright/support/fixtures/base.ts
  openmetadata-ui/src/main/resources/ui/playwright/utils/common.ts
  openmetadata-ui/src/main/resources/ui/playwright/utils/domain.ts
  openmetadata-ui/src/main/resources/ui/playwright/utils/entity.ts
  openmetadata-ui/src/main/resources/ui/playwright/utils/tier.ts

What is this file? It is the auto-generated half of Playwright's PR planner. It routes "if source X changes, run specs Y" by walking spec imports and cross-referencing getByTestId strings. Hand-authored routing in impact-map.json always wins on conflict.

What if I want to regenerate locally instead? Run this before pushing your next change to skip the bot commit:

python3 .github/scripts/generate_playwright_impact_map.py
git add .github/playwright/impact-map.generated.json
git commit --amend --no-edit  # or a separate commit

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ UI Checkstyle passed — lint findings in changed files

🔍 ESLint findings in this PR's files — 0 error(s), 1 warning(s)

Errors block the build. Warnings do not yet — they are rules whose backlog is still
being worked down, listed so this PR does not add to it. See docs/ui-code-quality-gate.md.

0 error(s), 1 warning(s) across 1 changed file(s).

Count Rule
1 sonarjs/cognitive-complexity
All findings
Location Rule Message
🟡 src/utils/EntityLineageNodeUtils.ts:266:3 sonarjs/cognitive-complexity Refactor this function to reduce its Cognitive Complexity from 16 to the 15 allowed.

Fix locally (fast - only checks files changed in this branch):

make ui-checkstyle-changed

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 72%
72.35% (98600/136275) 56.84% (58068/102158) 58.11% (19415/33409)

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

🚦 Removed from the merge queue — merge_conflict (2026-09-11T11:30:52Z)

The entry left the queue before it was built, so no checks ran against it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable lineage between a metric and table column

5 participants