Skip to content

Fixes 24764: add SQL-based lineage for SAP HANA, including Cloud - #33144

Open
mohittilala wants to merge 10 commits into
mainfrom
feat/saphana-cloud-lineage
Open

Fixes 24764: add SQL-based lineage for SAP HANA, including Cloud#33144
mohittilala wants to merge 10 commits into
mainfrom
feat/saphana-cloud-lineage

Conversation

@mohittilala

Copy link
Copy Markdown
Contributor

Describe your changes:

Fixes #24764

SaphanaLineageSource read exactly one thing, CDATA XML from _SYS_REPO.ACTIVE_OBJECT. That schema is the classic HANA repository, deprecated since 2018 and never carried into HANA Cloud, so a Cloud lineage run produced zero edges. It also inherited the bare Source step rather than the shared LineageSource, so no HANA deployment has ever had view, query or stored-procedure lineage.

I verified the scope on a live HANA Cloud instance: _SYS_REPO is absent, and all 969 views are VIEW_TYPE = ROW. There is not one calculation, analytic or attribute view, which are the only three types the CDATA parser reads. The existing strategy cannot apply to Cloud at all.

This inherits LineageSource, so view definitions produce column-level lineage and the plan cache produces table-to-table edges. The CDATA pass is unchanged and still serves repository models on on-prem instances. The two passes read different object kinds, so no edge has two origins.

Stored-procedure lineage stays unsupported. It needs StoredProcedureLineageMixin, which is deliberately not mixed in here, so that remains a follow-up.

Type of change:

  • Improvement

High-level design:

SaphanaLineageSource now extends SapHanaQueryParserSource and LineageSource, matching how roughly twenty other connectors are built. _iter runs the shared SQL passes and then the repository pass, each behind its existing sourceConfig flag.

I also evaluated SYS.OBJECT_DEPENDENCIES, HANA's own dependency catalog, and rejected it. Measured against a purpose-built fixture, the SQL parser recovered the same upstream tables for 6 of 6 views and added column-level lineage the catalog cannot express, so the catalog contributed nothing unique and would have produced a competing second set of edges over the same views.

One platform limit worth stating. CREATE TABLE ... AS SELECT lineage is not recoverable on SAP HANA. OBJECT_DEPENDENCIES records derivation rather than data movement, and the plan cache holds execution plans, so DDL never enters it. I confirmed directly that the instance had zero cached CREATE statements. INSERT INTO ... SELECT and the other DML forms are covered.

Tests:

Use cases covered

  • A SAP HANA Cloud service, which has no _SYS_REPO, produces view and column-level lineage where it previously produced none.
  • Joins, chained views, unions and quoted hyphenated names all resolve to the correct upstreams.
  • INSERT INTO ... SELECT produces a table-to-table edge with column lineage.
  • A run that produces no edges says so, and names what to check.
  • On-prem repository lineage is unchanged.

Unit tests

  • I added unit tests for the new/changed logic.
  • Files updated: ingestion/tests/unit/topology/database/test_sap_hana.py, 4 added and 1 updated for the new base class. 42 pass.

Backend integration tests

  • Not applicable, no backend API changes.

Ingestion integration tests

  • Not applicable. There is no HANA fixture in the integration suite, so this was verified against a live HANA Cloud instance instead, below.

Playwright (UI) tests

  • Not applicable, no UI changes.

Manual testing performed

Verified against a live SAP HANA Cloud instance (HANA 4.00.000.00) using a purpose-built fixture with a known lineage graph: 3 base tables, 6 views covering join, aggregate, chained, union and hyphenated-name shapes, plus a CTAS table, an INSERT INTO ... SELECT, and a stored procedure.

  1. Confirmed the failure condition: no _SYS_REPO, and every view on the instance is VIEW_TYPE = ROW.
  2. Ran the SQL parser over each fixture view definition. 6 of 6 matched their expected upstream tables, with no parse errors, producing 2 to 4 column pairs each.
  3. Ran the shipped query-history statement against the live plan cache. It returned the expected 8 columns and recovered LT_ORDER -> LT_ORDER_ARCHIVE with 3 column pairs.
  4. Confirmed the CTAS limitation is a platform constraint, not a gap in the filter: the plan cache contained zero CREATE statements.

One parser gap observed and not fixed here: the UNION ALL view resolved both upstream tables correctly but produced no column pairs, while every other shape produced them.

UI screen recording / screenshots:

Not applicable.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: not applicable, no schema changes.
  • For UI changes: not applicable.
  • I have added tests (unit / integration / Playwright as applicable) and listed them above.
  • I have added tests around the new logic.
  • For connector/ingestion changes: I updated the documentation.

The SAP HANA lineage source only read _SYS_REPO.ACTIVE_OBJECT, which is the
classic repository and does not exist on HANA Cloud. Cloud runs therefore
completed with zero lineage edges once the crash guard landed in #31357.

Read HANA's own dependency catalog instead, which is present on every
deployment. Split _iter into one pass per source so the repository pass and
the dependency pass sit alongside their own helpers.
SaphanaLineageSource read only _SYS_REPO.ACTIVE_OBJECT and inherited the bare
Source step. _SYS_REPO does not exist on SAP HANA Cloud, so Cloud produced no
lineage at all, and no deployment ever got view, query or stored-procedure
lineage.

Inherit LineageSource so view definitions give column-level lineage and the plan
cache gives table-to-table edges. The CDATA pass stays for repository models,
which are XML rather than SQL, so the two passes never describe the same object.
yield_procedure_lineage is a no-op without StoredProcedureLineageMixin, which
this source does not mix in, so the log line and docstring were describing
lineage that never gets produced.
Copilot AI lite review requested due to automatic review settings September 10, 2026 13:36
@mohittilala
mohittilala requested review from a team and akashverma0786 as code owners September 10, 2026 13:36
@mohittilala mohittilala added the safe to test Add this label to run secure Github workflows on PRs label Sep 10, 2026
@mohittilala mohittilala self-assigned this Sep 10, 2026
Comment thread ingestion/src/metadata/ingestion/source/database/saphana/lineage.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved lineage pass isolation, lifecycle, filtering, and test coverage findings remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds SQL-based view and query lineage for SAP HANA Cloud while retaining legacy _SYS_REPO CDATA lineage for on-premises deployments.

Changes:

  • Integrates shared SQL lineage processing with the existing repository pass.
  • Adds SAP HANA plan-cache lineage queries and DML filtering.
  • Expands unit tests and updates the Ruff baseline.
File summaries
File Summary and review findings
ingestion/tests/unit/topology/database/test_sap_hana.py Adds SAP HANA lineage tests. Moderate (3): fix the iter([]) short-circuit. Nit (1): add end-to-end SQL-path assertions, including missing-schema handling.
ingestion/src/metadata/ingestion/source/database/saphana/query_parser.py Adds the SAP HANA query-parser base class.
ingestion/src/metadata/ingestion/source/database/saphana/queries.py Adds the plan-cache lineage query.
ingestion/src/metadata/ingestion/source/database/saphana/lineage.py Combines SQL and CDATA lineage. Critical (2): isolate SQL-history failures from the repository pass. Moderate (3): delegate to super().close(). Moderate (2): gate CDATA lineage with processViewLineage. Moderate (1): support leading whitespace/comments in DML matching. Nits (2, 1): correct edge-count reporting and clarify stored-procedure support.
ingestion/.ruff-g004-baseline.json Removes resolved Ruff findings.
Review details

Suppressed comments (3)

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:76

  • [P1] Do not anchor DML matching at the first character of STATEMENT_STRING. These LIKE patterns only match text that starts exactly with INSERT/UPSERT/etc., so valid plan-cache entries with leading whitespace or comments are excluded and produce no lineage. Prefix the patterns with a wildcard (as the other lineage connectors do) or normalize the statement before filtering.
            UPPER(STATEMENT_STRING) LIKE 'INSERT INTO%%SELECT%%'
            OR UPPER(STATEMENT_STRING) LIKE 'UPSERT%%SELECT%%'
            OR UPPER(STATEMENT_STRING) LIKE 'REPLACE%%SELECT%%'
            OR UPPER(STATEMENT_STRING) LIKE 'MERGE INTO%%'
            OR UPPER(STATEMENT_STRING) LIKE 'UPDATE%%SET%%'

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:132

  • This message says stored-procedure lineage is “unaffected,” but this source deliberately does not implement StoredProcedureLineageMixin and LineageSource.yield_procedure_lineage is a no-op. On Cloud, that wording can imply stored-procedure lineage is supported; please remove that claim or state explicitly that it remains unsupported.
                    "SAP HANA Cloud, where the classic repository was never carried over. View, query and "
                    "stored-procedure lineage are unaffected. Cause: %s",

ingestion/tests/unit/topology/database/test_sap_hana.py:1527

  • These additions validate inheritance and text fragments, but never run the connector’s new SQL path and assert an emitted edge: test_iter_runs_both_passes mocks both passes, and the other tests only inspect class attributes. A wrong HANA alias, plan-cache row mapping, or _SYS_REPO fallback would still pass. Add a test that feeds representative view/query rows through the source and asserts AddLineageRequest output, including the 362 missing-schema case.
def test_iter_runs_both_passes() -> None:
    """_iter must run the shared SQL passes and the repository pass.

    Dropping either one silently halves lineage on the deployment that depends on it.
    """
  • Files reviewed: 5/5 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread ingestion/src/metadata/ingestion/source/database/saphana/lineage.py
Comment thread ingestion/src/metadata/ingestion/source/database/saphana/lineage.py
Comment thread ingestion/src/metadata/ingestion/source/database/saphana/lineage.py Outdated
Comment thread ingestion/tests/unit/topology/database/test_sap_hana.py Outdated
Comment thread ingestion/src/metadata/ingestion/source/database/saphana/lineage.py Outdated
_iter passes through whatever the shared framework yields, which includes
CreateQueryRequest from query lineage, so declaring only AddLineageRequest was
wrong and the edge counter was reporting query records as edges.

Count the two apart and split the zero-edge warning, since queries read but
unresolved points at metadata coverage while nothing read at all points at a
missing CATALOG READ grant.
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit 7b2d7931c19dd434383bcf922f19ef749c66034a in Playwright run 34568233959, attempt 1.

✅ 108 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 44m 48s

⏱️ Max setup 4m 55s · max shard execution 14m 19s · max shard-job elapsed before upload 19m 57s · reporting 7s

🌐 235.44 requests/attempt · 1.80 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

  • Browser traffic was 235.44 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 1.8 per UI scenario (225 boots / 125 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 46 0 0 0 0 0
✅ Shard ingestion-01 27 0 0 0 0 0
🟡 Shard ingestion-02 35 0 1 0 0 0
🟡 1 flaky test(s) (passed on retry)
  • Features/IncidentManager.spec.tsComplete Incident lifecycle with table owner (shard ingestion-02, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

…ndings

A failure in the shared SQL pass propagated out of _iter, so an unreadable
SYS.M_SQL_PLAN_CACHE stopped the repository pass and cost on-premise instances
the calculation-view lineage they already had.

Delegate close to the base so masked_query_cache is cleared, gate the repository
pass on processViewLineage since calculation views are views, and drop the %%
escaping that hdbcli never unescapes.
… findings

Cached statements keep the whitespace they were submitted with, so a filter
anchored at character one dropped real DML. Match on the trimmed statement and
keep the anchor, since a leading wildcard also matches a SELECT that merely
quotes the keyword.

Drop the last claim that stored-procedure lineage is unaffected, and add tests
that resolve a real plan-cache statement rather than only inspecting class
attributes.
database_field and schema_field are declared on QueryParserSource but never read by it, and no sibling query parser sets them. The module logger was unused too.

The filter and _iter comments had grown to restate each other, so they are cut back to the parts that are not obvious from the code.
Copilot AI review requested due to automatic review settings September 10, 2026 16:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate review findings remain in the SQL lineage path.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:71

  • The new Cloud lineage path is not exercised here: test_iter_runs_both_passes replaces LineageSource._iter, and the parser test bypasses yield_table_query/query_lineage_processor entirely. A regression in the HANA row aliases, NULL AS database_name/SCHEMA_NAME mapping, or the actual SQL-row-to-lineage emission would therefore pass these tests; please add a focused test that feeds a representative plan-cache row through yield_table_query and asserts the emitted lineage/query request.
    sql_stmt = SAPHANA_QUERY_HISTORY_STATEMENT

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:153

  • sql_queries is incremented only for CreateQueryRequest, which the shared processor emits after it has already found a lineage result. A plan-cache row whose SQL parses but has no resolvable endpoint therefore takes this branch and the log incorrectly says HANA “read no queries,” even though rows were read; that sends operators toward the wrong diagnosis. Either count raw TableQuery rows or change this message to describe the absence of emitted query-lineage records rather than the absence of reads.
            logger.warning(
                "SAP HANA lineage finished with no edges and read no queries. Check that the metadata "
                "workflow has already ingested the tables and views, that processViewLineage or "
                "processQueryLineage is enabled, and that the ingestion user holds CATALOG READ, without "
                "which SYS.M_SQL_PLAN_CACHE only returns the ingestion user's own statements."

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:85

  • The predicate is anchored after whitespace trimming, but it still rejects valid plan-cache statements that begin with a SQL comment, such as /* job */ INSERT INTO ... SELECT .... The shared parser explicitly accepts comment-prefixed DML (for example, test_query_parser.py:189-195), and this query already handles comment-prefixed applications below, so those HANA lineage statements are silently omitted. Normalize leading comments before applying the anchored DML test, and add a regression case for both block and line comments.
    # Matched against the trimmed statement, because cached statements keep the
    # whitespace they were submitted with. Still anchored, since a leading wildcard
    # also matches a SELECT that merely quotes the keyword.
    #
    # CREATE TABLE ... AS SELECT is missing by necessity: the plan cache holds no DDL.
    filters = f"""
        AND (
            {_STATEMENT} LIKE 'INSERT INTO%SELECT%'
            OR {_STATEMENT} LIKE 'UPSERT%SELECT%'
            OR {_STATEMENT} LIKE 'REPLACE%SELECT%'
            OR {_STATEMENT} LIKE 'MERGE INTO%'
            OR {_STATEMENT} LIKE 'UPDATE%SET%'
        )

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:115

  • This catch preserves the repository pass, but it consumes every SQL lineage exception without recording a StackTraceError or yielding an error. Since Source._iter then sees normal generator completion, a HANA Cloud run with a denied or failed SYS.M_SQL_PLAN_CACHE query can report success while producing no SQL lineage. Record the failure in self.status before continuing (or re-raise after the fallback) so the workflow exposes the SQL-pass failure.
        except Exception as exc:
            # yield_table_query does not guard its own execute, so an unreadable plan
            # cache raises here and would otherwise take the repository pass with it.
            logger.warning(
                "SAP HANA SQL lineage pass failed and produced %d edges before stopping. The repository "

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:109

  • The shared SQL lineage path does not emit direct AddLineageRequest objects for these cases: query parsing returns OMetaFQNLineageRequest, and view lineage is wrapped in OMetaLineageRequest. As a result sql_edges remains zero on the normal Cloud path even when edges are yielded, so the final warning falsely says no edges were produced (and, when CreateQueryRequest is counted, falsely blames unresolved endpoints). Count the framework's lineage wrapper types or the underlying lineage request instead.
                if isinstance(either.right, AddLineageRequest):
                    sql_edges += 1
                elif isinstance(either.right, CreateQueryRequest):
                    sql_queries += 1

ingestion/tests/unit/topology/database/test_sap_hana.py:1564

  • The new SQL-path tests validate the query text and standalone parser, but test_iter_runs_both_passes mocks LineageSource._iter, so it never executes HANA's yield_table_query, the plan-cache row-to-TableQuery mapping, or the processor that emits AddLineageRequest/CreateQueryRequest. A regression in the actual connector wiring could therefore pass all these tests. Add a focused test with a representative plan-cache row that runs the real SQL path and asserts the emitted lineage request (including the HANA schema/database context).
def test_iter_runs_both_passes() -> None:
    """_iter must run the shared SQL passes and the repository pass.

    Dropping either one silently halves lineage on the deployment that depends on it.
    """
    calls = []

    def record_sql(*_, **__):
        calls.append("sql")
        return iter([])

    def record_cdata():
        calls.append("cdata")
        return iter([])

    with (
        patch.object(LineageSource, "_iter", side_effect=record_sql),
        patch.object(SaphanaLineageSource, "yield_cdata_lineage", side_effect=record_cdata),
        patch.object(SaphanaLineageSource, "test_connection"),
        patch("metadata.ingestion.source.database.query_parser_source.get_ssl_connection"),
    ):
        source = SaphanaLineageSource(
            config=WorkflowSource(
                type="saphana-lineage",
                serviceName="test_sap_hana",
                serviceConnection=DatabaseConnection(
                    config=SapHanaConnection(
                        connection=SapHanaSQLConnection(username="test", password="test", hostPort="localhost:39015")
                    )
                ),
                sourceConfig=SourceConfig(config=DatabaseServiceQueryLineagePipeline()),
            ),
            metadata=create_autospec(OpenMetadata),
        )
        list(source._iter())

    assert calls == ["sql", "cdata"]
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread ingestion/src/metadata/ingestion/source/database/saphana/lineage.py Outdated
The guard sat around the whole shared _iter, so a view-pass failure was logged as a plan-cache failure and swallowed. On Cloud the view pass is the entire result, so that turned a real error into a silent empty run.

Override yield_query_lineage instead, which is the pass that actually reads the plan cache, and let everything else propagate.
Copilot AI review requested due to automatic review settings September 11, 2026 05:50
…rk emits

The shared passes wrap lineage in OMetaLineageRequest and OMetaFQNLineageRequest rather than yielding AddLineageRequest, so the edge counter stayed at zero and the no-edges warning fired on every successful run.

Strip leading comments before the DML match, since tools routinely prefix statements, and surface a query-history failure on the workflow status instead of only logging it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved issues affect edge detection, exception handling, comment-prefixed DML, query-duration units, and representative plan-cache test coverage.

Review details

Suppressed comments (5)

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:108

  • The shared lineage framework does not emit AddLineageRequest for these SQL passes: view edges are wrapped as OMetaLineageRequest and query edges as OMetaFQNLineageRequest. Consequently sql_edges remains zero even when SQL lineage was produced, so _iter logs the misleading “no edges” warning (and can take the no-edge path) on successful Cloud runs. Count any non-CreateQueryRequest successful result instead of checking only AddLineageRequest.
        records are counted apart, since the shared passes emit both.
        """
        sql_edges = 0
        sql_queries = 0

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:153

  • This catches every exception raised by the entire shared query-lineage generator, not only failures reading SYS.M_SQL_PLAN_CACHE. Parser, worker, or programming errors are therefore downgraded to a warning and the workflow can finish successfully with missing Cloud lineage, making an actual regression indistinguishable from a permissions issue. Catch only the expected plan-cache DBAPI/permission failures (or report and re-raise unexpected exceptions).
    def yield_query_lineage(self) -> Iterable[Either[AddLineageRequest | CreateQueryRequest]]:
        """Query-history lineage, guarded so a restricted plan cache is not fatal.

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:84

  • The predicate trims whitespace but not leading SQL comments. A valid plan-cache entry such as /* job */ INSERT INTO ... SELECT ... or a comment-prefixed MERGE INTO ... therefore starts with /* and matches none of these anchored patterns, so its lineage is discarded before LineageParser can process it. Normalize leading comments before applying the DML filter (or filter after parsing) and add a regression case.

    # Anchored rather than wildcarded, so a SELECT that merely quotes the keyword does
    # not match. CREATE TABLE ... AS SELECT is missing by necessity: no DDL is cached.
    filters = f"""
        AND (

ingestion/src/metadata/ingestion/source/database/saphana/queries.py:53

  • TOTAL_EXECUTION_TIME from HANA's plan cache is a microsecond counter, while TableQuery.duration is consumed in milliseconds (the other query-history adapters convert with / 1000). Dividing by 1000000 makes every emitted query duration 1000x too small; convert it with / 1000 here.
  TOTAL_EXECUTION_TIME / 1000000 AS duration,

ingestion/tests/unit/topology/database/test_sap_hana.py:1534

  • These tests cover orchestration by replacing both lineage passes with empty iterators, while the SQL test only checks rendered string fragments. Nothing exercises a TableQuery produced from a representative plan-cache row, so the Cloud path could have broken row mapping or source-level wiring while all new tests still pass; add a source-level regression test and, where possible, a real HANA integration fixture.
def test_iter_runs_both_passes() -> None:
    """_iter must run the shared SQL passes and the repository pass.

    Dropping either one silently halves lineage on the deployment that depends on it.
    """
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 11, 2026 06:02
Comment on lines +55 to +58
_STATEMENT = (
r"LTRIM(UPPER(REPLACE_REGEXPR('^(\s*(/\*.*?\*/|--[^\n]*\n))+' IN STATEMENT_STRING WITH '' OCCURRENCE ALL))"
r", ' ' || CHAR(9) || CHAR(13) || CHAR(10))"
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Edge Case: Comment-stripping regex misses multi-line block comments

The /\*.*?\*/ branch relies on . matching the comment body, but SAP HANA's REGEXPR engine (ICU) does not let . match line terminators unless the DOTALL flag is set. A DML statement prefixed with a multi-line /* ... */ header (common in tool- or ETL-generated SQL) therefore won't be fully stripped, so the anchored LIKE 'INSERT INTO%SELECT%' filters can silently drop that statement and its lineage. If multi-line comment headers matter here, add the DOTALL flag to REPLACE_REGEXPR (or change the block-comment branch to /\*[^]*?\*/-equivalent) so the body can span newlines.

Was this helpful? React with 👍 / 👎

@gitar-bot

gitar-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown
Code Review 👍 Approved with suggestions 1 resolved / 2 findings

Adds SQL-based lineage for SAP HANA Cloud by extending LineageSource to parse view definitions and query history, recovering table and column-level edges where the previous _SYS_REPO repository approach (unavailable in Cloud) produced none. The comment-stripping regex in the DML filter may miss multi-line block comments since SAP HANA's REGEXPR engine does not let . match newlines by default—consider adding the DOTALL flag or using a newline-aware pattern if tool-generated SQL with multi-line comment headers is expected.

💡 Edge Case: Comment-stripping regex misses multi-line block comments

📄 ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:55-58

The /\*.*?\*/ branch relies on . matching the comment body, but SAP HANA's REGEXPR engine (ICU) does not let . match line terminators unless the DOTALL flag is set. A DML statement prefixed with a multi-line /* ... */ header (common in tool- or ETL-generated SQL) therefore won't be fully stripped, so the anchored LIKE 'INSERT INTO%SELECT%' filters can silently drop that statement and its lineage. If multi-line comment headers matter here, add the DOTALL flag to REPLACE_REGEXPR (or change the block-comment branch to /\*[^]*?\*/-equivalent) so the body can span newlines.

✅ 1 resolved
Quality: Filters use %% escaping though HANA driver keeps it literal

📄 ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:70-78 📄 ingestion/src/metadata/ingestion/source/database/saphana/queries.py:59-61 📄 ingestion/tests/unit/topology/database/test_sap_hana.py:1484 📄 ingestion/tests/unit/topology/database/test_sap_hana.py:1515
The filters LIKE patterns use %% (e.g. 'INSERT INTO%%SELECT%%') and the test at test_sap_hana.py:1484,1515 calls .replace("%%", "%") on the assumption that %% collapses to a single % at runtime. It does not: str.format() does not touch %, and the SAP HANA SQLAlchemy dialect uses qmark paramstyle, so SQLAlchemy performs no percent-unescaping. The final SQL sent to the driver literally contains %%. This is harmless today because consecutive % wildcards in SQL LIKE behave the same as one, but it is inconsistent with the single-% used in the same statement's app-comment filters (queries.py:59-61) and could mislead a future maintainer. Consider using single % in the filters for consistency, since no percent-escaping layer applies on HANA.

🤖 Prompt for agents
Code Review: Adds SQL-based lineage for SAP HANA Cloud by extending `LineageSource` to parse view definitions and query history, recovering table and column-level edges where the previous `_SYS_REPO` repository approach (unavailable in Cloud) produced none. The comment-stripping regex in the DML filter may miss multi-line block comments since SAP HANA's REGEXPR engine does not let `.` match newlines by default—consider adding the DOTALL flag or using a newline-aware pattern if tool-generated SQL with multi-line comment headers is expected.

1. 💡 Edge Case: Comment-stripping regex misses multi-line block comments
   Files: ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:55-58

   The `/\*.*?\*/` branch relies on `.` matching the comment body, but SAP HANA's REGEXPR engine (ICU) does not let `.` match line terminators unless the DOTALL flag is set. A DML statement prefixed with a multi-line `/* ... */` header (common in tool- or ETL-generated SQL) therefore won't be fully stripped, so the anchored `LIKE 'INSERT INTO%SELECT%'` filters can silently drop that statement and its lineage. If multi-line comment headers matter here, add the DOTALL flag to REPLACE_REGEXPR (or change the block-comment branch to `/\*[^]*?\*/`-equivalent) so the body can span newlines.

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Moderate findings remain for DML filtering, query-duration conversion, and real connector-path test coverage.

Review details

Suppressed comments (5)

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:89

  • The comment says the plan cache preserves submitted whitespace, but these predicates require the exact single-space tokens INSERT INTO and MERGE INTO. Valid multiline or multiply-spaced DML such as INSERT\nINTO ... SELECT ... will be silently excluded and produce no lineage; normalize internal whitespace or use token-boundary-safe matching before applying these LIKE clauses.
            {_STATEMENT} LIKE 'INSERT INTO%SELECT%'
            OR {_STATEMENT} LIKE 'UPSERT%SELECT%'
            OR {_STATEMENT} LIKE 'REPLACE%SELECT%'
            OR {_STATEMENT} LIKE 'MERGE INTO%'
            OR {_STATEMENT} LIKE 'UPDATE%SET%'

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:79

  • The new Cloud lineage path is not exercised end-to-end by the added tests: the parser test calls LineageParser directly, the orchestration test patches LineageSource._iter, and the SQL test only checks rendered text. A regression in SaphanaLineageSource's query execution/row mapping or in this HANA statement would therefore remain green. Add a source-level test with representative plan-cache rows that asserts the emitted lineage/query requests, or a HANA integration fixture.
    sql_stmt = SAPHANA_QUERY_HISTORY_STATEMENT

ingestion/src/metadata/ingestion/source/database/saphana/lineage.py:58

  • Because the block-comment branch uses .*? without HANA's DOTALL flag, a leading /* ... */ that spans lines is not removed. The normalized value still starts with /*, so the anchored DML predicates drop valid plan-cache statements and their lineage; add FLAGS 's' (or an equivalent newline-aware pattern) and a regression case for a multiline header.
_STATEMENT = (
    r"LTRIM(UPPER(REPLACE_REGEXPR('^(\s*(/\*.*?\*/|--[^\n]*\n))+' IN STATEMENT_STRING WITH '' OCCURRENCE ALL))"
    r", ' ' || CHAR(9) || CHAR(13) || CHAR(10))"
)

ingestion/src/metadata/ingestion/source/database/saphana/queries.py:53

  • M_SQL_PLAN_CACHE.TOTAL_EXECUTION_TIME is reported in microseconds, while TableQuery.duration is defined in milliseconds. Dividing by 1,000,000 therefore sends seconds under the millisecond field and makes every emitted query duration 1,000× too small; convert with / 1000 instead.
  TOTAL_EXECUTION_TIME / 1000000 AS duration,

ingestion/tests/unit/topology/database/test_sap_hana.py:1566

  • This orchestration test replaces LineageSource._iter and yield_cdata_lineage with callbacks, so it verifies only that two mocked generators are called in order; it never exercises the real yield_table_query/yield_view_lineage path that implements the Cloud fix. A regression in the actual plan-cache row mapping or view-definition lineage could therefore stay green. Add a representative plan-cache/view-definition test (or a HANA-backed integration fixture) that asserts the emitted lineage requests.
def test_iter_runs_both_passes() -> None:
    """_iter must run the shared SQL passes and the repository pass.

    Dropping either one silently halves lineage on the deployment that depends on it.
    """
    calls = []

    def record_sql(*_, **__):
        calls.append("sql")
        return iter([])

    def record_cdata():
        calls.append("cdata")
        return iter([])

    with (
        patch.object(LineageSource, "_iter", side_effect=record_sql),
        patch.object(SaphanaLineageSource, "yield_cdata_lineage", side_effect=record_cdata),
        patch.object(SaphanaLineageSource, "test_connection"),
        patch("metadata.ingestion.source.database.query_parser_source.get_ssl_connection"),
    ):
        source = SaphanaLineageSource(
            config=WorkflowSource(
                type="saphana-lineage",
                serviceName="test_sap_hana",
                serviceConnection=DatabaseConnection(
                    config=SapHanaConnection(
                        connection=SapHanaSQLConnection(username="test", password="test", hostPort="localhost:39015")
                    )
                ),
                sourceConfig=SourceConfig(config=DatabaseServiceQueryLineagePipeline()),
            ),
            metadata=create_autospec(OpenMetadata),
        )
        list(source._iter())

    assert calls == ["sql", "cdata"]
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SAP HANA Cloud: no lineage path exists (_SYS_REPO is on-prem only)

2 participants