sap_hana: add SAP HANA polling input and hana driver support (ENG-1213) by squiidz · Pull Request #4775 · redpanda-data/connect

squiidz · 2026-09-04T17:04:06Z

Adopts and supersedes #4462 by @ness-david-dedu, keeping the original commits and authorship, and carries the work forward so CI can run and the outstanding review findings are addressed.

What this adds

sap_hana input with bulk, incrementing, timestamp, timestamp+incrementing, and query modes. Schema metadata comes from SYS.TABLE_COLUMNS (primary-key discovery, per-column nullability, precision-aware DECIMAL canonicalisation) and the high-water mark can be persisted to a cache resource for resume across restarts.
driver: hana support for the sql_* components.
Read benchmark harness under internal/impl/saphana/bench plus results in docs/benchmark-results/sap-hana.md.

New since #4462 (addresses the 2026-08-03 review)

Transient scan errors no longer lose rows. Cursor teardown after an error now rewinds the in-memory HWM to the last safe value, so rows that were scanned but discarded are re-read on the next poll instead of being skipped, and the empty-poll path can no longer persist an HWM past undelivered rows.
Checkpoint persistence is ack-order aware. HWM snapshots flow through a checkpoint.Capped tracker (bounded by a new advanced checkpoint_limit field, default 1024) and only persist once every earlier batch has resolved, so a later batch acking first cannot checkpoint past rows still in flight. Nacks resolve their slot like acks: with auto_replay_nacks (the default) they are replayed, and disabling it is the documented opt-in to drop.
driver: hana works in every distribution. go-hdb (Apache-2.0) is now registered in public/components/sql alongside the other supported drivers, instead of only via the enterprise saphana package import.
Generated docs committed (docs/modules/components/pages/inputs/sap_hana.adoc plus the regenerated sql_* pages).

Also: merged latest main, fixed all outstanding lint issues on the branch, and added sqlmock-based unit tests covering the HWM rewind and ack-ordering behaviour (each was verified to fail against the previous implementation).

Known gaps / follow-ups

numeric_mapping: best_fit currently behaves identically to none (both emit canonical decimal strings).
The spec's partition_count knob is not implemented.
Integration tests require a HANA Express instance (amd64-only Docker image) and are skipped by default.

Adds a sap_hana input with bulk, incrementing, timestamp, timestamp+incrementing, and query modes. Schema metadata comes from SYS.TABLE_COLUMNS (primary-key discovery, per-column nullability, precision-aware DECIMAL canonicalisation) and the high-water mark can be persisted to a cache resource for resume across restarts. Also adds driver: hana support to the sql_* components (go-hdb registered in every distribution) and a read benchmark harness with results. Checkpointing is ack-order aware: HWM snapshots flow through a checkpoint.Capped tracker bounded by an advanced checkpoint_limit field and only persist once every earlier batch has resolved. Cursor teardown after an error rewinds the in-memory HWM to the last safe value so scanned-but-undelivered rows are re-read instead of skipped. Supersedes the work started in an external PR; original implementation by David Dedu. Co-authored-by: ness-david-dedu <david.dedu@ness.com>

- Register the saphana package in the integration-test registry so the integration-test-filter CI job passes. - Fix the metadata docs to name the keys the input actually emits (table_name, database_schema, schema, primary_key_columns), state when each is present, and skip database_schema when schema_name is unset. - Implement numeric_mapping=best_fit: DECIMAL columns whose precision fits a double (<= 15) map to Float64 at the schema layer and emit as floats, with an equivalent digit-count fallback when no schema is available; corrected the documented behaviour of none. - Expose the query retry delay as a retry_backoff duration field instead of a hardcoded one-second base. - Issue the first poll immediately on startup instead of sleeping a full poll_interval before the first query. - Document the fetch_size / checkpoint_limit interaction (oversized batches serialize, they do not deadlock) and warn at startup. - Drop the write benchmark sections that measured a dedicated output removed from this PR; the harness returns with the write-path PR. - Add unit coverage for timestamp and timestamp+incrementing query shapes and HWM advancement, checkpoint resume across restart, and the retry path.

HANA Express is an amd64-only ~4GB image that needs ~8GB RAM, relaxed seccomp, and a multi-minute cold start per container, which standard CI runners cannot provide. The testcontainers harness also starts one container per test and passes the SAP license agreement as an env var where the image requires a container argument, so it needs rework on an amd64 host before it can run anywhere.

… (ENG-1213) Every []byte from go-hdb was converted to a Go string, and encoding/json mangles invalid UTF-8 in strings into U+FFFD replacement characters, silently corrupting VARBINARY/BLOB values while the emitted schema promised ByteArray. Text-typed columns still convert to string, but binary columns (and, without schema metadata, any invalid-UTF-8 bytes) now stay []byte so JSON base64-encodes them losslessly. A failed primary-key catalog lookup was also swallowed without a log, leaving primary_key_columns silently absent for the pipeline lifetime (the schema result is cached); it now emits a warning naming the table.

github-actions Bot added the run-integration-tests label Sep 4, 2026

squiidz force-pushed the eng-1213-sap-hana-input branch from 7f5c9a4 to 1367fba Compare September 4, 2026 17:06