Fix NPE reading STRING COLLATE columns via Arrow by msprunck · Pull Request #1548 · databricks/databricks-jdbc

msprunck · 2026-07-08T19:58:48Z

Description

A STRING column carrying a non-default collation (e.g. STRING COLLATE UTF8_LCASE) cannot be read when the warehouse reports its type via the Statement Execution API (SEA). Such a column arrives with a type_name that does not map to any ColumnInfoTypeName, so ColumnInfo.getTypeName() is null, and two things break:

Value read — ArrowToJavaObjectConverter.convert reaches switch (requiredType) with requiredType == null and throws a NullPointerException. This aborts every result set containing a collated string column, including a single-row result.
Metadata — DatabricksResultSetMetaData maps the null type name to Types.OTHER, so getColumnType() returns OTHER and the column is not usable as a string, even once the value read is fixed.

Trigger

The defect is latent in the driver's type handling — the converter and metadata builder both rely on the ColumnInfoTypeName enum resolving, and neither tolerates a null. What surfaces it is which metadata path the warehouse answers with: the Thrift path maps the collated column to STRING, while the SEA path returns the collated type text verbatim (STRING COLLATE UTF8_LCASE, over Arrow type_name = UNKNOWN_COLUMN_TYPE_NAME), which yields a null ColumnInfoTypeName. The converter source is unchanged across recent versions; the trigger is the metadata routing, gated on the advertised driver version. (Verified: same driver + query, one warehouse answered Thrift and read fine, another answered SEA and NPE'd; the SDK 0.69→0.106 bump was ruled out.)

Changes

DatabricksTypeUtil.recoverStringType(typeText) (new): returns ColumnInfoTypeName.STRING when the given type text denotes a (possibly collated) string, else null. Case-insensitive (Locale.ROOT) and matches on a word boundary, so STRING, STRING COLLATE ... and STRING(...) are recovered but a hypothetical future STRINGVIEW is not.
ArrowToJavaObjectConverter.convert (value path): when requiredType is null, recover it via recoverStringType(arrowMetadata). The switch is also guarded with an explicit DatabricksValidationException (logging only the metadata, not the raw cell value) so any genuinely unmappable type produces a clear error instead of a raw NPE.
DatabricksResultSetMetaData (SEA result manifest constructor, metadata path): when the column type name is null, recover it via recoverStringType(columnInfo.getTypeText()) so getColumnType() resolves to VARCHAR instead of OTHER. The collated type text is preserved so getColumnTypeName() still reports the server type (mirroring the existing TIMESTAMP_NTZ null-recovery).

Both call sites share the same recovery rule but pass their own source (arrow field metadata for the value path, manifest type_text for the metadata path), since those channels legitimately differ.

Testing

DatabricksTypeUtilTest#testRecoverStringType — plain/collated string in any case → STRING; word-boundary rejection (STRINGVIEW/STRINGSET); non-string and null → null.
ArrowToJavaObjectConverterTest: testCollatedStringWithNullRequiredType (upper), testCollatedStringLowerCaseMetadata (lower, covers case handling), testTypeStartingWithStringWordIsNotCoercedToString (word boundary → DatabricksValidationException), testUnresolvableTypeThrowsValidationException (null-guard).
DatabricksResultSetMetaDataTest: testColumnsWithCollatedString and testColumnsWithCollatedStringLowerCase — getColumnType() is VARCHAR, getColumnTypeName() preserves the collated text.
Affected suites pass (151 tests across the three classes).
Validated end-to-end against a live SQL warehouse whose view exposes rule_name COLLATE UTF8_LCASE / rule_description COLLATE UTF8_LCASE over the SEA path: with this change, all columns read successfully and getColumnType() reports VARCHAR, where the unpatched driver throws the NPE.

Additional Notes to the Reviewer

This revision consolidates the earlier two inline prefix checks into the single recoverStringType helper, addressing the review: the two sites now share one case-insensitive, word-boundary rule (no case drift between value and metadata paths), and a future unmappable type fails with a clear exception rather than an NPE.
Longer term, the more robust fix is to drive value conversion from the Arrow physical type and treat the server type name as a non-fatal string (as the databricks-sql-python and databricks-sql-go connectors do, which do not have this bug); that is a larger, hot-path change better owned by maintainers. This PR is the contained fix. See [BUG] NullPointerException reading STRING COLLATE UTF8_LCASE columns via Arrow (ColumnInfoTypeName.ordinal() - requiredType is null) #1547 for the design discussion.

vikrantpuppala · 2026-07-09T16:15:04Z

+      // A collated string column (e.g. "STRING COLLATE UTF8_LCASE") is reported with a type_name
+      // that does not map to any ColumnInfoTypeName, leaving requiredType null. Recover the type
+      // from the metadata prefix so the value is read as a string instead of throwing an NPE below.
+      if (requiredType == null && arrowMetadata.startsWith(STRING)) {


Medium · flagged by devil's-advocate + test reviewers, confirmed against head SHA.

The two recovery sites added in this PR normalize case differently:

Value path (this line, ArrowToJavaObjectConverter.java:96): arrowMetadata.startsWith(STRING) — case-sensitive.

Metadata path (DatabricksResultSetMetaData.java:113): columnInfo.getTypeText().toUpperCase().startsWith(STRING) — case-insensitive.

If any server build emits the collated type text in lower/mixed case (e.g. "string collate utf8_lcase"), the metadata path resolves the column to VARCHAR, but this value path's startsWith fails, requiredType stays null, and getObject throws DatabricksValidationException. The driver would then advertise a readable VARCHAR column that cannot actually be read — worse than a clean failure.

Suggestion: align both sites on the same case handling. Since the value-path siblings (ARRAY/STRUCT/MAP/…) are all case-sensitive by the Spark SqlName uppercase convention, either accept that in both places or normalize both with toUpperCase(Locale.ROOT).

_{Posted by code-review-squad · /full-review · feedback: #code-review-squad-feedback}

Fixed. Both sites now call the shared DatabricksTypeUtil.recoverStringType, which normalizes with Locale.ROOT, so the value path and metadata path resolve a lower/mixed-case collated type text identically. Added a lowercase test on each path (which would fail under the old case-sensitive value-path startsWith).

vikrantpuppala · 2026-07-09T16:15:06Z

+      // A collated string column (e.g. "STRING COLLATE UTF8_LCASE") is reported with a type_name
+      // that does not map to any ColumnInfoTypeName, leaving requiredType null. Recover the type
+      // from the metadata prefix so the value is read as a string instead of throwing an NPE below.
+      if (requiredType == null && arrowMetadata.startsWith(STRING)) {


Medium (open question) · flagged by devil's-advocate; consequence not verified at the branching point.

The read path (ArrowStreamResult.java:251-253) falls back to columnInfo.getTypeText() only when chunkIterator.getType() returns null — not when it returns a non-null value that maps to no known type. So this recovery keys off arrowMetadata (the Arrow field Spark:DataType:SqlName), which may differ from the manifest type_text.

Issue #1547 notes two warehouse builds on the same version label behaved differently (one NPE'd, one didn't). If some build's Spark:DataType:SqlName for a collated column is a value that doesn't start with "STRING", this recovery won't fire and the read now throws DatabricksValidationException instead of the old NPE — still unreadable for the user.

I could not confirm what Spark:DataType:SqlName actually holds for the failing build (UNKNOWN_COLUMN_TYPE_NAME is a protobuf/SDK enum sentinel → null after deserialization, not an Arrow-metadata string), so this is a question rather than a confirmed defect.

Question: for the exact warehouse build in #1547 that NPE'd, what does the Arrow field metadata Spark:DataType:SqlName contain for the collated column? If it can be non-STRING, consider recovering when either arrowMetadata or columnInfo.getTypeText() starts with "STRING".

_{Posted by code-review-squad · /full-review · feedback: #code-review-squad-feedback}

Confirmed against the affected warehouse: instrumented capture shows the converter receives arrowMetadata = "STRING COLLATE UTF8_LCASE" for the collated columns, so recoverStringType(arrowMetadata) fires and the value reads. End-to-end validation on that warehouse: all columns read and getColumnType() is VARCHAR. If a build ever reports the collated type only in the manifest type_text (not the Arrow field), the value path still falls back to columnInfo.getTypeText() via ArrowStreamResult when the Arrow field is null.

vikrantpuppala · 2026-07-09T16:15:08Z

+          if (columnTypeName == null
+              && columnInfo.getTypeText() != null
+              && columnInfo.getTypeText().toUpperCase().startsWith(STRING)) {
+            columnTypeName = ColumnInfoTypeName.STRING;


Medium · flagged by architecture + maintainability reviewers.

This recovery now lives in both DatabricksResultSetMetaData and ArrowToJavaObjectConverter because the SEA path has no single type-name resolution point — the metadata constructor recovers into a local variable and never writes back to the shared ColumnInfo, so the read path independently re-derives the null getTypeName(). Every future null-type_name server type must be patched in lockstep, and the two copies have already drifted on case handling (see F1).

This mirrors a pre-existing convention, so it's a maintainability cost to note rather than a blocker. Suggested direction (follow-up, not required here): a single resolveTypeName(ColumnInfo) helper applied right after SEA manifest deserialization that calls setTypeName(...) on the shared object, letting both consumers drop their local recovery.

_{Posted by code-review-squad · /full-review · feedback: #code-review-squad-feedback}

The decision rule is now single-sourced in recoverStringType. I kept two call sites rather than a single write-back on ColumnInfo on purpose: the value path keys off the Arrow field metadata (Spark:DataType:SqlName) and the metadata path off the manifest type_text — those channels legitimately differ, so a single write-back would change which source drives recovery. Sharing the rule (not the source) removes the case drift that caused F1 without that behavior change.

vikrantpuppala · 2026-07-09T16:15:11Z

+      // A collated string column (e.g. "STRING COLLATE UTF8_LCASE") is reported with a type_name
+      // that does not map to any ColumnInfoTypeName, leaving requiredType null. Recover the type
+      // from the metadata prefix so the value is read as a string instead of throwing an NPE below.
+      if (requiredType == null && arrowMetadata.startsWith(STRING)) {


Low · flagged by devil's-advocate.

A hypothetical future type whose name begins with the letters STRING (e.g. "STRINGVIEW", "STRINGSET") would be silently coerced to scalar STRING and read via toString() — producing wrong values rather than an error, precisely on the null-typeName path where a brand-new type is most likely to land. Consistent with the existing sibling prefix checks, so low priority.

Optional: match a word boundary — typeText.equals("STRING") || typeText.startsWith("STRING ") || typeText.startsWith("STRING(") — so "STRING COLLATE …" matches but "STRINGVIEW" does not.

_{Posted by code-review-squad · /full-review · feedback: #code-review-squad-feedback}

Fixed. recoverStringType matches on a word boundary (equals("STRING"), or startsWith("STRING ") / startsWith("STRING(")), so STRING COLLATE ... and STRING(...) are recovered but STRINGVIEW/STRINGSET are not. Added a rejection test asserting STRINGVIEW throws DatabricksValidationException rather than being coerced to STRING.

vikrantpuppala · 2026-07-09T16:15:13Z

  }

+  @Test
+  public void testCollatedStringWithNullRequiredType() throws Exception {


Low · flagged by test reviewer.

Both new tests feed already-uppercase literals, so:

The .toUpperCase() on the metadata side is a no-op in the test — delete it from production and the test still passes (covered by zero assertions).

Plain "STRING" via the null-recovery path is uncovered (the existing testNullHandlingInVarCharVector uses a non-null requiredType).

A converter case with lowercase metadata — convert(vector, 0, null, "string collate utf8_lcase", new ColumnInfo()) expecting the value — would currently fail and directly surface F1. Adding a lowercase metadata assertion covers the .toUpperCase() call. The two added tests are otherwise sound (real Arrow vectors, independent literal expectations, would fail if the fix were reverted).

_{Posted by code-review-squad · /full-review · feedback: #code-review-squad-feedback}

Added: a lowercase-metadata test on both the value path (testCollatedStringLowerCaseMetadata) and the metadata path (testColumnsWithCollatedStringLowerCase) — these exercise the case-insensitive recovery and would fail under the old case-sensitive value path. Also a direct DatabricksTypeUtilTest#testRecoverStringType (plain/collated/any-case, word-boundary rejection, null) and the STRINGVIEW word-boundary test.

vikrantpuppala · 2026-07-09T16:15:52Z

Code Review Squad — Review

Score: 89/100 — MODERATE RISK

Solid, well-scoped bug fix that faithfully follows the existing in-file prefix-recovery convention. No merge blockers — every finding below is a robustness or test-coverage gap around the recovery logic, verified against the PR head SHA. The highest-value item is the case-asymmetry between the two recovery sites (F1); the lowercase test in F5 would have caught it for free. Security and language reviewers found nothing at threshold.

msprunck · 2026-07-17T09:57:07Z

@vikrantpuppala thanks for the thorough review. Addressed the findings in the latest commit — replied inline under each one. Summary: consolidated the recovery into a single DatabricksTypeUtil.recoverStringType helper used by both the value and metadata paths (one case-insensitive, word-boundary rule, no drift between sites), verified end-to-end against the affected warehouse.

Left as a follow-up (not this PR): the larger refactor to drive value conversion from the Arrow physical type — see the design note on #1547.

A collated string column (e.g. STRING COLLATE UTF8_LCASE) is returned with a type_name that does not map to any ColumnInfoTypeName, so ColumnInfo.getTypeName() is null and ArrowToJavaObjectConverter.convert throws a NullPointerException at `switch (requiredType)` when reading the value (both list and single-row results). Recover the type from the arrow metadata prefix: when requiredType is null and the metadata starts with STRING (which covers "STRING COLLATE ..."), treat it as STRING. Also guard the switch with an explicit DatabricksValidationException so any future unmapped type produces a clear error instead of a raw NPE. Regression vs 3.3.3 (which reads these columns fine). Fixes databricks#1547. Added ArrowToJavaObjectConverterTest#testCollatedStringWithNullRequiredType. Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

- The null-guard error path no longer embeds the raw column value in the log message or exception (could expose sensitive result data or emit very large values); it now reports only the arrow metadata. - Added testUnresolvableTypeThrowsValidationException covering the guard branch (null requiredType + unmappable metadata -> DatabricksValidationException instead of NPE). Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

Collated string columns (e.g. "STRING COLLATE UTF8_LCASE") arrive from the SEA result manifest with a null ColumnInfoTypeName, so getColumnType() returned OTHER and getColumnTypeName() the raw collated text, even though getObject() now reads the value correctly. In the SEA ResultManifest metadata constructor, recover STRING from the typeText prefix when the type name is null (mirroring the existing TIMESTAMP_NTZ null-recovery), so getColumnType() resolves to VARCHAR. The collated typeText is preserved for getColumnTypeName(). Added DatabricksResultSetMetaDataTest#testColumnsWithCollatedString. Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

Replaces the two divergent prefix checks with a single DatabricksTypeUtil.recoverStringType(typeText), used by both the value path (ArrowToJavaObjectConverter) and the SEA metadata path (DatabricksResultSetMetaData). This fixes the review findings: - F1 (case asymmetry): both sites now share one case-insensitive rule (Locale.ROOT), so a lower/mixed-case collated type text resolves the same way on both paths. - F3 (duplication): the recovery decision lives in one place. The two call sites still pass their own source (arrow metadata vs typeText), since those channels legitimately differ, but the rule is shared. - F4 (over-match): match on a word boundary so "STRING COLLATE ..." and "STRING(...)" are recovered but a future "STRINGVIEW" is not. - F5 (test gap): add lowercase-metadata tests on both paths, a word-boundary rejection test, and a direct recoverStringType unit test. Validated end-to-end against the affected warehouse: value reads succeed and getColumnType() reports VARCHAR for the collated columns. Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

Already inside if (columnTypeName == null); assigning the (possibly null) result of recoverStringType leaves it null when no match, which is the pre-existing behavior (getColumnType(null) -> OTHER). Matches the converter call site's direct assignment. Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

msprunck mentioned this pull request Jul 9, 2026

[BUG] NullPointerException reading STRING COLLATE UTF8_LCASE columns via Arrow (ColumnInfoTypeName.ordinal() - requiredType is null) #1547

Open

vikrantpuppala reviewed Jul 9, 2026

View reviewed changes

msprunck added 5 commits July 17, 2026 12:20

msprunck force-pushed the fix/collated-string-npe branch from f35f00a to d7886d9 Compare July 17, 2026 10:20

msprunck marked this pull request as ready for review July 17, 2026 10:27

msprunck requested a review from vikrantpuppala July 17, 2026 12:54

Fix comment: recovery matches a word boundary, not a prefix

f12fcca

Signed-off-by: Matthieu Sprunck <msprunck@cisco.com>

msprunck force-pushed the fix/collated-string-npe branch from b4243f4 to f12fcca Compare July 17, 2026 13:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix NPE reading STRING COLLATE columns via Arrow#1548

Fix NPE reading STRING COLLATE columns via Arrow#1548
msprunck wants to merge 6 commits into
databricks:mainfrom
msprunck:fix/collated-string-npe

msprunck commented Jul 8, 2026 •

edited

Loading

Uh oh!

vikrantpuppala Jul 9, 2026

Uh oh!

msprunck Jul 17, 2026

Uh oh!

vikrantpuppala Jul 9, 2026

Uh oh!

msprunck Jul 17, 2026

Uh oh!

vikrantpuppala Jul 9, 2026

Uh oh!

msprunck Jul 17, 2026

Uh oh!

vikrantpuppala Jul 9, 2026

Uh oh!

msprunck Jul 17, 2026

Uh oh!

vikrantpuppala Jul 9, 2026

Uh oh!

msprunck Jul 17, 2026

Uh oh!

vikrantpuppala commented Jul 9, 2026

Uh oh!

msprunck commented Jul 17, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

msprunck commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Trigger

Changes

Testing

Additional Notes to the Reviewer

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vikrantpuppala commented Jul 9, 2026

Code Review Squad — Review

Uh oh!

msprunck commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msprunck commented Jul 8, 2026 •

edited

Loading

msprunck commented Jul 17, 2026 •

edited

Loading