fix(java): carry alterColumns cast type across FFI via C Data Interface - #8417
Merged
westonpace merged 2 commits intoAug 7, 2026
Merged
Conversation
`Dataset.alterColumns(...castTo...)` silently dropped the cast target type. The JNI `create_column_alteration` marshalled it by calling the Java `ArrowType.toString()` (e.g. `"Int(64, true)"`, `"FloatingPoint(DOUBLE)"`) and parsing the result with `arrow_schema::DataType::from_str`, then swallowing the parse failure with `.ok()`. Parameterized types do not round-trip through that grammar, so `data_type` became `None` and the cast was a no-op: the commit landed but the stored column type was unchanged. Transfer the cast target type through the Arrow C Data Interface instead, mirroring `addColumns(Schema)`: the Java side exports one field per requested cast (in alteration order) into an `ArrowSchema`, and the JNI imports it via `FFI_ArrowSchema` and attaches each type to the corresponding `ColumnAlteration`. Rename and nullability-only alterations are unaffected. Removes the now-unused `DataType`/`FromStr` imports. Adds `DatasetTest.testAlterColumnsCastType` covering an Int32->Int64 widen and a combined rename+cast, asserting the resulting Arrow type (the existing `testAlterColumns` only checked field names, so the dropped cast went unnoticed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ent) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The typed Arrow C Data transfer fixes the lossy Java-to-Rust boundary without extending display-string parsing, while preserving rename-only and nullability-only alterations. The regression coverage exercises a parameterized cast and a combined rename/cast.
westonpace
approved these changes
Aug 7, 2026
westonpace
left a comment
Member
There was a problem hiding this comment.
We do have String<->DataType somewhere I think but C FFI seems reasonable to me as well. No change to public APIs that I can see.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Dataset.alterColumns(...)with acastTo(...)alteration silently drops the cast: the commit lands (version bumps) but the stored column type is unchanged.Root cause is in the JNI
create_column_alteration(java/lance-jni/src/blocking_dataset.rs). The cast target type was marshalled by calling the JavaArrowType.toString()(e.g."Int(64, true)","FloatingPoint(DOUBLE)") and parsing the string witharrow_schema::DataType::from_str, then discarding any parse error with.ok():DataType'sFromStrgrammar does not accept theDebug-style strings thatArrowType.toString()produces for parameterized types, sodata_typebecameNonefor anything beyond the few types whosetoString()happens to match (e.g.Utf8). The existingDatasetTest.testAlterColumnsonly asserted field names after a cast, so the dropped type went unnoticed.Fix
Transfer the cast target type through the Arrow C Data Interface, mirroring the existing
addColumns(Schema)path:Dataset.alterColumns): export one field per requested cast — in the same order as the alterations — into anArrowSchema, and pass its memory address to the native method.inner_alter_columns): import the schema viaFFI_ArrowSchemaand attach each importedDataTypeto the correspondingColumnAlteration.Rename-only and nullability-only alterations are unaffected. Removes the now-unused
DataType/FromStrimports.Test
Adds
DatasetTest.testAlterColumnsCastType: widensidfromInt32toInt64, then does a combined rename+cast, asserting the resulting Arrow type (not just the field name).Context
Surfaced while implementing schema-evolution DDLs in
lance-spark(lance-format/lance-spark#752), whereALTER COLUMN ... TYPEhad to be rejected because of this bug. With this fix released, lance-spark can enable type changes.🤖 Generated with Claude Code