fix(spec): match Java whitespace for CHAR/VARCHAR partition values - #985
Open
jackylee-ch wants to merge 2 commits into
Open
jackylee-ch wants to merge 2 commits into
jackylee-ch wants to merge 2 commits into
Conversation
The CHAR/VARCHAR arm of the partition-value computer folded a value to the default partition name when `str::trim` left it empty, but Rust's whitespace set differs from Java's `Character.isWhitespace`: `str::trim` strips a non-breaking space (U+00A0) and U+2007 / U+202F, which Java keeps, and keeps U+001C-U+001F, which Java strips. Because the partition value is recomputed on both the write and read paths, such a value produced a different partition directory in Rust than in Java, so cross-engine readers miss the data. Reuse `is_java_whitespace_only` — already used by the Binary arm — so the string arm folds exactly what Java folds.
`format_partition_value` folded a CHAR/VARCHAR value to the default partition with `str::trim().is_empty()`, whose whitespace set differs from Java's `Character.isWhitespace`: U+00A0 (which Java keeps) folded in Rust, and U+001C-U+001F (which Java folds) did not, so a Format Table partition directory diverged from a Java-written one on both the write and read paths. The `name_prefix_pattern` pushdown skip used the same `str::trim` check. Reuse `is_java_whitespace_only` (the predicate the spec-side partition computer already uses) in both, matching Java `InternalRowPartitionComputer` (value folding) and `PartitionPathUtils.buildPartitionNamePrefixPattern` (the pushdown skip). Add a partition-value test covering both divergent directions.
jackylee-ch
force-pushed
the
fix/partition-value-whitespace-java-parity
branch
from
September 30, 2026 14:23
c69d471 to
85b3f91
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CHAR/VARCHAR arm of the partition-value computer folds a value to the default partition name when
str::trimleaves it empty, but Rust's whitespace set differs from Java'sCharacter.isWhitespace:str::trimstrips a non-breaking space (U+00A0) and U+2007 / U+202F, which Java keeps, and keeps U+001C-U+001F, which Java strips.The partition value is recomputed on both the write and read paths (BinaryRow -> path), so such a value produces a different partition directory in Rust than in Java, and a cross-engine reader misses the data. The Binary arm already handled this correctly via
is_java_whitespace_only(added in #958); the string arm was left onstr::trim.The same
str::trimfold also appeared in the Format Table partition paths —format_partition_value(the CHAR/VARCHAR directory value) and thename_prefix_patternpushdown skip — so those diverged the same way. They are fixed too, matching JavaInternalRowPartitionComputer(value folding) andPartitionPathUtils.buildPartitionNamePrefixPattern(the pushdown skip).Reuse
is_java_whitespace_onlyin every arm so the fold matches exactly what Java folds. Tests cover both divergent directions (U+001C folds, U+00A0 does not) for the spec-side computer and the Format Table value.