fix(variant): store shredded timestamps as microseconds - #983
Open
jackylee-ch wants to merge 1 commit into
Open
jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
The Variant binary stores timestamps as microseconds (encoding type codes 12 and 13 are `TIMESTAMP(MICROS)` / `TIMESTAMP_NTZ(MICROS)`), and the shredding spec requires the shredded `typed_value` column to use microseconds. A configured shredding schema, however, had its declared type copied through verbatim, so declaring `TIMESTAMP(3)` produced a millisecond-typed `typed_value` column that still held the microsecond count. Rust round-trips this losslessly because it ignores the unit on both read and write, but the file violates the spec and any precision-aware reader (Spark, DuckDB, Arrow, Java `Timestamp.fromMicros`) reads the value 1000x off. Pin the shredded timestamp `typed_value` precision to microseconds so the written column is annotated and scaled consistently. Non-timestamp scalars are unchanged.
Contributor
Author
Contributor
Author
|
Reopening. Re-reading the Parquet Variant shredding spec, a shredded timestamp |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Variant binary stores timestamps as microseconds (encoding type codes 12 and 13 are
TIMESTAMP(MICROS)/TIMESTAMP_NTZ(MICROS)), and the shredding spec requires the shreddedtyped_valuecolumn to use microseconds. A configured shredding schema, however, had its declared type copied through verbatim, so declaringTIMESTAMP(3)produced a millisecond-typedtyped_valuecolumn that still held the microsecond count.Rust round-trips this losslessly because it ignores the unit on both sides, but the file violates the spec and any precision-aware reader (Spark, DuckDB, Arrow, Java
Timestamp.fromMicros) reads it 1000x off.Pin the shredded timestamp
typed_valueto microseconds so the column is annotated consistently; other scalars are unchanged. A test asserts this holds even when the configured schema declares another precision.