fix(variant): convert shredded timestamps to the column's time unit

[jackylee-ch]

ShreddedValue::Timestamp is always micros, but the shredded leaf's Arrow unit follows the declared precision (MILLIS <4, MICROS 4..=6, NANOS >6). Both sides passed the integer through unchanged, so a TIMESTAMP leaf outside 4..=6 was 1000x off. A NANOS leaf is spec-conforming, so foreign files were mis-read too.

Write now scales micros to the leaf unit, flooring into MILLIS, and errors on micros beyond i64 nanoseconds (~1677..2262) instead of wrapping. Read keys off the array's TimeUnit, so a foreign file decodes from its own schema.

Disclosure: files paimon-rust already wrote outside 4..=6 hold micros under a MILLIS/NANOS annotation. They read back correctly only because the reader repeated the mistake; after this change they decode 1000x wrong, and nothing in the file tells them apart. Only an explicit variant.shreddingSchema produces such a leaf — inference picks precision 6.

[JingsongLi]

Reviewed head b75b8b9. Requirement fit: SUPPORTED. The seven focused shredded-timestamp tests pass, but two follow-ups remain. Integration/rebase follow-up: main commit d05cb03 (#884) added TimeUnit::Second for precision-zero timestamps after this branch diverged. This PR still has the earlier 0..=3 millisecond arm and no Seconds reader arm. GitHub marks the PR CONFLICTING, so this is not a demonstrated regression in a merged result; please reconcile the two changes on rebase, preserve Seconds↔micros conversion for TIMESTAMP(0) and TIMESTAMP_LTZ(0), and add precision-zero read/write tests. Persisted-file compatibility: as the PR body discloses, older Rust writes with explicit precision 1–3 or 7–9 shredding schemas stored micros under millis/nanos annotations. The new reader would silently reinterpret those typed values, and the files carry no marker to distinguish them from conforming files. Please make the migration/compatibility policy explicit (for example, a deliberate legacy mode or required rewrite) and include an old-writer fixture so the impact is testable.

[JingsongLi]

Reviewed head a9eff7c6. Fixing the Arrow unit mismatch has real end-to-end value, especially for spec-conforming files written by another engine. The precision-zero reconciliation and old-writer fixture address parts of my earlier comment. I found two production blockers:

[P1] New writes silently change a Variant timestamp that the declared leaf cannot represent. try_typed_shred (crates/paimon/src/variant.rs:2998-3003) puts any microsecond timestamp into typed_value. The new timestamp_array conversion (crates/paimon/src/arrow/shredding/variant.rs:1058-1068) floors it to seconds/millis, and reassembly reconstructs from that rounded typed value because the original value is absent. I ran a temporary complete Variant shred→assemble roundtrip: a TIMESTAMP_NTZ value of 1_700_000_000_123_456 microseconds with a TIMESTAMP(3) shredding schema came back as 1_700_000_000_123_000. Please keep the original Variant in the value fallback when it is not exactly representable in the typed leaf, and add a persisted-file roundtrip test. The same representability check should send a valid microsecond timestamp outside the nanosecond leaf's range to value instead of failing the entire write batch.

[P1] The on-disk unit change needs an executable migration and rollout path. For explicit shredding schemas at precision 0–3 or 7–9, a new reader reinterprets old Rust files, while an old reader misinterprets newly written conforming files. The old-writer unit test now documents one wrong-result case, but it does not prevent mixed-version reads/writes or identify which existing files need rewriting; precision-zero historical files need coverage too. Before a production merge, please document how to inventory affected tables/files, stop or coordinate old writers/readers, rewrite affected data using a version that still interprets it correctly, verify values, and roll back safely. Otherwise this can silently change persisted timestamps during a rolling upgrade.

Validation: all 20 arrow::shredding::variant::tests passed and head CI is green. My temporary full roundtrip regression failed with the exact values above, then I removed it and restored a clean checkout. The Parquet Variant shredding specification permits retaining an unrepresentable typed value in the Variant value field.

[jackylee-ch]

Rebased onto main. Precision-0 SECOND arm + tests, plus a rewrite policy and migration fixture.

[JingsongLi]

Re-reviewed head a1baa6eb after the author reply. Requirement fit remains SUPPORTED, and the precision-zero SECOND arm now reconciles #884. All 20 focused arrow::shredding::variant::tests pass, as do formatting and diff checks.

The two P1 production blockers from my previous review remain unresolved:

1. try_typed_shred still puts every microsecond timestamp into typed_value without checking whether the declared leaf can represent it. timestamp_array then floors a TIMESTAMP(0..3) value and value has no original fallback, so the full Variant roundtrip still loses the sub-second/sub-millisecond part. The same path still errors the entire write batch when a valid microsecond timestamp is outside the nanosecond leaf's i64 range. The new tests explicitly assert truncation at the array-conversion layer, but do not test that the full Variant value survives. Please fix representability before shredding and add a persisted-file roundtrip test.

2. The new “rewrite policy” is only a unit-test comment; there is still no usable inventory, rewrite, mixed-version rollout, verification, or rollback procedure for files written by earlier paimon-rust versions with explicit precision 0–3 or 7–9. The fixture confirms that old data is silently reinterpreted (1000× for the millis example). This remains a production migration blocker. Please publish an executable migration plan and cover precision-zero historical files too.

The rebase/merge conflict concern is resolved. I am leaving the PR open for these fixes.

[jackylee-ch]

Closing in favor of #983. This PR rescales the shredded timestamp to the column's declared precision, but the Parquet Variant shredding spec requires a shredded timestamp typed_value to be MICROS or NANOS, so a MILLIS/SECOND leaf is not spec-conformant and the rescale also drops sub-millisecond precision. #983 pins the shredded typed_value to microseconds instead, which removes the precision loss entirely (microseconds are stored as-is) and needs no data migration: files written by the earlier code stored the raw microsecond value and the reader reads it back unchanged, while new files are now correctly annotated for other engines. Thanks for the detailed review — the microsecond approach in #983 resolves both blockers raised here.