Repository navigation
fix: bit-packed arrays cast match more often - #10803
Conversation
|
The Generated by Claude Code |
e5cb28a to
d24aa06
Compare
BitPacked's CastKernel only took its direct unpack path for widening integer casts. A patch-free bit-packed array holds only values in 0..2^bit_width, so a narrowing cast whose target holds every such value cannot overflow. Take the same unpack path for those casts too, skipping the full-length intermediate buffer and the generic cast's bounds-check scan. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
d24aa06 to
fd2091f
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 1↓)
No file size changes detected. |
Merging this PR will regress 1 benchmark4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.938x ➖, 2↑ 2↓)
datafusion / parquet / ns (0.974x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.048x ➖, 0↑ 1↓)
|
|
The CodSpeed run on the rebased commit ( Generated by Claude Code |
Summary
Split out of #10761. This keeps only the
vortex-fastlanescast change; the bit-packed Arrow dictionary export from that PR is dropped.BitPacked'sCastKernelonly took its direct unpack path for widening integer casts. Bit-packing rejects negative integers, so a patch-free array packed at a globalbit_widthholds only values in0..2^bit_width. A narrowing cast whose target holds every such value cannot overflow.One place this applies is the Arrow export of Vortex dictionaries.
dict_to_dictexports the codes throughto_arrow_primitive, which casts them to the requested key type. Dictionary codes are often bit-packed, so a key type narrower than the codes' ptype hits this kernel.Changes
packed_values_fit, which checks that a patch-free array's bit width, plus a sign bit for signed targets, fits the target type.test_cast_bitpacked_narrowing_fits, which checks narrowing casts (toU8,I8,U16,I16) against a plain primitive cast, including sliced and nullable inputs.I haven't measured a speedup for this change on its own.
Checks run:
cargo clippy -p vortex-fastlanes --all-targets --all-features -- -D warnings,cargo +nightly-2026-09-10 fmt -p vortex-fastlanes --check, andcargo test -p vortex-fastlanes --lib cast(45 passed).🤖 Generated with Claude Code
https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC