Repository navigation
Speed up PolarSignals scans: spread filtered V2 splits and export primitive dictionaries natively - #10761
Speed up PolarSignals scans: spread filtered V2 splits and export primitive dictionaries natively#10761joseph-isaacs wants to merge 7 commits into
Conversation
…tion pieces Filtered V2 scans used fixed 1,048,576-row chunks. A DataFusion byte-range partition of a ~1M-row file then became a single chunk, so when the surviving rows sat in one partition (PolarSignals time ranges) one thread did all the work. Size filtered chunks by the serialized bytes the projection reads per row and by the partition's rows per thread, keeping 1M-row chunks for cheap projections over large ranges. ProjectionMorsel also wrapped its pieces in a ChunkedArray, which the Arrow export concatenated, re-merging nested dictionaries. Emit pieces as separate batches, grouping only pieces under 4,096 rows. PolarSignals SF1 on DataFusion, V2 vs V1 geomean: 1.79x -> 0.93x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Exporting a non-dictionary Vortex array to an Arrow dictionary type went through arrow_cast, which hashes each primitive value with SipHash and builds an ArrowError per value. PolarSignals requests its nested `locations` fields as dictionaries, so this was ~90% of scan CPU for both V1 and V2. Encode primitive values directly, hashing their bits with FxHash and keeping nulls in the codes, and fall back to arrow_cast for other types or when the codes overflow. PolarSignals SF1 on DataFusion (V2): q00-q03 ~3.6x faster, total 4.28s -> 1.90s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
…ictionaries Hash-map growth and hashing still dominated exporting PolarSignals' nested dictionary fields. Keys spanning under 65,536 values (line numbers) are now looked up in a dense table. When a sample of the values is nearly all distinct (addresses), the values become the dictionary with sequential codes, reusing their buffer, since Arrow dictionaries may hold duplicates. Other inputs hash into a map sized from the sample. PolarSignals SF1 on DataFusion (V2): q00-q03 another 22-31% faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will degrade performance by 13.13%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (7)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.564x ✅, 5↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.971x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.990x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.917x ➖, 1↑ 0↓)
|
A bit-packed array without patches only holds values in 0..2^bit_width, so its unpacked values are already codes into that range. Exporting it to an Arrow dictionary now unpacks straight into the codes type and uses 0..2^bit_width as the values, instead of decoding to the values type and dictionary-encoding the result. Applies up to 16-bit widths when the dictionary is no larger than the array. To unpack into narrower codes, BitPacked's cast kernel now also handles narrowing integer casts when the array has no patches and the target holds every value the bit width allows, skipping the generic bounds check. PolarSignals SF1 on DataFusion (V2): q00-q03 another ~2x faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
…0erx1 Signed-off-by: Claude <noreply@anthropic.com> # Conflicts: # vortex-arrow/src/executor/dictionary.rs
…ange The primitive dictionary export scanned every key for its range before choosing an encoding, even for high-cardinality columns such as addresses that can never use the dense table. Sample first, and scan the full range only when the sampled keys fit the dense table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
|
CodSpeed Performance Analysis fails on
There's no fix to port for these. I'll re-check them on the next push. Generated by Claude Code |
Summary
On PolarSignals with DataFusion, scan V2 from #10706 was 1.79x slower than V1 (geomean). This PR fixes that. It also speeds up both V1 and V2 by making the Arrow dictionary export cheaper; that export was ~90% of scan CPU on these queries for both scans. V2 and V1 now perform about the same on PolarSignals.
CI
action/bench-sqlona47ec03againstdevelop(CI's PolarSignals job runs V1):Local runs at SF1 on 4 cores, median of 3 rounds × 5 iterations, sides alternated:
Changes
Filtered V2 splits (
vortex-layout). Filtered scans used fixed ~1M-row chunks. A DataFusion byte-range partition of a ~1M-row file was therefore one chunk, so when the selected rows sat in one partition, a single thread did all the work. Chunks are now sized two ways:ScanCache;Cheap projections over large ranges keep the 1M-row chunks.
No re-concatenation (
vortex-layout).ProjectionMorselemitted its pieces as oneChunkedArray, which the Arrow export then concatenated, re-merging nested dictionaries. Pieces are now separate batches; only pieces under 4,096 rows are grouped.Primitive dictionary export (
vortex-arrow). This replacesarrow_cast, which hashed each value with SipHash and built anArrowErrorper value. The export now uses, in order of preference:The full min/max scan runs only when the sample's range could fit the dense table. Nulls stay in the codes. Other types, and codes that would overflow, fall back to
arrow_cast.Bit-packed arrays exported as dictionaries (
vortex-arrow,vortex-fastlanes). A bit-packed array without patches holds only values in0..2^bit_width, so the unpacked values serve directly as codes into that range. This applies up to 16-bit widths.BitPacked'sCastKernelnow also handles narrowing integer casts when there are no patches and the target type holds every value the bit width allows.Tests cover nulls, float bit-equality, a full
UInt8code space, the overflow fallback, the all-distinct path, bit-packed export, and narrowing bit-packed casts.The CodSpeed micro-benchmarks it flags don't run the code this PR changes; see the comment below.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC