Stream selected V2 chunk planners from one file planner by joseph-isaacs · Pull Request #10628 · vortex-data/vortex

joseph-isaacs · 2026-10-20T09:13:19Z

Summary

V2 currently finishes walking the selection and allocating the selected chunk planners before workers can start. A single file planner now emits child planners incrementally through a bounded queue, allowing other tasks to plan, read, and evaluate earlier chunks while the producer continues scanning.

This is a separate draft against develop, including the selection-aware planning changes from #10625. It preserves that draft and provides an alternative to #10621.

Changes

Replace eager split and planner vectors with a lazy selection cursor and one file planner. Indexed and Roaring selections seek to occupied chunks.
Skip empty selections before allocating child planners. Restrict plans to selected row spans and avoid masks and filtering for contiguous selections.
Schedule emitted children independently with bounded lookahead, preserving ordered/unordered scan behavior, shared dictionary values, cancellation, and error propagation.
Keep the core pipeline driver, compiler, Planner trait, and list layouts unchanged.
Retain the historical random-access measurement contract: layout-reader caching and outer Canonical materialization, with nested values remaining lazy. V1 and V2 use matching settings; VORTEX_BENCH_SCAN=v1 selects the baseline.

Validation and performance

151 targeted scan tests passed, including selection boundaries, sparse ranges, producer cancellation, and a two-worker regression test proving child execution overlaps.
Scoped formatting, release-debug benchmark build, and patch whitespace checks passed.
Tests/build used installed Rust 1.98.0; formatting used pinned nightly 2026-09-10. CI will validate the repository-pinned toolchain.

Local sequential measurements covered 36 cases across standard/compact formats, cached/reopen scans, and the default datasets:

Comparison	Geometric mean latency ratio
Standard V2 / V1	0.974×
Compact V2 / V1	0.966×
Standard V2 / pre-change V2	0.990×

These aggregate results do not establish parity in every case. Standard cached taxi uniform access remains 14.9% slower than V1 (3.773 ms versus 3.283 ms); the legacy standard cached case remains 22.9% slower. The pre-change taxi result was 3.866 ms.

Reproduce each V1/V2 run with the same binary and dataset cache:

VORTEX_BENCH_SCAN=v2 taskset -c 2-9 target/release_debug/random-access-bench \
  --formats vortex,vortex-compact --time-limit 2 --open-mode both \
  --display-format gh-json -o results-v2.jsonl

Repeat with VORTEX_BENCH_SCAN=v1 and a separate output file. Normal CI and the random-access/compression benchmark workflows are being requested for this draft.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Use the historical layout-reader cache and outer Canonical materialization for both V1 and V2. Keep nested values lazy so the existing CI series compares equivalent work. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Walk selections lazily and emit chunk planners through a bounded queue so chunk planning and execution overlap the file's selection traversal. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

codspeed · 2026-10-20T09:20:37Z

Merging this PR will regress 3 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

20 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

test_scan_filter[int]

chunked_dict_primitive_into_canonical[f64, (1000, 1000, 10)]

test_scan_filter[int_str_float]

test_scan_filter[int_float]

test_repeated_scan[int_str_float]

multipolygon_mixed_validity

test_scan_filter[int_str]

take_varbin

bench_compare_sliced_dict_primitive[(3333, 10000)]

test_scan[int_str_float]

…and 10 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

test_scan_filter[int]

test_scan_filter_to_arrow_array[int_float]

test_repeated_scan_filter[int_float]

test_repeated_scan_filter[int]

test_scan_filter_to_arrow_array[int]

test_repeated_scan_scalar_at[int_str_float]

test_repeated_scan_scalar_at[int]

test_scan_scalar_at[int_str_float]

test_scan_filter[int_str_float]

test_scan_filter[int_float]

…and 30 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 6 improved benchmarks
❌ 3 regressed benchmarks
✅ 2399 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`starts_with_avx2[8, VarBin]`	40.4 µs	46.6 µs	-13.3%
❌	Simulation	`take[routing/grouped_chunks/primitive/nonnull/chunks=32/indices=1000]`	385 µs	428.5 µs	-10.16%
❌	WallTime	`words_gather_dispatch_avx512[65536]`	915 ns	1,017 ns	-10.03%
⚡ ⚠️	Simulation	`test_scan_filter[int]`	3.4 ms	2.8 ms	+22.83%
⚡ ℹ️	Simulation	`test_scan_filter_to_arrow_array[int_float]`	4.1 ms	3.5 ms	+15.77%
⚡	WallTime	`bitpack_blocked_compress_avx2`	7.6 µs	6.7 µs	+13.46%
⚡ ℹ️	Simulation	`test_repeated_scan_filter[int_float]`	3.8 ms	3.4 ms	+12.23%
⚡ ℹ️	Simulation	`test_repeated_scan_filter[int]`	2.7 ms	2.4 ms	+11.17%
⚡ ℹ️	Simulation	`test_scan_filter_to_arrow_array[int]`	2.8 ms	2.5 ms	+11.07%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/v2-file-planner (d5187c9) with develop (9be4ef1)}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-10-20T09:20:46Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`d5187c9`	random-access-bench	1	Explore Profiling Data
🟢 Done	`d5187c9`	compress-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-10-20T09:28:08Z

Benchmarks: Compression 📖

Commits: PR d5187c94 vs base 61ff28d9

vortex / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	749112068 / 758524188 / -1.2%
compress time/Bimbo	3191221140 / 3271273938 / -2.4%
compress time/CMSprovider	2433222933 / 2415200639 / +0.7%
compress time/Euro2016	253624586 / 254412764 / -0.3%
compress time/Food	304828656 / 309889717 / -1.6%
compress time/HashTags	498551757 / 504567333 / -1.2%
compress time/TPC-H l_comment canonical	564941575 / 565803887 / -0.2%
compress time/TPC-H l_comment chunked	565373627 / 565947036 / -0.1%
compress time/taxi	361097717 / 359860443 / +0.3%
compress time/wide table cols=100 chunks=1 rows=1000	4261569 / 4177299 / +2.0%
compress time/wide table cols=100 chunks=50 rows=1000	4153634 / 4222323 / -1.6%
compress time/wide table cols=1000 chunks=1 rows=1000	40516862 / 41049901 / -1.3%
compress time/wide table cols=1000 chunks=50 rows=1000	39906912 / 41957638 / -4.9%
compress time/wide table cols=10000 chunks=1 rows=1000	475613514 / 470827873 / +1.0%
compress time/wide table cols=10000 chunks=50 rows=1000	464541161 / 472878133 / -1.8%
decompress time/Arade	11836880 / 12018830 / -1.5%
decompress time/Bimbo	55934589 / 56279836 / -0.6%
decompress time/CMSprovider	37334053 / 37422648 / -0.2%
decompress time/Euro2016	11177294 / 11816148 / -5.4%
decompress time/Food	3228798 / 3298628 / -2.1%
decompress time/HashTags	36149109 / 33980554 / +6.4%
decompress time/TPC-H l_comment canonical	13430661 / 13338587 / +0.7%
decompress time/TPC-H l_comment chunked	13391288 / 13235338 / +1.2%
decompress time/taxi	8858006 / 8941793 / -0.9%
decompress time/wide table cols=100 chunks=1 rows=1000	1160583 / 1176628 / -1.4%
decompress time/wide table cols=100 chunks=50 rows=1000	1211102 / 1183174 / +2.4%
decompress time/wide table cols=1000 chunks=1 rows=1000	11552647 / 11375090 / +1.6%
decompress time/wide table cols=1000 chunks=50 rows=1000	11559136 / 11370878 / +1.7%
decompress time/wide table cols=10000 chunks=1 rows=1000	125648722 / 123760742 / +1.5%
decompress time/wide table cols=10000 chunks=50 rows=1000	125056521 / 123149143 / +1.5%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164125572 / 164130260 / -0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (0.998x ➖, 0↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	1.03005855 / 1.03502475 / -0.5%
vortex:parquet-zstd ratio compress time/Bimbo	1.64451559 / 1.64015989 / +0.3%
vortex:parquet-zstd ratio compress time/CMSprovider	2.30607104 / 2.2780073 / +1.2%
vortex:parquet-zstd ratio compress time/Euro2016	0.393480233 / 0.39467854 / -0.3%
vortex:parquet-zstd ratio compress time/Food	0.966622244 / 0.973404696 / -0.7%
vortex:parquet-zstd ratio compress time/HashTags	1.46261638 / 1.46470403 / -0.1%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	0.534644668 / 0.532902529 / +0.3%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	0.535812868 / 0.534069589 / +0.3%
vortex:parquet-zstd ratio compress time/taxi	1.88590913 / 1.87134793 / +0.8%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	1.16061205 / 1.18883001 / -2.4%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	1.19286138 / 1.21911144 / -2.2%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	1.12427697 / 1.15070208 / -2.3%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	1.09941721 / 1.16388149 / -5.5%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	1.2561195 / 1.26557569 / -0.7%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	1.24830939 / 1.26838178 / -1.6%
vortex:parquet-zstd ratio decompress time/Arade	0.242146905 / 0.245570846 / -1.4%
vortex:parquet-zstd ratio decompress time/Bimbo	0.627704604 / 0.63097111 / -0.5%
vortex:parquet-zstd ratio decompress time/CMSprovider	0.36304694 / 0.365313018 / -0.6%
vortex:parquet-zstd ratio decompress time/Euro2016	0.0745317365 / 0.0789361881 / -5.6%
vortex:parquet-zstd ratio decompress time/Food	0.129362382 / 0.13230067 / -2.2%
vortex:parquet-zstd ratio decompress time/HashTags	0.0864687477 / 0.0810134218 / +6.7%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.163405687 / 0.162533318 / +0.5%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.163150327 / 0.161570429 / +1.0%
vortex:parquet-zstd ratio decompress time/taxi	0.151321938 / 0.152816464 / -1.0%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.571051319 / 0.576607428 / -1.0%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.592519134 / 0.57788421 / +2.5%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.490509875 / 0.482566342 / +1.6%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.499396057 / 0.483326112 / +3.3%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.491963269 / 0.491314093 / +0.1%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.499250455 / 0.492123032 / +1.4%
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33458157 / 1.33461969 / -0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (1.000x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	727251926 / 732856084 / -0.8%
parquet_rs-zstd compress time/Bimbo	1940523497 / 1994484780 / -2.7%
parquet_rs-zstd compress time/CMSprovider	1055137892 / 1060225153 / -0.5%
parquet_rs-zstd compress time/Euro2016	644567540 / 644607543 / -0.0%
parquet_rs-zstd compress time/Food	315354481 / 318356505 / -0.9%
parquet_rs-zstd compress time/HashTags	340862966 / 344484157 / -1.1%
parquet_rs-zstd compress time/TPC-H l_comment canonical	1056667370 / 1061739917 / -0.5%
parquet_rs-zstd compress time/TPC-H l_comment chunked	1055169932 / 1059687816 / -0.4%
parquet_rs-zstd compress time/taxi	191471429 / 192300126 / -0.4%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	3671829 / 3513790 / +4.5%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	3482076 / 3463443 / +0.5%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	36038150 / 35673787 / +1.0%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	36298242 / 36049751 / +0.7%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	378637156 / 372026642 / +1.8%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	372136238 / 372820031 / -0.2%
parquet_rs-zstd decompress time/Arade	48883053 / 48942414 / -0.1%
parquet_rs-zstd decompress time/Bimbo	89109732 / 89195583 / -0.1%
parquet_rs-zstd decompress time/CMSprovider	102835333 / 102439952 / +0.4%
parquet_rs-zstd decompress time/Euro2016	149966907 / 149692407 / +0.2%
parquet_rs-zstd decompress time/Food	24959327 / 24932814 / +0.1%
parquet_rs-zstd decompress time/HashTags	418059819 / 419443510 / -0.3%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	82192127 / 82066786 / +0.2%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	82079443 / 81916834 / +0.2%
parquet_rs-zstd decompress time/taxi	58537487 / 58513283 / +0.0%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2032362 / 2040605 / -0.4%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2043988 / 2047424 / -0.2%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	23552323 / 23572075 / -0.1%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23146230 / 23526306 / -1.6%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	255402649 / 251897399 / +1.4%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	250488547 / 250240560 / +0.1%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (0.997x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	99502497 / 100215714 / -0.7%
arrow-ipc compress time/Bimbo	622549898 / 635589410 / -2.1%
arrow-ipc compress time/CMSprovider	389069136 / 390948643 / -0.5%
arrow-ipc compress time/Euro2016	38288215 / 38595296 / -0.8%
arrow-ipc compress time/Food	33858183 / 34177711 / -0.9%
arrow-ipc compress time/HashTags	78608701 / 79122787 / -0.6%
arrow-ipc compress time/TPC-H l_comment canonical	938836551 / 940180187 / -0.1%
arrow-ipc compress time/TPC-H l_comment chunked	936281596 / 941274915 / -0.5%
arrow-ipc compress time/taxi	74921318 / 75583954 / -0.9%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	200893 / 204921 / -2.0%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	201674 / 201027 / +0.3%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2240869 / 2260609 / -0.9%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2227587 / 2218879 / +0.4%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25373683 / 25653899 / -1.1%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25359460 / 25493022 / -0.5%
arrow-ipc decompress time/Arade	210909503 / 212031427 / -0.5%
arrow-ipc decompress time/Bimbo	409616940 / 413855809 / -1.0%
arrow-ipc decompress time/CMSprovider	695417602 / 679048161 / +2.4%
arrow-ipc decompress time/Euro2016	85742579 / 83994413 / +2.1%
arrow-ipc decompress time/Food	65845608 / 66250501 / -0.6%
arrow-ipc decompress time/HashTags	164603067 / 161450843 / +2.0%
arrow-ipc decompress time/TPC-H l_comment canonical	631126974 / 623843409 / +1.2%
arrow-ipc decompress time/TPC-H l_comment chunked	629976073 / 624218751 / +0.9%
arrow-ipc decompress time/taxi	50902514 / 51288021 / -0.8%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	320512 / 318607 / +0.6%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	320159 / 320260 / -0.0%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3511073 / 3583377 / -2.0%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3488723 / 3479159 / +0.3%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	36102359 / 36848739 / -2.0%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36623472 / 36411339 / +0.6%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

github-actions · 2026-10-20T09:50:28Z

Benchmarks: Random Access 📖

Commits: PR d5187c94 vs base 61ff28d9
Verdict: No clear signal (medium confidence)
Attributed Vortex impact: -8.0%
Engines: random-access No clear signal (-8.0%, medium confidence)
Vortex (geomean): hot 0.918x ➖
Parquet (geomean): hot 0.997x ➖
Shifts: Parquet (control) -0.3% · Median polish -3.2%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

vortex / arrow-ipc / ns (1.066x ➖, 1↑ 5↓)

name	ns (PR / base / %diff)
random-access/arrow-tokio-local-disk	73442 / 67936 / +8.1%
random-access/arrow-tokio-local-disk-footer	170116 / 158515 / +7.3%
random-access/feature-vectors/correlated/arrow-tokio-local-disk	2520874 / 2553640 / -1.3%
random-access/feature-vectors/correlated/arrow-tokio-local-disk-footer	2098326 / 1913523 / +9.7%
random-access/feature-vectors/uniform/arrow-tokio-local-disk	31729522 / 54103073 / -41.4% 🟢
random-access/feature-vectors/uniform/arrow-tokio-local-disk-footer	56277382 / 29949513 / +87.9% 🔴
random-access/nested-lists/correlated/arrow-tokio-local-disk	57850 / 55338 / +4.5%
random-access/nested-lists/correlated/arrow-tokio-local-disk-footer	104647 / 97032 / +7.8%
random-access/nested-lists/uniform/arrow-tokio-local-disk	1026850 / 1025451 / +0.1%
random-access/nested-lists/uniform/arrow-tokio-local-disk-footer	1108506 / 1110731 / -0.2%
random-access/nested-structs/correlated/arrow-tokio-local-disk	54543 / 49037 / +11.2% 🔴
random-access/nested-structs/correlated/arrow-tokio-local-disk-footer	95416 / 85395 / +11.7% 🔴
random-access/nested-structs/uniform/arrow-tokio-local-disk	886800 / 792307 / +11.9% 🔴
random-access/nested-structs/uniform/arrow-tokio-local-disk-footer	911222 / 837566 / +8.8%
random-access/taxi/correlated/arrow-tokio-local-disk	123138 / 113520 / +8.5%
random-access/taxi/correlated/arrow-tokio-local-disk-footer	223065 / 212011 / +5.2%
random-access/taxi/uniform/arrow-tokio-local-disk	2968730 / 2663156 / +11.5% 🔴
random-access/taxi/uniform/arrow-tokio-local-disk-footer	2964423 / 2840547 / +4.4%

random-access / vortex-file-compressed / ns (0.918x ➖, 7↑ 1↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/vortex-tokio-local-disk	239611 / 234005 / +2.4%
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer	1152616 / 1199748 / -3.9%
random-access/feature-vectors/uniform/vortex-tokio-local-disk	1014696 / 1370878 / -26.0% 🟢
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer	2213062 / 2576171 / -14.1% 🟢
random-access/nested-lists/correlated/vortex-tokio-local-disk	155591 / 200355 / -22.3% 🟢
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer	298005 / 384975 / -22.6% 🟢
random-access/nested-lists/uniform/vortex-tokio-local-disk	704866 / 678080 / +4.0%
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer	869094 / 879596 / -1.2%
random-access/nested-structs/correlated/vortex-tokio-local-disk	219622 / 234324 / -6.3%
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer	412659 / 427555 / -3.5%
random-access/nested-structs/uniform/vortex-tokio-local-disk	486492 / 595545 / -18.3% 🟢
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer	789436 / 862067 / -8.4%
random-access/taxi/correlated/vortex-tokio-local-disk	642554 / 596962 / +7.6%
random-access/taxi/correlated/vortex-tokio-local-disk-footer	1221237 / 1184712 / +3.1%
random-access/taxi/uniform/vortex-tokio-local-disk	1842493 / 2472705 / -25.5% 🟢
random-access/taxi/uniform/vortex-tokio-local-disk-footer	2623417 / 3425154 / -23.4% 🟢
random-access/vortex-tokio-local-disk	486271 / 383160 / +26.9% 🔴
random-access/vortex-tokio-local-disk-footer	871950 / 859433 / +1.5%

random-access / parquet / ns (0.997x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/parquet-tokio-local-disk	107251968414 / 112308985889 / -4.5%
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer	108257264222 / 112844970780 / -4.1%
random-access/feature-vectors/uniform/parquet-tokio-local-disk	107183270683 / 106964234141 / +0.2%
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer	107516670266 / 107637654388 / -0.1%
random-access/nested-lists/correlated/parquet-tokio-local-disk	146822153 / 149164855 / -1.6%
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer	145399572 / 144865077 / +0.4%
random-access/nested-lists/uniform/parquet-tokio-local-disk	146517681 / 144796783 / +1.2%
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer	147687381 / 144804018 / +2.0%
random-access/nested-structs/correlated/parquet-tokio-local-disk	45217685 / 44577966 / +1.4%
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer	45504757 / 44946796 / +1.2%
random-access/nested-structs/uniform/parquet-tokio-local-disk	45370123 / 44603223 / +1.7%
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer	44919846 / 44777426 / +0.3%
random-access/parquet-tokio-local-disk	117578454 / 118927964 / -1.1%
random-access/parquet-tokio-local-disk-footer	117448981 / 117487949 / -0.0%
random-access/taxi/correlated/parquet-tokio-local-disk	176668512 / 176466380 / +0.1%
random-access/taxi/correlated/parquet-tokio-local-disk-footer	176197645 / 176544200 / -0.2%
random-access/taxi/uniform/parquet-tokio-local-disk	185877830 / 185059062 / +0.4%
random-access/taxi/uniform/parquet-tokio-local-disk-footer	185703093 / 189985177 / -2.3%

random-access / lance / ns (1.010x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/lance-tokio-local-disk	212293 / 213609 / -0.6%
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer	1026458 / 1010450 / +1.6%
random-access/feature-vectors/uniform/lance-tokio-local-disk	1066487 / 1040514 / +2.5%
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer	1884176 / 1869792 / +0.8%
random-access/lance-tokio-local-disk	547420 / 548107 / -0.1%
random-access/lance-tokio-local-disk-footer	1309160 / 1291762 / +1.3%
random-access/nested-lists/correlated/lance-tokio-local-disk	160981 / 160509 / +0.3%
random-access/nested-lists/correlated/lance-tokio-local-disk-footer	497704 / 488460 / +1.9%
random-access/nested-lists/uniform/lance-tokio-local-disk	905424 / 914257 / -1.0%
random-access/nested-lists/uniform/lance-tokio-local-disk-footer	1438467 / 1410465 / +2.0%
random-access/nested-structs/correlated/lance-tokio-local-disk	279769 / 279140 / +0.2%
random-access/nested-structs/correlated/lance-tokio-local-disk-footer	571553 / 563064 / +1.5%
random-access/nested-structs/uniform/lance-tokio-local-disk	2469072 / 2324098 / +6.2%
random-access/nested-structs/uniform/lance-tokio-local-disk-footer	2650225 / 2602463 / +1.8%
random-access/taxi/correlated/lance-tokio-local-disk	787848 / 796747 / -1.1%
random-access/taxi/correlated/lance-tokio-local-disk-footer	1818009 / 1823407 / -0.3%
random-access/taxi/uniform/lance-tokio-local-disk	8257129 / 8214856 / +0.5%
random-access/taxi/uniform/lance-tokio-local-disk-footer	8980391 / 8946234 / +0.4%

joseph-isaacs added 4 commits October 19, 2026 23:59

Plan V2 projections from known row selections

3cd67a2

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Preserve unchanged selection masks across projection fields

5012ca3

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Restore the random-access benchmark measurement contract

d326929

Use the historical layout-reader cache and outer Canonical materialization for both V1 and V2. Keep nested values lazy so the existing CI series compares equivalent work. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Stream V2 chunk planners from a single file planner

d5187c9

Walk selections lazily and emit chunk planners through a bounded queue so chunk planning and execution overlap the file's selection traversal. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

joseph-isaacs added action/bench-compress Run only the compression benchmark on this PR action/bench-random-access Run only the random-access benchmark on this PR labels Oct 20, 2026 — with ChatGPT Codex Connector

github-actions Bot removed action/bench-compress Run only the compression benchmark on this PR action/bench-random-access Run only the random-access benchmark on this PR labels Oct 20, 2026

joseph-isaacs added the changelog/performance A performance improvement label Oct 20, 2026 — with ChatGPT Codex Connector

joseph-isaacs closed this Oct 20, 2026

Navigation Menu

Repository navigation

Stream selected V2 chunk planners from one file planner - #10628

Stream selected V2 chunk planners from one file planner#10628
joseph-isaacs wants to merge 4 commits into
developfrom
ji/v2-file-planner

joseph-isaacs commented Oct 20, 2026

Uh oh!

codspeed Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

joseph-isaacs commented Oct 20, 2026

Summary

Changes

Validation and performance

Uh oh!

codspeed Bot commented Oct 20, 2026

Merging this PR will regress 3 benchmarks

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Oct 20, 2026

Benchmarks: Compression 📖

Uh oh!

github-actions Bot commented Oct 20, 2026

Benchmarks: Random Access 📖

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 20, 2026 •

edited

Loading