Plan V2 random access by the segments a selection reads by joseph-isaacs · Pull Request #10621 · vortex-data/vortex

joseph-isaacs · 2026-10-19T19:01:18Z

Summary

Makes the V2 scan builder faster than the default scan for random access overall, and switches the random-access benchmark to V2 for every Vortex file. This is an alternative to #10537. The changes stay in the V2 planner, the plans, the selection handling and V2's segment IO. They don't change the Planner trait or the planning driver, and they add no layout-reader caching.

A V2 take was slower than the default scan because of work that did not depend on the rows it selected:

A support check walked the whole layout tree. Every scan deserialized every chunk of the file before reading any row. The walk is removed. Lowering already fails, when a split plans it, any layout that has no pipeline implementation. The default writer never writes a list layout, so V2 neither writes nor reads one.
The same segment was read several times. Fixed 64K-row work chunks cut taxi's 178K-row stored chunks into three, and each split read and decoded the same segments again (1,002 segment requests for 337 segments on taxi/uniform). Selected runs that read a stored segment in common now share a planner, across chunk boundaries, while the planner spans at most 256K rows. A segment that several planners still reach is read once: the scan's IO counts each segment's planners and hands every planner the same read.
Plans were built for unselected chunks. A planner's single hull split made the pipeline's shared-segment count reach, and therefore lower, every chunk between selected runs (441 lowered layouts to read 9 segments on feature-vectors/correlated). Consecutive runs that read exactly the same segments share one masked split, which provably visits no extra chunk. Other runs get splits of their own, all active at once. Scan::try_new counts shared segments over each split's rows instead of their hull, which finds the same shared segments and readers. Feature-vectors/correlated now lowers 14 layouts.
Per-scan allocation. Each lowered chunked layout re-read every chunk's row count instead of using the offsets the layout already holds, and each scan built a hash map over every segment, which is now a binary search when segment locations are in offset order.

Changes

vortex-layout/src/scan/v2: split_rows groups selected runs into bounded planners and pipeline splits by the segments they reach, and counts segments read by several planners. SegmentScanIo shares those reads. The eager layout support walk is removed. SegmentRanges searches sorted locations in place.
vortex-layout/src/scan/execution: a projection planner carries its pipeline splits, runs them all at once, and orders the output pieces by split.
vortex-layout/src/plan/physical: Scan::try_new reaches each split's rows instead of their hull. lower_chunked uses ChunkedLayout::chunk_offsets. PlanChildren::lazy allocates once.
vortex-bench/src/random_access/take.rs: Vortex takes use V2, as in Add planner-owned selection and bounded ListLayout scans to V2 #10537.
benchmarks/random-access-bench/benches/v2_take.rs: a divan benchmark of each query with both scans, plus an empty V2 scan.
vortex-sqllogictest/slt/tpch/generate_data.sh: pins tpchgen-cli@4.0.0 and uses its parquet subcommand. Version 4.0.0 was released on October 19, 2026 and broke the unpinned call on every branch.

Tests: splits_read_each_stored_chunk_once covers runs in one stored chunk, runs in one work chunk but different stored chunks, and runs in different work chunks but one stored chunk. sessions_share_a_segment_read covers the shared reads, and segment_ranges_find_each_segment covers lookup with sorted and unsorted locations. The existing matches_v1 row-index cases check the output order of concurrent splits.

CI random-access results

From the action/bench-random-access run on 2dab888, against the default scan on the base (cached handle, ns per take). The Vortex geomean is 0.964x, down from 1.235x on the first commit of this PR.

Query	Default scan (base)	V2 (`2dab888`)
feature-vectors/correlated	233,649	193,818
feature-vectors/uniform	1,378,299	1,221,799
nested-lists/correlated	218,309	183,990
nested-lists/uniform	677,257	701,926
nested-structs/correlated	242,718	228,535
nested-structs/uniform	586,191	618,898
taxi/correlated	594,014	688,362
taxi/uniform	2,343,191	2,332,568
taxi, fixed indices	387,324	491,468

With the file reopened for every take, V2 is at or below the default scan on every row except nested-lists/uniform and nested-structs/uniform. The remaining gaps are in small taxi selections. There, per-take decoding and plan setup that the default scan's layout-reader cache avoids are still a large share of the cost.

Checks: cargo test -p vortex-layout --lib (550 passed); cargo clippy -p vortex-layout -p vortex-bench --all-targets --all-features -- -D warnings; cargo clippy -p random-access-bench --all-targets -- -D warnings (the lance feature needs protoc, which is not installed here); cargo fmt --check with the nightly pinned in CI. The TPC-H generation step runs locally with ZSTD output, and SQL logic tests pass in CI.

🤖 Generated with Claude Code

https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Rebase draft PR #10537 onto develop 92a7db0. Apply selections through a composable planner, resolve list offsets before selected elements, and split substantial fixed-width child projections at stored segment boundaries. The projection planner owns subdivision, byte-budget admission, child task execution and ordered collection. Unsupported splits retain the serial path. Keep Planner and Morsel trait methods unchanged. Replace the experimental completion callbacks, WorkFuture and ChildExecutor traits with one PlannerOutput::Await variant; morsels only publish the assembled result. Preserve develop's global local-file coalescing configuration. The 8 KiB experiment caused tiny-lookup regressions and is not part of this version. Preserve masked lazy lowering, bounded dense-list grouping, shared value caches, validated chunk offsets and the fieldless nullable-struct fix. Retain the diagnostic examples and regression benchmark workflow. Backtracking: previous PR head 8442d46; experimental CI head 60fda25; local pre-review checkpoint a253e5727 and reviewed checkpoint e8fd21a4a. The byte budget covers in-flight decoded work, not retained output or encoded/coalesced buffers. Full performance revalidation remains necessary. Validation: 692 layout and 79 scan library tests passed, followed by all 7 await protocol tests including an additional await/IO regression. Scoped all-target/all-feature clippy passed with -D warnings; formatted with nightly-2026-09-10. Focused E2E measurements follow this checkpoint. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Use the planner-driven V2 ScanBuilder directly for every Vortex random access measurement, including object-store reads. Remove the reader toggle, its configuration plumbing, V1 documentation, and unused reader-cache setup. Remove the experiment-only regression workflow, driver script, and three fixture/selection examples, along with their futures dev dependency. Keep the planner implementation and regression tests. The previous commit c4fa781 preserves the diagnostic harnesses and benchmark context. Inspected the resulting diff and searched for stale switch/example references. No tests, builds, or benchmarks were rerun for this cleanup. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Once a ListSource has its offsets, it previously compiled a nested Scan over the element plan. When the selected element ranges reach only segment chunks (Concat/Filter/Zoned/LegacyStats over SegmentScan), the list source now issues those reads itself: each touched segment is requested once, all together, decoded on arrival and sliced or masked into the ranges it holds, in list order. Other element plans (dict, struct, nested lists) keep the nested Scan. No operator, pipeline or driver API changes. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Drop the child projection path and every core change it needed: PlannerOutput::Await and the driver's task tracking, SourceBlocked::Child, the Cx child inbox, Turn::Child and Scan::complete_child, AdaptorStep::Child, Compiler::plan_children, the Children executor and its split planning, and the ProjectionPlanner/ProjectedBatch conversion. List sources now always run serially, reading segment elements directly or through a nested scan. Also restore Reach, Concat::reach and Shares to develop. Scan::try_new now counts shared readers over the row runs the splits select, merged when adjacent, so chunks in unselected gaps stay lazy without a mask-aware Reach variant. Each run counts as a reader, so a segment reused by distinct chunks is still decoded once. Splits with more than 1024 runs count their whole range. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Reduce this branch to a minimal change against develop: drop list support, the selection planner and every compiler and generic trait change. V2 split planning now groups each fixed chunk's selected rows. A selected run joins the group before it while the unselected gap is at most one writer row block (8,192 rows), or 1 MiB of serialized data when that spans more rows. Each group becomes its own planner root over just its rows, with the selection's keep mask, so rows in wider gaps get no planner, plan or read. The random-access benchmark reads files with the V2 scan when it supports their layouts and falls back to the default scan otherwise. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Replace the 8,192-row / 1 MiB gap estimate: within each fixed chunk, a selected run joins the previous group when both reach a stored segment in common, so no segment is read by two groups. Local random-access benchmarks against develop's V2 (3 interleaved passes) show uniform access regressing by 8-14% with both this grouping and the gap estimate: extra planner roots split each chunk's reads across tasks. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Revert the selected-run grouping to develop's one planner root per fixed chunk, and narrow that root to the rows from its first selected row to its last, with the selection mask sliced to match. Develop's compiler already builds nothing for chunks the mask does not touch, so rows outside the span get no plan and no read, without splitting a chunk's reads across tasks. Local random-access benchmarks against develop's V2, 3 interleaved passes each (median, 4-core VM, run-to-run spread up to 20%): - this change: geomean -0.7%, no consistent regression; - grouping by an 8,192-row / 1 MiB gap: geomean +0.0%, uniform +8-11%; - grouping by touched segments: geomean +2.8%, uniform +8-14%. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

codspeed · 2026-10-19T19:08:21Z

Merging this PR will regress 1 benchmark

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

20 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

chunked_dict_primitive_into_canonical[f64, (1000, 1000, 10)]

compress_fsst[(500, 64, 8)]

test_scan_filter[int_str_float]

test_scan_filter[int]

multipolygon_mixed_validity

test_scan_filter[int_str]

test_scan_filter[int_float]

random_i128[0.01]

test_scan[int_str]

take_varbin

…and 10 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

test_scan_filter[int_str_float]

test_scan_filter[int]

test_repeated_scan_scalar_at[int_str_float]

test_scan_filter_to_arrow_array[int_str_float]

test_scan_scalar_at[int_str_float]

test_repeated_scan[int]

test_repeated_scan_filter[int_str_float]

test_repeated_scan_scalar_at[int_float]

test_scan_read_all_to_arrow_array[int]

test_repeated_scan_scalar_at[int_str]

…and 30 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 2404 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`take[routing/grouped_chunks/primitive/nonnull/chunks=32/indices=1000]`	385 µs	429.5 µs	-10.36%
⚡	Simulation	`chunked_opt_bool_canonical_into[(1000, 10)]`	65.4 µs	55.1 µs	+18.6%
⚡	WallTime	`starts_with_avx2[64, VarBin]`	60.3 µs	53.3 µs	+13.03%
⚡	WallTime	`words_gather_dispatch_neon[65536]`	2 µs	1.8 µs	+10.04%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/charming-cerf-9hlmns (2d5cd2b) with develop (9be4ef1)}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

joseph-isaacs · 2026-10-19T19:09:11Z

CodSpeed Performance Analysis is red, but none of its flagged regressions run code this PR changes:

On 2dab888 against develop (61ff28d), CodSpeed flags filter_neon[8], filter_neon[64] and filter_neon[80] (encodings/fastlanes/benches/bitpacking_filter.rs), bitpack_blocked_compress_avx2 (encodings/fastlanes/benches/bitpack_blocked.rs), and take[routing/grouped_chunks/…] (vortex-array/benches/take_chunked.rs). The first four are WallTime runs on standard hosted runners, which CodSpeed warns give inconsistent data.
This PR changes neither fastlanes nor vortex-array. In the same run, density_sweep_dense_runs[0.9], chunked_opt_bool_canonical_into and test_repeated_scan_scalar_at[int_str], which this PR also leaves alone, improved. That fits run-to-run noise.
This PR's code changes are in vortex-layout's V2 scan, planning, pipeline setup and V2 segment IO, which no CodSpeed benchmark here exercises, plus the random-access benchmark harness and the TPC-H data script.

No code fix applies here. Acknowledging the result on CodSpeed is left to a maintainer. The random-access benchmark (action/bench-random-access) is the relevant measurement for this change.

Generated by Claude Code

github-actions · 2026-10-19T19:14:51Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`2dab888`	random-access-bench	1	Explore Profiling Data

Previous Runs (2)

Status	Commit	Job	Attempt	Link
🟢 Done	`37d689f`	random-access-bench	1	Explore Profiling Data
🟢 Done	`a15cf39`	random-access-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-10-19T19:42:12Z

Benchmarks: Random Access 📖

Commits: PR 2dab8888 vs base 92a7db07
Verdict: No clear signal (environment too noisy confidence)
Attributed Vortex impact: -4.9%
Engines: random-access No clear signal (-4.9%, environment too noisy confidence)
Vortex (geomean): hot 0.964x ➖
Parquet (geomean): hot 1.015x ➖
Shifts: Parquet (control) +1.5% · Median polish -1.6%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

vortex / arrow-ipc / ns (1.016x ➖, 2↑ 3↓)

name	ns (PR / base / %diff)
random-access/arrow-tokio-local-disk	70881 / 68607 / +3.3%
random-access/arrow-tokio-local-disk-footer	164722 / 168527 / -2.3%
random-access/feature-vectors/correlated/arrow-tokio-local-disk	1358448 / 2522808 / -46.2% 🟢
random-access/feature-vectors/correlated/arrow-tokio-local-disk-footer	2037039 / 3493249 / -41.7% 🟢
random-access/feature-vectors/uniform/arrow-tokio-local-disk	55364284 / 30138993 / +83.7% 🔴
random-access/feature-vectors/uniform/arrow-tokio-local-disk-footer	47678462 / 32308217 / +47.6% 🔴
random-access/nested-lists/correlated/arrow-tokio-local-disk	65548 / 60045 / +9.2%
random-access/nested-lists/correlated/arrow-tokio-local-disk-footer	108714 / 104170 / +4.4%
random-access/nested-lists/uniform/arrow-tokio-local-disk	1126676 / 1078953 / +4.4%
random-access/nested-lists/uniform/arrow-tokio-local-disk-footer	1192103 / 1097061 / +8.7%
random-access/nested-structs/correlated/arrow-tokio-local-disk	57328 / 54612 / +5.0%
random-access/nested-structs/correlated/arrow-tokio-local-disk-footer	97157 / 94861 / +2.4%
random-access/nested-structs/uniform/arrow-tokio-local-disk	995370 / 903216 / +10.2% 🔴
random-access/nested-structs/uniform/arrow-tokio-local-disk-footer	986394 / 927192 / +6.4%
random-access/taxi/correlated/arrow-tokio-local-disk	119950 / 124134 / -3.4%
random-access/taxi/correlated/arrow-tokio-local-disk-footer	218391 / 225219 / -3.0%
random-access/taxi/uniform/arrow-tokio-local-disk	2819093 / 2813735 / +0.2%
random-access/taxi/uniform/arrow-tokio-local-disk-footer	2981337 / 2962933 / +0.6%

random-access / vortex-file-compressed / ns (0.964x ➖, 5↑ 2↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/vortex-tokio-local-disk	193818 / 233649 / -17.0% 🟢
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer	1108195 / 1147837 / -3.5%
random-access/feature-vectors/uniform/vortex-tokio-local-disk	1221799 / 1378299 / -11.4% 🟢
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer	2219648 / 2565372 / -13.5% 🟢
random-access/nested-lists/correlated/vortex-tokio-local-disk	183990 / 218309 / -15.7% 🟢
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer	310819 / 353664 / -12.1% 🟢
random-access/nested-lists/uniform/vortex-tokio-local-disk	701926 / 677257 / +3.6%
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer	869308 / 873786 / -0.5%
random-access/nested-structs/correlated/vortex-tokio-local-disk	228535 / 242718 / -5.8%
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer	397906 / 437522 / -9.1%
random-access/nested-structs/uniform/vortex-tokio-local-disk	618898 / 586191 / +5.6%
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer	801509 / 796521 / +0.6%
random-access/taxi/correlated/vortex-tokio-local-disk	688362 / 594014 / +15.9% 🔴
random-access/taxi/correlated/vortex-tokio-local-disk-footer	1115932 / 1199816 / -7.0%
random-access/taxi/uniform/vortex-tokio-local-disk	2332568 / 2343191 / -0.5%
random-access/taxi/uniform/vortex-tokio-local-disk-footer	3109937 / 3363875 / -7.5%
random-access/vortex-tokio-local-disk	491468 / 387324 / +26.9% 🔴
random-access/vortex-tokio-local-disk-footer	843520 / 871456 / -3.2%

random-access / parquet / ns (1.015x ➖, 1↑ 1↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/parquet-tokio-local-disk	109223243294 / 106355839970 / +2.7%
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer	161460405068 / 113042205755 / +42.8% 🔴
random-access/feature-vectors/uniform/parquet-tokio-local-disk	108819705937 / 132160209457 / -17.7% 🟢
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer	107422037507 / 107764144489 / -0.3%
random-access/nested-lists/correlated/parquet-tokio-local-disk	147530285 / 143249847 / +3.0%
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer	147235043 / 145152372 / +1.4%
random-access/nested-lists/uniform/parquet-tokio-local-disk	147833292 / 145444723 / +1.6%
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer	148190371 / 147220794 / +0.7%
random-access/nested-structs/correlated/parquet-tokio-local-disk	43920938 / 44732303 / -1.8%
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer	43960941 / 44823933 / -1.9%
random-access/nested-structs/uniform/parquet-tokio-local-disk	44052494 / 44715601 / -1.5%
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer	44343466 / 45113025 / -1.7%
random-access/parquet-tokio-local-disk	121159893 / 119938138 / +1.0%
random-access/parquet-tokio-local-disk-footer	120310077 / 118538394 / +1.5%
random-access/taxi/correlated/parquet-tokio-local-disk	182021627 / 179834753 / +1.2%
random-access/taxi/correlated/parquet-tokio-local-disk-footer	180276482 / 177573135 / +1.5%
random-access/taxi/uniform/parquet-tokio-local-disk	191524902 / 192945410 / -0.7%
random-access/taxi/uniform/parquet-tokio-local-disk-footer	193568055 / 187342599 / +3.3%

random-access / lance / ns (0.995x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/lance-tokio-local-disk	213052 / 209684 / +1.6%
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer	976123 / 1009277 / -3.3%
random-access/feature-vectors/uniform/lance-tokio-local-disk	1065867 / 1065477 / +0.0%
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer	1828645 / 1894783 / -3.5%
random-access/lance-tokio-local-disk	543589 / 548108 / -0.8%
random-access/lance-tokio-local-disk-footer	1314945 / 1317622 / -0.2%
random-access/nested-lists/correlated/lance-tokio-local-disk	162987 / 159403 / +2.2%
random-access/nested-lists/correlated/lance-tokio-local-disk-footer	490088 / 495934 / -1.2%
random-access/nested-lists/uniform/lance-tokio-local-disk	893151 / 915993 / -2.5%
random-access/nested-lists/uniform/lance-tokio-local-disk-footer	1428214 / 1423728 / +0.3%
random-access/nested-structs/correlated/lance-tokio-local-disk	278910 / 278613 / +0.1%
random-access/nested-structs/correlated/lance-tokio-local-disk-footer	561660 / 561231 / +0.1%
random-access/nested-structs/uniform/lance-tokio-local-disk	2225491 / 2270739 / -2.0%
random-access/nested-structs/uniform/lance-tokio-local-disk-footer	2611232 / 2646167 / -1.3%
random-access/taxi/correlated/lance-tokio-local-disk	784724 / 790786 / -0.8%
random-access/taxi/correlated/lance-tokio-local-disk-footer	1815568 / 1825701 / -0.6%
random-access/taxi/uniform/lance-tokio-local-disk	8176181 / 8031996 / +1.8%
random-access/taxi/uniform/lance-tokio-local-disk-footer	8718376 / 8661295 / +0.7%

V2 takes were slower than the default scan on CI. Micro benchmarks of each random-access query, over files in the page cache, traced it to work that does not depend on the selected rows: - Each scan checked support by walking the file's whole layout tree, deserializing every chunk: 245 us on feature vectors before any row. Lowering already fails a layout without a pipeline implementation when a split plans it, so the walk is removed. No list layout is written by default, and V2 neither writes nor reads one. - Fixed 64K-row work chunks cut taxi's 178K-row stored chunks into three, and each split read and decoded the same segments again: 1,002 segment requests for 337 segments on taxi/uniform. Selected runs that read a stored segment in common now share a planner, across chunk boundaries, so every segment is read once. - A planner's single hull split made the pipeline reach, and so lower, every chunk between selected runs: 441 lowered layouts for 9 segments on feature-vectors/correlated. Runs reading exactly the same segments share a masked split; others get splits of their own, all active at once, and the pipeline's shared-segment count reaches each split's rows instead of their hull. 441 lowered layouts become 14. - Lowering a chunked layout reused none of the chunk offsets the layout already holds, reading every chunk's row count again. The random-access benchmark reads every Vortex file with V2, without the support probe or the default scan's layout reader cache. A v2_take divan benchmark runs each query with both scans. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Each V2 scan built a hash map from every segment's byte range to its id, so the IO source could answer range requests. Writers lay segments out in offset order, so the file's location table is searched in place instead, falling back to the map when it is not sorted. An empty scan of the feature-vectors file, 3,917 segments, drops from 85 to 59 us. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

tpchgen-cli 4.0.0, released 2026-10-19, removed the --format and --parquet-compression flags the SQL logic test data script passed to an unpinned `uvx tpchgen-cli`, so the job failed before any test ran. Pin 4.0.0 and use `parquet --compression='ZSTD(3)'`, which writes ZSTD column chunks as before. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

joseph-isaacs · 2026-10-20T00:07:42Z

SQL logic tests failed on 37d689f, but not because of this PR's changes. The job failed while generating data, before any test ran:

uvx tpchgen-cli -s 0.1 --format=parquet ...
error: unexpected argument '--format' found

vortex-sqllogictest/slt/tpch/generate_data.sh runs tpchgen-cli unpinned. Version 4.0.0 was published on October 19, 2026 at 23:32 UTC, seven minutes before this job started, and it replaced --format/--parquet-compression with a parquet subcommand. Every branch, develop included, fails the same way.

I found no existing fix, so 85613cf adds one to this PR: it pins tpchgen-cli@4.0.0 and calls parquet --compression='ZSTD(3)'. I checked locally that the script generates all eight TPC-H tables with ZSTD column chunks. The AGENTS.md benchmark-data note is updated to the new flag. Once develop has the same fix, this change no longer does anything.

Generated by Claude Code

CI showed taxi random access regressing: a column whose segments span most of the file made every selected run share a segment, so the whole take became one planner on one thread. A planner now grows only while it spans at most 256K rows, so taxi/uniform runs as 11 planners. A segment that several planners still reach is read once: the scan's IO counts each segment's planners and hands every planner the same read, dropping it once the last has taken it. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Planner tasks were spawned only when the stream's consumer first polled it, after the scan was handed back from the CPU pool. The first `concurrency` tasks now start on the CPU thread right after planning; the rest still start as earlier ones finish. In memory, taxi takes start their tasks 20 to 34 us sooner. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

The scan planned every planner, reaching every selected run, before it spawned any. Planners are now yielded one at a time: a planner is complete once the run after it is known, so it is spawned while the rest are still planned. Shared reads are registered between neighbouring planners as each closes, since a segment two planners read is read by planners next to each other. In memory, taxi/uniform's first planner task starts at about 370 us instead of 830 us. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

A V2 scan now has one root planner, the file planner, which plans the scan on its first step and hands out one unit per step after that. The driver gains distributing roots, whose children are handed back to the caller instead of run, and can admit a handed-off planner or morsel as a root. A new pool in vortex-scan runs those units. Any spare worker steps the file planner, and each unit it hands out waits in a shared queue until a worker claims it and admits it to its own run. Everything a unit emits stays on that worker's run. A worker steps the file planner before its own work whenever fewer units wait than workers are parked, spawns another worker when a unit has no parked worker to take it, and claims more units while the ones it holds wait for IO. Output is reordered by hand-out order when the scan is ordered, and the file planner stops while the consumer falls behind. This replaces the task per unit and the CPU-pool hop that prepared the scan before any unit was planned. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

joseph-isaacs · 2026-10-20T09:44:24Z

Two CI failures on 2d5cd2b come from outside this PR's diff:

Rust tests (cpu-features, sve): the runner timed out connecting to static.rust-lang.org while installing the toolchain, before anything was built or tested.
Rust tests (linux-arm64): vortex-io runtime::tokio::tests::test_spawn_and_abort failed at tokio.rs:220, assertion failed: send.send(()).is_err(). The other 11,141 tests passed. This PR doesn't touch vortex-io, and develop has no fix for it yet.

The race is in the test. When the task is aborted before its first poll, the unresumed future drops its captured values in capture order, so done_send is dropped before recv. That wakes block_on(done_recv), and send.send(()) can then run while recv is still alive. Capturing the two as one tuple fixes the order, because a tuple's fields drop first to last:
```
let held = (recv, done_send);
let fut = async move {
    let (recv, _done) = held;
    let _ = recv.await;
    c.fetch_add(1, Ordering::SeqCst);
};
```
Both before and after the first poll, recv is then dropped before done_send.

I'm not widening this PR with that change. I'll re-run the failed jobs once when the workflow run finishes.

Generated by Claude Code

joseph-isaacs and others added 7 commits October 19, 2026 09:43

joseph-isaacs added changelog/performance A performance improvement action/bench-random-access Run only the random-access benchmark on this PR labels Oct 19, 2026 — with Claude

github-actions Bot removed the action/bench-random-access Run only the random-access benchmark on this PR label Oct 19, 2026

joseph-isaacs mentioned this pull request Oct 19, 2026

Plan V2 random access from known row selections #10625

Merged

joseph-isaacs added the action/bench-random-access Run only the random-access benchmark on this PR label Oct 19, 2026 — with Claude

github-actions Bot removed the action/bench-random-access Run only the random-access benchmark on this PR label Oct 19, 2026

claude added 2 commits October 19, 2026 23:53

joseph-isaacs changed the title ~~Plan each V2 work chunk only over its selected span~~ Plan V2 random access by the segments a selection reads Oct 20, 2026

joseph-isaacs added the action/bench-random-access Run only the random-access benchmark on this PR label Oct 20, 2026 — with Claude

github-actions Bot removed the action/bench-random-access Run only the random-access benchmark on this PR label Oct 20, 2026

claude added 3 commits October 20, 2026 07:52

joseph-isaacs mentioned this pull request Oct 20, 2026

Stream selected V2 chunk planners from one file planner #10628

Closed

Don't link the private file planner from into_stream's docs

2d5cd2b

Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rtiSjzL4EauVCM7NS84hj

joseph-isaacs closed this Oct 20, 2026

Navigation Menu

Repository navigation

Plan V2 random access by the segments a selection reads - #10621

Plan V2 random access by the segments a selection reads#10621
joseph-isaacs wants to merge 15 commits into
developfrom
ji/charming-cerf-9hlmns

joseph-isaacs commented Oct 19, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

joseph-isaacs commented Oct 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

joseph-isaacs commented Oct 20, 2026 •

edited

Loading

Uh oh!

joseph-isaacs commented Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joseph-isaacs commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

CI random-access results

Uh oh!

codspeed Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 1 benchmark

Performance Changes

Footnotes

Uh oh!

joseph-isaacs commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Random Access 📖

Uh oh!

joseph-isaacs commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joseph-isaacs commented Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joseph-isaacs commented Oct 19, 2026 •

edited

Loading

codspeed Bot commented Oct 19, 2026 •

edited

Loading

joseph-isaacs commented Oct 19, 2026 •

edited

Loading

github-actions Bot commented Oct 19, 2026 •

edited

Loading

github-actions Bot commented Oct 19, 2026 •

edited

Loading

joseph-isaacs commented Oct 20, 2026 •

edited

Loading