Add selection planners and ListLayout scans to V2

[joseph-isaacs]

Adds composable selection planning and ListLayout execution to the explicit V2 scan API. A selection planner passes its surviving row scope and mask to the projection planner. Lists resolve selected offsets and validity before scanning exact element ranges, preserving nullable and nested-list behavior.

Based on merged #10540 (d3bac42ef). This keeps the base's explicit API, ScanAdaptor, aligned row partitions, scheduler, and one output per work chunk. The follow-up is 31 files, +2,566/−141, including tests and benchmark tooling, versus the original 34 files, +4,593/−259. Original V2 remains at ad83fcbfc on ji/scan-v2-regression-ci.

The implementation includes these correctness fixes and measured changes:

• Preserve row counts for nullable fieldless structs; validate layout support while lowering reached plans.
• Size sparse identity work within the existing 8 MiB serialized-byte estimate, capped by useful worker/selected-row work; narrow compact selections and skip unselected chunks and gaps.
• Reuse sorted segment locations and validated chunk boundaries; preserve duplicate-location handling and indexed fallback.
• Store short planner priority paths inline and cache exact statistics for generated list offsets.
• Allow two grouped list ranges to overlap only when whole fixed-width chunks fit an 8 MiB work estimate; one-core, unknown-width, and oversized cases stay sequential.

The original stored-layout splitter, alternate executor, persistent preparation cache, and 32 MiB list-work admission budget remain omitted. The new estimates are not hard heap limits, and overlapping two ranges does not provide independent CPU scheduling. Signed-off commits document the deviations from the base.

Draft: the all-cases V1 performance goal remains unmet.

The completed combined CI comparison tests runtime 8442d4695 (tree-identical to local 95fda7fc8) against the previous PR head 5231b7091. The benchmark wrapper is 991c0dd33. Its artifact was checksum-verified.

• Standard: 0/36 cases regress more than 3% versus the previous PR head; maximum +2.32%.
• Full Arrow: 0/48 effective cases regress more than 3%, including four longer reruns; maximum +1.48%.
• Nine standard cases still regress more than 3% versus V1. The prior offset-copy-only 1 MiB list reopen flag does not reproduce in the combined candidate: its longer rerun improves 1.7% versus the previous PR head.

CI case	Cores	V1 ms	Previous V2 ms	Current V2 ms	Change vs previous V2
Clustered feature vectors, cached	30	0.2632	0.2732	0.2261	−17.2%
Clustered 1 MiB lists, cached	30	0.2347	0.3064	0.2433	−20.6%
Clustered 1 MiB lists, reopen	30	0.3469	0.4166	0.3548	−14.8%
Clustered 64 KiB lists, cached	30	0.2022	0.2470	0.2099	−15.0%
Six-row taxi lookup, cached	30	0.4178	0.5801	0.5673	−2.2%
Clustered taxi, cached	1	3.4058	3.9054	3.9196	+0.4%

Standard CI uses its existing generators, canonical-array output, cached/reopen modes, six-row taxi lookup, and two alternating five-second rounds with both parent/current V1 controls. Arrow comparisons use 100 selected rows, warm local disk, full Arrow decode/drop, three seeds, two alternating 64-query rounds, and four 128-query rounds for flagged cases; warmups are discarded. CI uses native ARM compiler settings. Local timing values use different files/compiler settings and are not combined with this table. Workflow success means the experiment completed, not that V1 was beaten everywhere.

Validation: 630 layout unit tests, 73 scan unit tests, and 72 real-file V1 full-scan-plus-take comparisons pass. Strict all-target/all-feature Clippy for the affected layout/scan crates, pinned formatting, whitespace, release benchmark builds, Ruff, and strict workflow YAML checks pass. Tests cover empty/null/nested lists, malformed offsets, FIFO/LIFO completion, shared segments, chunk-boundary validation, and priority ordering. Optional all-feature benchmark Clippy previously required an unavailable protoc; no local workspace-wide checks or doctests ran.

Two isolated list experiments remain outside this PR's validated runtime. Their full dedicated CI comparisons are complete, and all downloaded artifacts were SHA256-verified:

Experiment	CI	Arrow cases >3% slower than parent	Standard cases >3% slower than parent	Standard cases >3% slower than V1
List masks (9c82c79cd / published e3d88b5ab)	38985771977	0/48	0/36	10/36
Null-row payload (02f33ea52 / published 69448718c)	38985772941	1/48	0/36	12/36

Arrow counts replace matching initial measurements with longer reruns (four mask reruns, ten null-payload reruns). Maximum parent slowdowns are +1.63%/+1.31% for mask Arrow/standard, and +11.18%/+1.20% for null-payload Arrow/standard. The remaining null-payload flag is one-core cached clustered 64 KiB lists, 0.366907 -> 0.407918 ms. A focused eight-case native comparison on ji/scan-v2-list-control-ci compares the exact mask/null-payload runtime checkpoints, both V1 controls, three seeds, four 4096-query rounds with 512 warmups, and all cached/reopen, one/30-core, 64 KiB/1 MiB clustered controls. Its runtime excludes the early-admission experiment.

Both original jobs also completed 128 large-fixture correctness commands and 96 timing cases. Their initial effective results each had three >3% parent flags. The completed long singleton confirmation compares all three exact runtime checkpoints in one job, with four 4096-query rounds per seed and all V1 controls. None of those six earlier flags reproduces across the five distinct affected cases. The previously flagged skewed primitive singleton changes by -0.07% for masks and +0.09% for null-payload retention. All eight generated fixture hashes match the earlier runs. This confirmation does not cover the separate original-list flag above.

Native large singleton, 30 cores, cached	Before masks ms	Masks ms	Null-payload ms	V1 ms
Skewed nested, 64 KiB chunks	2.0326	0.7692	0.6201	0.5571
Skewed nested, 1 MiB chunks	0.7929	0.4468	0.3476	0.3219

These gains hold, but the final times remain 11.3% and 8.0% behind V1. Null-payload retention only applies to dense fixed-width spans in fully selected local scopes under an 8 MiB decoded-span estimate. The candidate passes 650 layout tests, strict Clippy, 200 real-file comparisons, and five malformed-selector checks. Original large-run correctness filenames collided for empty/100-row uniform selections: all 128 commands passed but only 112 result JSONs survived. The long confirmation fixes the filenames and preserves all 128 results.

Peak memory is not uniformly improved. Local null-payload allocation checks show matching request counts/bytes and median retained bytes across 32 cases; the large nested 1 MiB singleton drops 13.1% in median peak allocation, while some grouped large cases rise 4-5%. Longer original-list checks leave a 64 KiB cached seed42 median-peak increase of 23.4% (p95 +1.55%, observed max +0.33%). That non-null-list control does not take the new payload path, so attribution remains open. Independently scheduled large-row CPU work and the all-cases V1 goal remain outstanding.

[codspeed]

Merging this PR will improve performance by 25.45%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 15 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_scan_filter[int_str_float]
• non_nullable[2]
• test_scan_filter[int_str]
• random_i128[0.01]
• test_scan_filter[int_float]
• decompress[u16, (4000, 256)]
• decompress[u32, (1000, 256)]
• test_scan[int_str_float]
• test_scan[int_str]
• take_varbin
• …and 5 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

• test_repeated_scan_scalar_at[int_float]
• test_scan[int_float]
• test_scan_read_all_to_arrow_array[int_float]
• test_scan_scalar_at[int_str]
• test_repeated_scan[int_float]
• test_scan_filter[int_str_float]
• test_repeated_scan_scalar_at[int]
• test_repeated_scan_scalar_at[int_str_float]
• test_scan_scalar_at[int_str_float]
• test_scan_filter[int_str]
• …and 30 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 12 improved benchmarks
✅ 2384 untouched benchmarks
⏩ 12 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	take_map[(0.1, 1.0)]	352.2 µs	221.2 µs	+59.22%
⚡	WallTime	filter_neon[8]	1,382 ns	909 ns	+52.04%
⚡	WallTime	threshold_neon[8]	1.9 µs	1.4 µs	+41.78%
⚡	WallTime	filter_neon[64]	2.2 µs	1.7 µs	+25.72%
⚡ ℹ️	Simulation	test_repeated_scan_scalar_at[int_float]	490.9 µs	392.2 µs	+25.17%
⚡	Simulation	take_map[(0.1, 0.5)]	182.6 µs	146.4 µs	+24.72%
⚡	WallTime	bitpack_blocked_compress_avx512	5.7 µs	4.7 µs	+20.64%
⚡	WallTime	threshold_neon[64]	3 µs	2.5 µs	+20.31%
⚡	WallTime	filter_neon[80]	2.9 µs	2.6 µs	+12.98%
⚡	Simulation	take_map[(0.1, 0.05)]	87 µs	77.4 µs	+12.42%
⚡	WallTime	threshold_neon[80]	3.7 µs	3.3 µs	+10.9%
⚡	WallTime	words_gather_dispatch_neon[65536]	2.3 µs	2 µs	+10.35%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/decompression-v2-scan (8442d46) with develop (27c6b0d)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩