feat: switch vortex compress to use v2 scan api by joseph-isaacs · Pull Request #10540 · vortex-data/vortex

joseph-isaacs · 2026-10-16T11:17:23Z

Summary

Benchmark-only PR, stacked on #10530 (planner-driven V2 executor). It makes compress-bench decode Vortex on the V2 scan executor, so the PR compression benchmark compares V2 decompression with develop's V1 baseline on CI's runners. Do not merge: if this landed, develop's stored Vortex decompress numbers would switch to V2.

The only new commit on top of #10530 is 4632a21.

Describe how to land the vortex-scan planning driver running develop's pipeline::Scan decode path, with IO through the planning protocol, as a small PR stack extracted from 42b29cc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com>

…e fork Port the planning driver (`Run`/`Driver`) into vortex-scan and a projection-only V2 executor into vortex-layout that runs develop's pipeline `Scan` through the planning protocol, with every segment read issued as an `IoRequest` served from the file's `SegmentSource`. `ScanBuilder`'s lazy prepare step is the single place a streaming scan picks its executor: `v2::select` returns V1 unless `VORTEX_SCAN_V2=1`, the builder carries a `ScanFile`, and the scan has no filter, no limit and no list layout. `VortexFile::scan()` attaches the `ScanFile`; no caller changes. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

…ndoff doc Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

Develop's lower keeps Zoned and LegacyStats nodes in the plan, so the boundary walk stopped at them and every default-written file ran as a single split. Follow their data child. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

Each split had its own ValueCache, so a dictionary shared by many chunks was decoded once per split (HashTags 2.2x, CMSprovider 1.7x slower than V1). Share one cache per scan, as V1's reader does. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

Reuse V1 split policies, avoid redundant identity-projection planning, and release decoded dictionary values after their consuming splits finish. Reduce pipeline allocation, plan retention, and ready-I/O scheduling overhead. Validated with repeated V1/V2/Parquet decompression comparisons, output parity across 14 datasets, and 175 focused pipeline and scan tests. Some workloads still retain modest RSS and parallel duplicate-read overhead versus V1. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Measure preparation, compilation and in-memory segment delivery for narrow and wide pipeline scans in the existing CodSpeed layout shard. Document the stable identity of scoped dictionary-cache keys and use an explicit Arc clone in the V2 test so the scan code passes the CI Clippy configurations. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Keep private implementation names as code and remove the redundant v2 link target so the planner documentation builds with warnings denied. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Adds ScanBuilder::with_scan_v2 to choose the planner-driven executor for one scan, overriding VORTEX_SCAN_V2, and has compress-bench's Vortex decompression call it, so CI measures V2 decompression against develop's V1 baseline without relying on an environment variable. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtNyxw9KerLDyt5eXnpnRE Signed-off-by: Claude <noreply@anthropic.com>

github-actions · 2026-10-16T11:22:09Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`d445ce4`	compress-bench	1	Explore Profiling Data

Previous Runs (4)

Status	Commit	Job	Attempt	Link
🟢 Done	`26c83e0`	compress-bench	1	Explore Profiling Data
🟢 Done	`d57516d`	compress-bench	1	Explore Profiling Data
🟢 Done	`9859da6`	compress-bench	1	Explore Profiling Data
🟢 Done	`4632a21`	compress-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

codspeed · 2026-10-16T11:24:02Z

Merging this PR will improve performance by 19.43%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

16 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

test_repeated_scan_filter[int]

test_scan_filter[int_str]

test_scan_filter[int_float]

test_repeated_scan[int_str]

test_scan_filter[int_str_float]

zstd_compress_string

test_scan[int_str_float]

pcodec_compress_f64

multipolygon_mixed_validity

decompress[u16, (4000, 256)]

…and 6 more

⚡ 29 improved benchmarks
✅ 2358 untouched benchmarks
🆕 9 new benchmarks
⏩ 12 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	WallTime	`threshold_neon[8]`	2.4 µs	1.3 µs	+80.41%
⚡	WallTime	`threshold_neon[80]`	4.9 µs	3.3 µs	+47.36%
⚡	WallTime	`filter_neon[8]`	1,339 ns	918 ns	+45.86%
⚡	WallTime	`threshold_neon[64]`	3.5 µs	2.5 µs	+40.64%
⚡	WallTime	`filter_neon[64]`	2.3 µs	1.8 µs	+30%
⚡	Simulation	`isolated_runs[u64, (161, 944)]`	168.2 µs	137.7 µs	+22.12%
⚡	Simulation	`embedding_code_runs_expanded_baseline[4096]`	1.9 ms	1.6 ms	+20.65%
⚡	Simulation	`embedding_code_runs_expanded_baseline[2048]`	1,016.3 µs	850.6 µs	+19.48%
⚡	Simulation	`embedding_code_runs_expanded_baseline[1024]`	548.4 µs	465.1 µs	+17.89%
⚡	Simulation	`isolated_runs[u64, (256, 896)]`	219.7 µs	189.2 µs	+16.11%
⚡	Simulation	`embedding_runs_expanded_baseline[1024]`	609 µs	525.7 µs	+15.84%
⚡	Simulation	`embedding_runs_expanded_baseline[1536]`	913.6 µs	789.1 µs	+15.77%
⚡	Simulation	`fragmented_runs_expanded_baseline[1024]`	623.2 µs	540.2 µs	+15.37%
⚡	Simulation	`embedding_code_runs_expanded_baseline[512]`	319.9 µs	278.9 µs	+14.72%
⚡	Simulation	`isolated_runs[u64, (161, 0)]`	180.4 µs	157.2 µs	+14.72%
⚡	Simulation	`isolated_runs[u64, (256, 0)]`	192.8 µs	169.3 µs	+13.86%
⚡	Simulation	`embedding_runs_expanded_baseline[768]`	516.1 µs	454.6 µs	+13.51%
⚡	Simulation	`isolated_runs[u64, (256, 1020)]`	196.6 µs	173.5 µs	+13.34%
⚡	Simulation	`isolated_runs[u32, (128, 0)]`	111.6 µs	98.6 µs	+13.17%
⚡	Simulation	`isolated_runs[u32, (160, 0)]`	113.8 µs	101 µs	+12.67%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/upbeat-volta-adp0xd (a9e2601) with develop (6289e58)²}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on develop (bc9e0e6) during the generation of this report, so 6289e58 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

github-actions · 2026-10-16T11:31:58Z

Benchmarks: Compression 📖

Commits: PR d445ce44 vs base 18aebeaa

vortex / vortex-file-compressed / ns (0.908x ➖, 7↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	778948894 / 787435434 / -1.1%
compress time/Bimbo	3630225497 / 3692508802 / -1.7%
compress time/CMSprovider	2532706474 / 2545029989 / -0.5%
compress time/Euro2016	269722171 / 267512769 / +0.8%
compress time/Food	320818283 / 317789581 / +1.0%
compress time/HashTags	465918037 / 463547621 / +0.5%
compress time/TPC-H l_comment canonical	612338835 / 600311849 / +2.0%
compress time/TPC-H l_comment chunked	616631069 / 603742733 / +2.1%
compress time/taxi	388534529 / 387878829 / +0.2%
compress time/wide table cols=100 chunks=1 rows=1000	7165533 / 7082137 / +1.2%
compress time/wide table cols=100 chunks=50 rows=1000	7129732 / 7133163 / -0.0%
compress time/wide table cols=1000 chunks=1 rows=1000	70528868 / 71662532 / -1.6%
compress time/wide table cols=1000 chunks=50 rows=1000	71016184 / 70054687 / +1.4%
compress time/wide table cols=10000 chunks=1 rows=1000	758144910 / 769400585 / -1.5%
compress time/wide table cols=10000 chunks=50 rows=1000	775353107 / 778349156 / -0.4%
decompress time/Arade	7848233 / 9053434 / -13.3% 🟢
decompress time/Bimbo	26617236 / 37381017 / -28.8% 🟢
decompress time/CMSprovider	22407836 / 27149413 / -17.5% 🟢
decompress time/Euro2016	9576915 / 9746141 / -1.7%
decompress time/Food	1734807 / 2908948 / -40.4% 🟢
decompress time/HashTags	26297732 / 43177640 / -39.1% 🟢
decompress time/TPC-H l_comment canonical	8154354 / 11031972 / -26.1% 🟢
decompress time/TPC-H l_comment chunked	8008373 / 10909365 / -26.6% 🟢
decompress time/taxi	4746637 / 5181307 / -8.4%
decompress time/wide table cols=100 chunks=1 rows=1000	1244041 / 1345212 / -7.5%
decompress time/wide table cols=100 chunks=50 rows=1000	1265861 / 1348698 / -6.1%
decompress time/wide table cols=1000 chunks=1 rows=1000	11345594 / 12460271 / -8.9%
decompress time/wide table cols=1000 chunks=50 rows=1000	11345720 / 12409842 / -8.6%
decompress time/wide table cols=10000 chunks=1 rows=1000	121162272 / 133669276 / -9.4%
decompress time/wide table cols=10000 chunks=50 rows=1000	121970146 / 133497275 / -8.6%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164130260 / 164125572 / +0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (0.935x ➖, 10↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	0.34777227 / 0.350073615 / -0.7%
vortex:parquet-zstd ratio compress time/Bimbo	0.308709429 / 0.311439563 / -0.9%
vortex:parquet-zstd ratio compress time/CMSprovider	0.396090646 / 0.398612965 / -0.6%
vortex:parquet-zstd ratio compress time/Euro2016	0.250477784 / 0.2472584 / +1.3%
vortex:parquet-zstd ratio compress time/Food	0.433823731 / 0.429202695 / +1.1%
vortex:parquet-zstd ratio compress time/HashTags	0.254329318 / 0.25390901 / +0.2%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	0.233695914 / 0.228396024 / +2.3%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	0.23428701 / 0.229279183 / +2.2%
vortex:parquet-zstd ratio compress time/taxi	0.356411818 / 0.356144023 / +0.1%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	1.81455236 / 1.79102265 / +1.3%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	1.78770091 / 1.76216895 / +1.4%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	1.57765045 / 1.61095158 / -2.1%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	1.58992271 / 1.58194927 / +0.5%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	1.64366022 / 1.67738835 / -2.0%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	1.66511531 / 1.72058246 / -3.2%
vortex:parquet-zstd ratio decompress time/Arade	0.0141115717 / 0.0162564564 / -13.2% 🟢
vortex:parquet-zstd ratio decompress time/Bimbo	0.0166224276 / 0.0233227159 / -28.7% 🟢
vortex:parquet-zstd ratio decompress time/CMSprovider	0.0163578321 / 0.0199083036 / -17.8% 🟢
vortex:parquet-zstd ratio decompress time/Euro2016	0.0287615916 / 0.0293179881 / -1.9%
vortex:parquet-zstd ratio decompress time/Food	0.0101734745 / 0.017115439 / -40.6% 🟢
vortex:parquet-zstd ratio decompress time/HashTags	0.0508191366 / 0.0835751162 / -39.2% 🟢
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.0163967845 / 0.0221928205 / -26.1% 🟢
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.0161160439 / 0.0219569447 / -26.6% 🟢
vortex:parquet-zstd ratio decompress time/taxi	0.0217544937 / 0.0238150044 / -8.7%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.602812191 / 0.653945343 / -7.8%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.606115931 / 0.637994781 / -5.0%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.482722585 / 0.533848905 / -9.6%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.477397233 / 0.539979908 / -11.6% 🟢
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.492884729 / 0.552586204 / -10.8% 🟢
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.482615187 / 0.553478274 / -12.8% 🟢
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33461969 / 1.33458157 / +0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (1.004x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	2239824623 / 2249342423 / -0.4%
parquet_rs-zstd compress time/Bimbo	11759360604 / 11856261171 / -0.8%
parquet_rs-zstd compress time/CMSprovider	6394259752 / 6384714530 / +0.1%
parquet_rs-zstd compress time/Euro2016	1076830712 / 1081915796 / -0.5%
parquet_rs-zstd compress time/Food	739512987 / 740418419 / -0.1%
parquet_rs-zstd compress time/HashTags	1831947809 / 1825644635 / +0.3%
parquet_rs-zstd compress time/TPC-H l_comment canonical	2620237657 / 2628381341 / -0.3%
parquet_rs-zstd compress time/TPC-H l_comment chunked	2631947329 / 2633220887 / -0.0%
parquet_rs-zstd compress time/taxi	1090128074 / 1089106665 / +0.1%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	3948926 / 3954242 / -0.1%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	3988213 / 4047945 / -1.5%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	44705003 / 44484597 / +0.5%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	44666438 / 44283776 / +0.9%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	461254037 / 458689596 / +0.6%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	465645293 / 452375388 / +2.9%
parquet_rs-zstd decompress time/Arade	556155841 / 556913129 / -0.1%
parquet_rs-zstd decompress time/Bimbo	1601284524 / 1602772898 / -0.1%
parquet_rs-zstd decompress time/CMSprovider	1369853647 / 1363723072 / +0.4%
parquet_rs-zstd decompress time/Euro2016	332975836 / 332428711 / +0.2%
parquet_rs-zstd decompress time/Food	170522569 / 169960467 / +0.3%
parquet_rs-zstd decompress time/HashTags	517476954 / 516632725 / +0.2%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	497314215 / 497096436 / +0.0%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	496919285 / 496852597 / +0.0%
parquet_rs-zstd decompress time/taxi	218191104 / 217564814 / +0.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2063729 / 2057071 / +0.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2088480 / 2113964 / -1.2%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	23503342 / 23340445 / +0.7%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23765785 / 22982044 / +3.4%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	245822735 / 241897599 / +1.6%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	252727534 / 241196956 / +4.8%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (0.992x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	99194057 / 100526703 / -1.3%
arrow-ipc compress time/Bimbo	625346877 / 632004766 / -1.1%
arrow-ipc compress time/CMSprovider	383772921 / 388818942 / -1.3%
arrow-ipc compress time/Euro2016	38670013 / 38778483 / -0.3%
arrow-ipc compress time/Food	33746007 / 34273067 / -1.5%
arrow-ipc compress time/HashTags	78662680 / 79055014 / -0.5%
arrow-ipc compress time/TPC-H l_comment canonical	986576509 / 952545520 / +3.6%
arrow-ipc compress time/TPC-H l_comment chunked	957804684 / 958239676 / -0.0%
arrow-ipc compress time/taxi	74725978 / 75598596 / -1.2%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	209127 / 213106 / -1.9%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	203394 / 203702 / -0.2%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2234287 / 2273386 / -1.7%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2226290 / 2276210 / -2.2%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25617318 / 25869661 / -1.0%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25445863 / 25846631 / -1.6%
arrow-ipc decompress time/Arade	140136628 / 142125027 / -1.4%
arrow-ipc decompress time/Bimbo	415657571 / 432178101 / -3.8%
arrow-ipc decompress time/CMSprovider	473530074 / 476421809 / -0.6%
arrow-ipc decompress time/Euro2016	63810059 / 64155739 / -0.5%
arrow-ipc decompress time/Food	48763217 / 49190058 / -0.9%
arrow-ipc decompress time/HashTags	122417160 / 122222178 / +0.2%
arrow-ipc decompress time/TPC-H l_comment canonical	634438011 / 641278112 / -1.1%
arrow-ipc decompress time/TPC-H l_comment chunked	635668179 / 641921911 / -1.0%
arrow-ipc decompress time/taxi	45234127 / 45692253 / -1.0%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	318517 / 320811 / -0.7%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	322949 / 320196 / +0.9%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3688961 / 3558015 / +3.7%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3646068 / 3676421 / -0.8%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	35734567 / 36985971 / -3.4%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36481606 / 36763188 / -0.8%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtNyxw9KerLDyt5eXnpnRE Signed-off-by: Claude <noreply@anthropic.com>

Use an explicit V2 ScanBuilder import for benchmark decompression and restore the V1 builder. Remove runtime executor selection and fallback, restrict planning and graph helper visibility, and cover the direct stream and Arrow paths. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Start from 740e1ef and retain its explicit V2 ScanBuilder, fixed row chunks sized by serialized bytes, ScanAdaptor, and one output per work chunk. SelectionPlanner hands its selected rows to the existing ProjectionPlanner. Add ListPack compilation that reads selected boundaries/validity before scanning element ranges. This replaces the follow-up at ad83fcb; that history is preserved by ji/scan-v2-before-pr10540 locally and by the existing ji/scan-v2-regression-ci remote branch. The shared ancestor is 7b15cfc. Reconstructing the compatible additions avoids resurrecting files removed or replaced by the new base. Deviations resolved in favor of PR #10540 before benchmarking: - Keep its explicit API; drop the V1-builder environment gate. - Keep its 8,192-row multiples, at most 65,536 rows, and 8 MiB serialized target; drop the stored-chunk/list-element sparse splitter and separate 32 MiB list-work admission cap. - Keep its ScanAdaptor and aggregate output; drop the alternate ScanGraph, direct point morsel, and per-split streaming channels. - Keep its I/O implementation; drop sparse range lookup, eager request registration/coalescing heuristics, and ready-completion shortcuts. - Keep its file and plan lifetime; drop persistent preparation, support caches, and dictionary retention across file scans. - Keep its scheduler, ValueCache, lowering, and pipeline activation; drop inline ready queues, deferred concat producers, and capacity knobs. - Keep its sharing traversal; drop Reach::Selected/Reach::Ranges pruning. List elements still use exact offset-derived ranges for correctness. - Drop the grouped-list two-range overlap heuristic pending measurements. - Keep the base's documentation deletion and compress-bench V2 selection. The initial diff is 14 files, +1493/-26 including tests and benchmark infrastructure, down from 34 files, +4593/-259. Removed diagnostic modes that depended on the old builder's mapping API. No removed optimization has been restored without post-rebase measurements. The dedicated CI workflow on ji/scan-v2-pr10540 compares the original ad83fcb V2, this V2, and current V1 using identical data and compiler flags. It retains all 48 cases, full Arrow decode/drop, three seeds, alternating order, and longer reruns. Results are pending at this commit. Validation: release_debug selection/rewrite examples built; 573 layout and 65 scan unit tests passed; 72 real-file V1 full-scan-plus-take checks passed across all six benchmark datasets, one and 30 cores, empty, singleton, clustered, uniform, larger, and full selections. Pinned format, Ruff, strict workflow yamllint, and diff whitespace checks passed. No workspace-wide checks, Clippy, or doctests were run. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Keep the benchmark worker-thread timing from develop and use the explicit V2 scan builder in the decompression task. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Ports #10540's planner-driven executor from `vortex_io::request` onto `vortex_io::segments`: - vortex-scan: `Run`/`Driver` submit each root's requests to its `SegmentIoSession` and hold the handles. Each item's fetches finish through its own `FuturesUnordered`, so a `poll_ready` report yields exactly the finished fetches without rescanning outstanding ones. `Run::poll_io` replaces `Run::complete`; sessions are dropped instead of cleared. The scripted harness is a recording `SegmentIoSession` and the driver tests are ported. - vortex-layout: `ScanAdaptor` fetches each segment once by `IoKey::segment` while any scan read of it is outstanding, and hands the result to every waiting read. `ScanFile` carries the file's `RegisteredSource`; each split reads through a new session of its service. The old `SegmentScanIo` adapter is removed. - vortex-file: `VortexFile::scan_file` registers the file's segment source with the session's `FileSegmentIo`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Deviation from PR #10540 retained after the rebase: PackSource already knows the number of consumed rows, but assemble derived it from the first field. A nullable struct with no fields has only a validity inlet, leaving no field from which to recover its length. Pass the consumed length through the existing private helper. This does not restore the old point-projection executor or its assembly API. The regression test failed before the fix: 0 rows instead of 20 for all rows, and 0 instead of 10 for every-other selection. Both pass afterward. Validation: 578 vortex-layout and 65 vortex-scan unit tests passed in the current worktree; no workspace-wide checks or Clippy were run. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Measured deviation from PR #10540: remove the recursive supports_layout preflight. The explicit V2 builder has no fallback to select, and lower already reports unsupported layouts. Keep unrelated layout children lazy instead of deserializing every child before a sparse scan can start. Unknown roots and reached unknown chunks still fail. A chunk outside the requested row range can now remain unsupported and unread; tests cover both outcomes. No persistent file/support cache is restored. After rebasing, one-core clustered feature-vector reopen scans made about 26,100 allocations vs 1,400 before the rebase. In three-seed alternating 64-query runs, removing the walk improved 3.262 ms to 1.252 ms. Thirty-core 64 KiB list clustered reopen improved 1.192 ms to 0.428 ms (previous V2: 0.420 ms). Taxi remained about 31 ms vs 14 ms because fixed work chunks request 133 MB vs 48 MB of unique segment data, a separate issue. The initial base-preferred dedicated CI run 38931231485 completed all 48 cases and 38 longer confirmations; 35 confirmed cases exceed +3% versus ad83fcb. Passing workflow status is not a performance-regression gate. Local raw results and deviations are under random-access-v2/rebase-10540 on /mnt/vortex-ssd/vortex-scanning-v2. This change's focused comparison is lazy-validation-comparison; final full-matrix results are pending. Validation: 578 vortex-layout and 65 vortex-scan unit tests passed, including unsupported-layout cases and the prior nullable-struct fix. Release_debug example built; diff whitespace check passed. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Measured deviation from PR #10540: its 65,536-row ceiling repeatedly reads large stored segments when sparse selections span adjacent work chunks. Preserve fixed aligned chunks, the 8 MiB serialized-byte target, and the base executor. For at most 1/16 selected rows, allow larger power-of-two multiples while targeting four work chunks per worker. Dense scans, non-identity projections, and wide-row sizing are unchanged. This replaces the old stored-layout/list-element splitter with about 20 lines of sizing logic; no additional execution or cache path is added. Three-seed alternating full Arrow decode/drop comparisons after rebasing: - Taxi, 1 core, uniform cached: 31.743 -> 13.950 ms; previous V2 13.598. - Taxi, 1 core, uniform reopen: 32.428 -> 14.737 ms; previous V2 14.433. - Nested structs, 1 core, uniform cached: 6.652 -> 4.324 ms; previous 4.538. Instrumented queries confirm the cause: taxi requests fall from 825 / 133 MB to 327 / 48 MB; nested structs fall from 96 / 71 MB to 48 / 36 MB. Taxi peak live allocation increases from about 18 MB to 40 MB, below the prior implementation's measured peak. The byte target is an estimate of serialized work, not a hard bound on live heap or decoded values. A simpler byte-only variant improved 30-core uniform reads but worsened 30-core clustered taxi by about 9% relative to this worker-limited variant; it was rejected. Keep the base's multi-core sizing in those cases. Other pre-rebase optimizations remain omitted, so this is not a no-regression claim. The final dedicated 48-case comparison is pending publication. Validation: 583 vortex-layout and 65 vortex-scan unit tests passed; 72 real-file full-scan-plus-take checks passed across six datasets, 1/30 cores, empty/singleton/clustered/uniform/larger/full selections. Built instrumented and uninstrumented examples; pinned formatting and diff whitespace checks passed. No workspace-wide checks, Clippy, or doctests. Artifacts: /mnt/vortex-ssd/vortex-scanning-v2/random-access-v2/rebase-10540/ (sparse-budget-comparison, byte-budget-only-comparison, final-diagnostics, final-correctness). Original work remains on ji/scan-v2-before-pr10540 locally and ji/scan-v2-regression-ci remotely. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Start from 740e1ef and retain its explicit V2 ScanBuilder, fixed row chunks sized by serialized bytes, ScanAdaptor, and one output per work chunk. SelectionPlanner hands its selected rows to the existing ProjectionPlanner. Add ListPack compilation that reads selected boundaries/validity before scanning element ranges. This replaces the follow-up at ad83fcb; that history is preserved by ji/scan-v2-before-pr10540 locally and by the existing ji/scan-v2-regression-ci remote branch. The shared ancestor is 7b15cfc. Reconstructing the compatible additions avoids resurrecting files removed or replaced by the new base. Deviations resolved in favor of PR #10540 before benchmarking: - Keep its explicit API; drop the V1-builder environment gate. - Keep its 8,192-row multiples, at most 65,536 rows, and 8 MiB serialized target; drop the stored-chunk/list-element sparse splitter and separate 32 MiB list-work admission cap. - Keep its ScanAdaptor and aggregate output; drop the alternate ScanGraph, direct point morsel, and per-split streaming channels. - Keep its I/O implementation; drop sparse range lookup, eager request registration/coalescing heuristics, and ready-completion shortcuts. - Keep its file and plan lifetime; drop persistent preparation, support caches, and dictionary retention across file scans. - Keep its scheduler, ValueCache, lowering, and pipeline activation; drop inline ready queues, deferred concat producers, and capacity knobs. - Keep its sharing traversal; drop Reach::Selected/Reach::Ranges pruning. List elements still use exact offset-derived ranges for correctness. - Drop the grouped-list two-range overlap heuristic pending measurements. - Keep the base's documentation deletion and compress-bench V2 selection. The initial diff is 14 files, +1493/-26 including tests and benchmark infrastructure, down from 34 files, +4593/-259. Removed diagnostic modes that depended on the old builder's mapping API. No removed optimization has been restored without post-rebase measurements. The dedicated CI workflow on ji/scan-v2-pr10540 compares the original ad83fcb V2, this V2, and current V1 using identical data and compiler flags. It retains all 48 cases, full Arrow decode/drop, three seeds, alternating order, and longer reruns. Results are pending at this commit. Validation: release_debug selection/rewrite examples built; 573 layout and 65 scan unit tests passed; 72 real-file V1 full-scan-plus-take checks passed across all six benchmark datasets, one and 30 cores, empty, singleton, clustered, uniform, larger, and full selections. Pinned format, Ruff, strict workflow yamllint, and diff whitespace checks passed. No workspace-wide checks, Clippy, or doctests were run. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Deviation from PR #10540 retained after the rebase: PackSource already knows the number of consumed rows, but assemble derived it from the first field. A nullable struct with no fields has only a validity inlet, leaving no field from which to recover its length. Pass the consumed length through the existing private helper. This does not restore the old point-projection executor or its assembly API. The regression test failed before the fix: 0 rows instead of 20 for all rows, and 0 instead of 10 for every-other selection. Both pass afterward. Validation: 578 vortex-layout and 65 vortex-scan unit tests passed in the current worktree; no workspace-wide checks or Clippy were run. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Measured deviation from PR #10540: remove the recursive supports_layout preflight. The explicit V2 builder has no fallback to select, and lower already reports unsupported layouts. Keep unrelated layout children lazy instead of deserializing every child before a sparse scan can start. Unknown roots and reached unknown chunks still fail. A chunk outside the requested row range can now remain unsupported and unread; tests cover both outcomes. No persistent file/support cache is restored. After rebasing, one-core clustered feature-vector reopen scans made about 26,100 allocations vs 1,400 before the rebase. In three-seed alternating 64-query runs, removing the walk improved 3.262 ms to 1.252 ms. Thirty-core 64 KiB list clustered reopen improved 1.192 ms to 0.428 ms (previous V2: 0.420 ms). Taxi remained about 31 ms vs 14 ms because fixed work chunks request 133 MB vs 48 MB of unique segment data, a separate issue. The initial base-preferred dedicated CI run 38931231485 completed all 48 cases and 38 longer confirmations; 35 confirmed cases exceed +3% versus ad83fcb. Passing workflow status is not a performance-regression gate. Local raw results and deviations are under random-access-v2/rebase-10540 on /mnt/vortex-ssd/vortex-scanning-v2. This change's focused comparison is lazy-validation-comparison; final full-matrix results are pending. Validation: 578 vortex-layout and 65 vortex-scan unit tests passed, including unsupported-layout cases and the prior nullable-struct fix. Release_debug example built; diff whitespace check passed. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Measured deviation from PR #10540: its 65,536-row ceiling repeatedly reads large stored segments when sparse selections span adjacent work chunks. Preserve fixed aligned chunks, the 8 MiB serialized-byte target, and the base executor. For at most 1/16 selected rows, allow larger power-of-two multiples while targeting four work chunks per worker. Dense scans, non-identity projections, and wide-row sizing are unchanged. This replaces the old stored-layout/list-element splitter with about 20 lines of sizing logic; no additional execution or cache path is added. Three-seed alternating full Arrow decode/drop comparisons after rebasing: - Taxi, 1 core, uniform cached: 31.743 -> 13.950 ms; previous V2 13.598. - Taxi, 1 core, uniform reopen: 32.428 -> 14.737 ms; previous V2 14.433. - Nested structs, 1 core, uniform cached: 6.652 -> 4.324 ms; previous 4.538. Instrumented queries confirm the cause: taxi requests fall from 825 / 133 MB to 327 / 48 MB; nested structs fall from 96 / 71 MB to 48 / 36 MB. Taxi peak live allocation increases from about 18 MB to 40 MB, below the prior implementation's measured peak. The byte target is an estimate of serialized work, not a hard bound on live heap or decoded values. A simpler byte-only variant improved 30-core uniform reads but worsened 30-core clustered taxi by about 9% relative to this worker-limited variant; it was rejected. Keep the base's multi-core sizing in those cases. Other pre-rebase optimizations remain omitted, so this is not a no-regression claim. The final dedicated 48-case comparison is pending publication. Validation: 583 vortex-layout and 65 vortex-scan unit tests passed; 72 real-file full-scan-plus-take checks passed across six datasets, 1/30 cores, empty/singleton/clustered/uniform/larger/full selections. Built instrumented and uninstrumented examples; pinned formatting and diff whitespace checks passed. No workspace-wide checks, Clippy, or doctests. Artifacts: /mnt/vortex-ssd/vortex-scanning-v2/random-access-v2/rebase-10540/ (sparse-budget-comparison, byte-budget-only-comparison, final-diagnostics, final-correctness). Original work remains on ji/scan-v2-before-pr10540 locally and ji/scan-v2-regression-ci remotely. Rebase update: PR #10540 merged as d3bac42. Replay on develop preserves every incoming base change, including the new BitPacked/FSST take paths and writer defaults. The merged base tree equals a9e2601, so the replay has no additional code conflicts. Earlier timing figures above describe the 740e1ef checkpoint; new-base results are pending. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

The final branch now extends develop at d3bac42 (#10540), preserving the base's merged BitPacked/FSST changes and writer defaults. Cancel older runs of this experiment branch when a newer head is published, so the benchmark runner validates the intended source instead of queuing behind a superseded checkpoint. Other branches use different concurrency groups. The 740e1ef base-preferred run 38931231485 is complete and archived. Run 38935258274 targets the previous 15f2fed checkpoint and can be superseded. Its results must not be described as the merged-base comparison. Validation: strict yamllint for the changed workflow and diff whitespace check passed. This commit changes no Rust behavior. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

CI run 38936290519 failed before timing: rewrite_list_layout still called WriteStrategyBuilder::with_list_layout(), removed by upstream #10563 and inherited through the merged #10540 base. Keep the base's edition-driven writer policy and adapt the benchmark helper instead of restoring the API. Enable the experimental list edition only for list/list-elements modes. Flat mode leaves it disabled, preserving the flat control. Drop the obsolete environment guard and explicit writer toggle. Scanner runtime is unchanged, so the completed local 48-case merged-comparison remains applicable. Validation: built both release_debug examples with --locked; rewrote all 1,000,000 rows in flat, list, list-elements/1 MiB and list-elements/64 KiB modes; all four full Arrow equality and layout-kind checks passed, as did four V2 selection comparisons against full V1 scans followed by take. Pinned formatting and diff whitespace checks passed. No new Rust-library checks were needed for this example-only API adaptation. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

#10540 landed on develop with the V2 executor on vortex_io::request. Keep this branch's port onto vortex_io::segments for the V2 executor files, and drop scan/v2/io.rs again. Signed-off-by: Claude <noreply@anthropic.com>

ListSource supplies disjoint offset-derived element ranges to Scan. The merged-base sharing walk took their bounding range and eagerly lowered unread chunks between those ranges. Walk the union of sorted disjoint ranges instead, merging adjacent ranges to preserve the dense-scan path. Keep the existing bounding walk for overlapping or unordered splits. This restores only the measured gap pruning benefit from the original pre-#10540 branch, without changing its scheduler, projection morsels, sharing implementation, or cache lifetimes. Selection planners still hand selected row scopes to projection planners. Two lazy-gap regression cases fail before this change and pass after it. Additional cases cover gaps, overlaps, unordered/empty ranges, FIFO/LIFO completion, and shared segments read once across active split counts. 601 vortex-layout and 69 vortex-scan unit tests pass, as do 72 real-file checks against V1 full scan plus take. Layout all-target/all-feature Clippy with -D warnings, pinned format, and diff whitespace checks pass. The uninstrumented local comparison against ad47e105d uses the same six stored datasets, 3 seeds, and 2 alternating rounds of 64 queries. In its 14 targeted cases, uniform 64 KiB lists improve 12.1/25.5 percent with one core (cached/reopen) and 20.3/33.0 percent with 30 cores. No case regresses more than 0.9 percent. Separate diagnostics reduce 30-core uniform-reopen allocations from 20,636 to 11,143 with requested bytes unchanged at 11.55 MB. These are local measurements, not the earlier native CI comparison. Results: /mnt/vortex-ssd/vortex-scanning-v2/random-access-v2/post-rebase/ gaps-comparison/results.json and gaps-diagnostics/summary.json. Full 48-case comparison is next. Five of these 14 cases remain over 3 percent slower than V1; this checkpoint does not establish the all-cases performance goal or fix repeated I/O across independent projection tasks. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

A projection morsel runs one physical split with a selection mask. The sharing walk still lowered every chunk in its bounding range, even though compilation later skipped chunks with no selected rows. Carry the mask through linear row mappings and skip such children before lowering them. Preserve full reads when a child changes row domains, including dictionary values and list offsets. Dense and multiple-split scans keep their existing sharing walk. Use cached indices or bitmap range counts without making a new sliced mask for each child. This is a measured restoration of selected reach pruning from the original pre-#10540 branch; it does not restore its splitter, executor, or persistent caches. SelectionPlanner still returns ProjectionPlanner with absolute row coordinates. The preceding published checkpoint is 26aa0be. Four masked-gap regression cases fail before this change. Eight cases now cover nested/nonzero ranges, empty masks, and cached/uncached indices, and all 609 layout unit tests pass. All 72 real-file full-scan-plus-take checks pass. Layout all-target/all-feature Clippy with -D warnings, pinned format, and whitespace checks pass. No wider workspace checks or doctests ran. In separate instrumented runs, uniform feature-vector allocation counts fall about 19 percent cached and 31 percent reopening. Requested bytes stay at 85.02 MB; peak memory varies, so no general peak-memory win is claimed. The 20-case uninstrumented comparison uses three seeds and two alternating rounds of 64 queries, full Arrow decode/drop, one and 30 pinned cores. Feature-vector uniform reopen improves 5.6 percent on one core and 11.2 percent on 30 cores against 26aa0be. No tested case regresses more than 1.8 percent. Five cases remain over 3 percent slower than V1, including clustered lists; the all-random-access performance goal remains open. Artifacts: /mnt/vortex-ssd/vortex-scanning-v2/random-access-v2/post-rebase/ mask-comparison/results.json and mask-diagnostics/summary.json. The standard random-access harness also needs explicit V2 coverage; that benchmark-only switch is a separate follow-up, not part of this checkpoint. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Copy the already validated chunk start offsets into the existing Concat Arc instead of repeatedly asking the layout for each child's row count. Keep V1's offset storage and the base planner/scheduler unchanged. This is a measured deviation from PR #10540, isolated for comparison with 5231b70. Four regression cases cover regular, empty, absent, and zero-row chunks; three fail before the change. No child layouts are materialized to lower the boundaries. Production change is +7/-10 lines in two files. Local four-round, five-second standard comparisons show clustered feature vectors improving 8.0% cached / 3.5% reopen on one core and 17.1% / 4.9% on 30 cores. The initial taxi cached +6.4% reduced to +0.8% in the longer run. Nested-struct reopen is still +4.9%; identical-binary controls vary up to 7.8%, so that result is unresolved rather than dismissed as noise. Sixteen full-Arrow list cases initially show at most +2.01% slowdown; longer list confirmation and isolated parent CI remain pending. Repeated memory checks show no increase in taxi maximum live allocation, and median requested read counts/bytes are unchanged. No all-cases V1 win is claimed. Validation: 618 layout tests, all-target/all-feature layout Clippy with -D warnings, pinned formatting, whitespace, release benchmark builds, and 72 real-file V1 full-scan-plus-take correctness comparisons passed. No workspace-wide checks or doctests were run. Keep separate from the draft PR until the exact-parent CI comparison is assessed. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

claude and others added 9 commits October 15, 2026 18:12

chore(scan): clippy fixes and record the single-fork design in the ha…

7c1cb9b

…ndoff doc Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

docs(scan): fix planner documentation links

7b15cfc

Keep private implementation names as code and remove the redundant v2 link target so the planner documentation builds with warnings denied. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

joseph-isaacs added the action/bench-compress Run only the compression benchmark on this PR label Oct 16, 2026 — with Claude