bench: V2 scan decompression on the segment IO protocol by joseph-isaacs · Pull Request #10569 · vortex-data/vortex

joseph-isaacs · 2026-10-16T16:47:44Z

Summary

Benchmark-only, do not merge. Combines #10540 (compress-bench decompresses Vortex on the planner-driven V2 executor) with the segment IO work in #10568 (protocol) and #10527 (FileSegmentIo). It runs the PR compression benchmark so V2 decompression on the new IO protocol can be compared with develop's V1 baseline on CI's runners. If this landed, develop's stored Vortex decompress numbers would switch to V2.

The V2 executor's IO is ported from vortex_io::request onto vortex_io::segments, and compress-bench reads through FileSegmentIo, which reuses the file's v1 segment source (cache, request sharing, coalescing driver).

Changes on top of #10540 and #10527

vortex-scan driver:
- Run and Driver submit each root's requests to its SegmentIoSession and hold the handles.
- Each item's fetches finish through its own FuturesUnordered, so a poll_ready report yields exactly the finished fetches, without rescanning outstanding ones on wide tables.
- Run::poll_io replaces Run::complete, and sessions are dropped instead of cleared.
- The scripted harness is a recording SegmentIoSession, and the driver tests are ported. Two tests were dropped because the protocol no longer allows the case they covered: a source answering a segment with a size, and explicit clear calls.
vortex-layout V2:
- ScanAdaptor fetches each segment once by IoKey::segment while any scan read of it is outstanding, and hands the result to every waiting read.
- ScanFile carries the file's RegisteredSource, and each split reads through a new session of its service.
- The old SegmentScanIo adapter is removed. The V2 tests use a small test service over their in-memory segment sources.
vortex-file: VortexFile::scan_file registers the file's segment source with the session's FileSegmentIo.

Checks run:

cargo clippy -p vortex-io -p vortex-scan -p vortex-layout -p vortex-file --all-targets --all-features -- -D warnings
cargo clippy -p compress-bench --all-targets -- -D warnings
cargo +nightly-2026-09-10 fmt -p vortex-scan -p vortex-layout -p vortex-file
cargo test --lib: vortex-io (118), vortex-scan (58), vortex-layout (539), vortex-file (170)

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9

Generated by Claude Code

Replaces the byte-range planning protocol (`vortex_io::request`) with one keyed by segment, and adds the service that reads opened files through it. - vortex-io: `segments` holds the protocol. A request names a `SegmentKey` (`SourceId`, `SegmentId`) with an `IoIntent`. A `SegmentIoSession` registers a whole batch before any read is wanted and returns one `SegmentHandle` per request; dropping the last handle withdraws the read. `poll_ready` reports which owners' fetches finished. `SegmentId` moves here and vortex-layout re-exports it. - vortex-scan: planners and morsels publish `SegmentRequest`s and receive `deliver(key, bytes)`; they no longer implement an IO consumer. - vortex-file: `FileSegmentIo`, one per session, serves every opened file. Requests are answered from the segment cache (including the footer's initial segments), by slicing buffered files, or through the file's coalescing read driver, sharing reads of a byte range across sessions. `VortexFile` carries its registration as `segment_io()`. - vortex-layout: segment caches can answer synchronously via `SegmentCache::get_if_ready`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

The segment IO service borrowed v1's read driver from `FileSegmentSource`, which tied it to v1's `Request`/`Polled`/`Dropped` events and left the new system depending on the old one's internals. `segment_io::driver::SegmentDriver` is a per-file task with a three-message interface: register, want (urgent or background), withdraw. It chooses physical reads only when the reader has a free slot, after taking every queued message, so a batch registered together is coalesced together. A read starts at a wanted range, fetches ahead of prefetches, and extends over registered neighbours within the reader's coalescing window. v1's `FileSegmentSource` is unchanged. The service moves to a top-level `segment_io` module: `mod.rs` (registry, files, reads, sessions), `driver.rs`, and `tests.rs`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

…icated one Drops `segment_io::driver::SegmentDriver`. `FileSegmentIo` goes back to borrowing the file's v1 read driver, which is the configuration that was benchmarked. A TODO on `Backend::Driver` records the follow-up: a driver of its own, started lazily and optimised independently of v1's `ReadEvent`s. The service stays in the top-level `segment_io` module with its tests in `tests.rs`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Describe how to land the vortex-scan planning driver running develop's pipeline::Scan decode path, with IO through the planning protocol, as a small PR stack extracted from 42b29cc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com>

`SegmentCacheSourceAdapter::request` answered from `get_if_ready` before asking its source, which changed the path every v1 segment read takes. Only planner-driven scans need `get_if_ready`, so the adapter goes back to how it is on develop, along with its test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

`FileSegmentIo` now wraps a registered `SegmentSource` instead of reaching into `FileSegmentSource`'s driver, so v1 is untouched: open, `VortexFile`, the segment sources and caches are back to develop. `SegmentSource::request` registers a read with the coalescing driver without starting it, so every request calls it; fetches and prefetches are then polled once after the whole batch is registered, and dropping the handles drops the request. Cache hits, initial footer segments and request sharing come from the source as for v1 reads. `SegmentKey` holds the segment's `u32` id, so `SegmentId` stays in vortex-layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

…e fork Port the planning driver (`Run`/`Driver`) into vortex-scan and a projection-only V2 executor into vortex-layout that runs develop's pipeline `Scan` through the planning protocol, with every segment read issued as an `IoRequest` served from the file's `SegmentSource`. `ScanBuilder`'s lazy prepare step is the single place a streaming scan picks its executor: `v2::select` returns V1 unless `VORTEX_SCAN_V2=1`, the builder carries a `ScanFile`, and the scan has no filter, no limit and no list layout. `VortexFile::scan()` attaches the `ScanFile`; no caller changes. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

…ndoff doc Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

Review fixes for the segment IO protocol and `FileSegmentIo`: - Prefetches are tracked by the session and driven from `poll_ready`. A prefetch whose first poll waited on an asynchronous cache lookup was never marked wanted, so its read never started. - Every fetch is reported by `poll_ready`, including one finished at submit (a cache hit or an error), instead of leaving its owner unreported and `poll_ready` failing with nothing in flight. - `IoIntent::Forget` is removed: withdrawal is dropping an announcement's handles. Docs now say the session holds fetches and prefetches until they finish, that a service *may* read fetches first, and that a session keeps a source it has read readable. - `SourceId`s are unique across services, so a key submitted to the wrong service is rejected. Each session keeps one keep-alive per source rather than allocating one per request, and a registration holds its registry strongly. - `deliver` documents that failed fetches are never delivered. New tests: a fetch completed through `poll_ready` alone, a prefetch behind a pending cache (fails without the fix), fetches finished at submit, and one session's announcement widening another session's fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Develop's lower keeps Zoned and LegacyStats nodes in the plan, so the boundary walk stopped at them and every default-written file ran as a single split. Follow their data child. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

Each split had its own ValueCache, so a dictionary shared by many chunks was decoded once per split (HashTags 2.2x, CMSprovider 1.7x slower than V1). Share one cache per scan, as V1's reader does. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

Reuse V1 split policies, avoid redundant identity-projection planning, and release decoded dictionary values after their consuming splits finish. Reduce pipeline allocation, plan retention, and ready-I/O scheduling overhead. Validated with repeated V1/V2/Parquet decompression comparisons, output parity across 14 datasets, and 175 focused pipeline and scan tests. Some workloads still retain modest RSS and parallel duplicate-read overhead versus V1. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Measure preparation, compilation and in-memory segment delivery for narrow and wide pipeline scans in the existing CodSpeed layout shard. Document the stable identity of scoped dictionary-cache keys and use an explicit Arc clone in the V2 test so the scan code passes the CI Clippy configurations. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Keep private implementation names as code and remove the redundant v2 link target so the planner documentation builds with warnings denied. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Adds ScanBuilder::with_scan_v2 to choose the planner-driven executor for one scan, overriding VORTEX_SCAN_V2, and has compress-bench's Vortex decompression call it, so CI measures V2 decompression against develop's V1 baseline without relying on an environment variable. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtNyxw9KerLDyt5eXnpnRE Signed-off-by: Claude <noreply@anthropic.com>

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtNyxw9KerLDyt5eXnpnRE Signed-off-by: Claude <noreply@anthropic.com>

Use an explicit V2 ScanBuilder import for benchmark decompression and restore the V1 builder. Remove runtime executor selection and fallback, restrict planning and graph helper visibility, and cover the direct stream and Arrow paths. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Signed-off-by: Claude <noreply@anthropic.com>

Work in progress, does not compile yet: vortex-layout's V2 tests still need a segment IO service over test segment sources. - vortex-scan: `Run`/`Driver` submit to a root's `SegmentIoSession` and hold the handles; each item's fetches finish through its own `FuturesUnordered`, so a `poll_ready` report yields exactly the finished fetches. The scripted harness is a recording `SegmentIoSession`; driver tests are ported. - vortex-layout: `ScanAdaptor` publishes one fetch per segment key; `ScanFile` carries the file's `RegisteredSource`; the old `SegmentScanIo` adapter is removed. - vortex-file: `VortexFile::scan_file` registers with `FileSegmentIo`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

A request now names `IoKey { source: SegmentSourceId, request: SourceRequest }` where `SourceRequest ::= Size | Segment(id)`, so one service can serve many segment sources and answer more than segment reads. Reads resolve to a `SourceResult` (`Size(u64)` or `Bytes(BufferHandle)`), and planners and morsels receive `deliver(key, result)`. Renames: `SourceId` -> `SegmentSourceId`, `SegmentKey` -> `IoKey`, `SegmentRequest` -> `IoRequest`, `SegmentHandle` -> `IoHandle`, `SharedBytes` -> `SharedResult`. `FileSegmentIo::register` takes the source's size and answers `Size` at submit without a read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Signed-off-by: Claude <noreply@anthropic.com>

A file opened with a cached footer never reads its size, so `FileSegmentIo::register` takes `Option<u64>`; a size request of a source registered without one fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Signed-off-by: Claude <noreply@anthropic.com>

Ports #10540's planner-driven executor from `vortex_io::request` onto `vortex_io::segments`: - vortex-scan: `Run`/`Driver` submit each root's requests to its `SegmentIoSession` and hold the handles. Each item's fetches finish through its own `FuturesUnordered`, so a `poll_ready` report yields exactly the finished fetches without rescanning outstanding ones. `Run::poll_io` replaces `Run::complete`; sessions are dropped instead of cleared. The scripted harness is a recording `SegmentIoSession` and the driver tests are ported. - vortex-layout: `ScanAdaptor` fetches each segment once by `IoKey::segment` while any scan read of it is outstanding, and hands the result to every waiting read. `ScanFile` carries the file's `RegisteredSource`; each split reads through a new session of its service. The old `SegmentScanIo` adapter is removed. - vortex-file: `VortexFile::scan_file` registers the file's segment source with the session's `FileSegmentIo`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Replaces the byte-range planning protocol (`vortex_io::request`) with `vortex_io::segments`, keyed by segment source and source request. - A request names `IoKey { source: SegmentSourceId, request: SourceRequest }`, where `SourceRequest ::= Size | Segment(id)`, with an `IoIntent`: `Fetch`, `Prefetch` or `Announce`. One service serves many segment sources, and reads resolve to a `SourceResult` (`Size(u64)` or `Bytes(BufferHandle)`). - `SegmentIoSession::submit` registers a whole batch before any read starts and returns one `IoHandle` per request. The session holds fetches and prefetches until they finish; an announcement is withdrawn by dropping its handles. - `poll_ready` reports every fetch's owner once it finishes and drives the session's prefetches, through the shared `FetchTracker` helper. `SegmentIoService::session` opens one session per scope, and `RegisteredSource` is a source's registration with a service. - vortex-scan: `Planner` and `Morsel` drop the `IoConsumer` supertrait, return `NeedsIO(Vec<IoRequest>)`, and receive `deliver(key, result)` once per successful fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Signed-off-by: Claude <noreply@anthropic.com>

#10540 landed on develop with the V2 executor on vortex_io::request. Keep this branch's port onto vortex_io::segments for the V2 executor files, and drop scan/v2/io.rs again. Signed-off-by: Claude <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

…egment-io Signed-off-by: Claude <noreply@anthropic.com>

codspeed · 2026-10-16T17:53:44Z

Merging this PR will degrade performance by 7.26%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

density_sweep_dense_runs[0.9]

⚠️

18 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

test_repeated_scan_scalar_at[int_str_float]

decompress[u16, (4000, 256)]

compress_fsst[(500, 4, 8)]

random_i128[0.01]

test_scan_filter[int_float]

test_scan[int_str_float]

test_scan_filter[int_str]

test_scan[int_str]

multipolygon_mixed_validity

test_scan_filter[int_str_float]

…and 8 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

test_repeated_scan_scalar_at[int_str_float]

test_scan_filter[int]

test_scan_read_all_to_arrow_array[int_float]

test_scan_filter_to_arrow_array[int_float]

test_repeated_scan_filter[int_float]

test_scan_scalar_at[int_float]

test_repeated_scan_to_arrow_array[int_float]

test_repeated_scan_scalar_at[int]

test_repeated_scan_filter[int_str]

test_scan_filter_to_arrow_array[int_str]

…and 30 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 4 regressed benchmarks
✅ 2391 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌ ⚠️	Simulation	`test_repeated_scan_scalar_at[int_str_float]`	530.9 µs	606.4 µs	-12.46%
❌	WallTime	`bitpack_blocked_compress_avx2`	6.8 µs	7.6 µs	-11.19%
❌ ℹ️	Simulation	`test_scan_filter[int]`	3.1 ms	3.4 ms	-10.84%
❌	WallTime	`threshold_avx2[80]`	2.6 µs	2.9 µs	-10.32%
⚡	WallTime	`words_gather_dispatch_neon[65536]`	2.3 µs	2.1 µs	+10.34%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/v2-scan-segment-io (d3f81d8) with develop (d3bac42)}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

joseph-isaacs · 2026-10-16T17:54:07Z

CodSpeed Performance Analysis flags one regression: vortex-python test_scan_filter[int], 3.1 ms → 3.4 ms (−10.94%). I don't think this PR causes it:

The path: that benchmark runs a filtered scan, which goes through the V1 executor (the V2 builder doesn't take filters). This PR changes only V2 scan code, its IO protocol and FileSegmentIo.
History: the same family of Python scan benchmarks reported −11% to −13% on feat: switch vortex compress to use v2 scan api #10540.
Measurement: CodSpeed itself flags these runs for heavy system-call time and different runtime environments.

No fix is needed, and CodSpeed can't be re-run from here. The result this PR exists for is the compression benchmark (action/bench-compress).

Generated by Claude Code

github-actions · 2026-10-16T17:57:30Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`32b5cd1`	compress-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-10-16T18:04:43Z

Benchmarks: Compression 📖

Commits: PR 32b5cd11 vs base d3bac42e

vortex / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	778795387 / 782455081 / -0.5%
compress time/Bimbo	3617582993 / 3636251356 / -0.5%
compress time/CMSprovider	2540589384 / 2525795841 / +0.6%
compress time/Euro2016	263788630 / 265804923 / -0.8%
compress time/Food	317263855 / 315791004 / +0.5%
compress time/HashTags	514939094 / 520955599 / -1.2%
compress time/TPC-H l_comment canonical	612818893 / 611819111 / +0.2%
compress time/TPC-H l_comment chunked	614184961 / 610641715 / +0.6%
compress time/taxi	390942286 / 389059351 / +0.5%
compress time/wide table cols=100 chunks=1 rows=1000	4150636 / 4101773 / +1.2%
compress time/wide table cols=100 chunks=50 rows=1000	4192674 / 4256724 / -1.5%
compress time/wide table cols=1000 chunks=1 rows=1000	41614298 / 42691475 / -2.5%
compress time/wide table cols=1000 chunks=50 rows=1000	42527534 / 41576501 / +2.3%
compress time/wide table cols=10000 chunks=1 rows=1000	481937464 / 517345945 / -6.8%
compress time/wide table cols=10000 chunks=50 rows=1000	520648235 / 522687193 / -0.4%
decompress time/Arade	12472894 / 12478406 / -0.0%
decompress time/Bimbo	63700865 / 61686766 / +3.3%
decompress time/CMSprovider	38457697 / 38546594 / -0.2%
decompress time/Euro2016	11720447 / 12011198 / -2.4%
decompress time/Food	4062475 / 3782857 / +7.4%
decompress time/HashTags	37399739 / 35231760 / +6.2%
decompress time/TPC-H l_comment canonical	13849606 / 13737991 / +0.8%
decompress time/TPC-H l_comment chunked	13793601 / 13770952 / +0.2%
decompress time/taxi	9214113 / 9078078 / +1.5%
decompress time/wide table cols=100 chunks=1 rows=1000	1264576 / 1210604 / +4.5%
decompress time/wide table cols=100 chunks=50 rows=1000	1267045 / 1217537 / +4.1%
decompress time/wide table cols=1000 chunks=1 rows=1000	12419976 / 11857128 / +4.7%
decompress time/wide table cols=1000 chunks=50 rows=1000	12463941 / 11936426 / +4.4%
decompress time/wide table cols=10000 chunks=1 rows=1000	134479483 / 130285008 / +3.2%
decompress time/wide table cols=10000 chunks=50 rows=1000	135037506 / 129912544 / +3.9%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164125572 / 164130260 / -0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (1.008x ➖, 0↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	3.87209855 / 3.77715158 / +2.5%
vortex:parquet-zstd ratio compress time/Bimbo	3.6184274 / 3.64586918 / -0.8%
vortex:parquet-zstd ratio compress time/CMSprovider	5.20088798 / 5.16207769 / +0.8%
vortex:parquet-zstd ratio compress time/Euro2016	0.836537092 / 0.837504751 / -0.1%
vortex:parquet-zstd ratio compress time/Food	4.56927712 / 4.46137794 / +2.4%
vortex:parquet-zstd ratio compress time/HashTags	3.22609131 / 3.33171436 / -3.2%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	2.99426177 / 2.96642835 / +0.9%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	2.99139663 / 2.89465426 / +3.3%
vortex:parquet-zstd ratio compress time/taxi	4.00173082 / 4.3392379 / -7.8%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	3.28023279 / 3.1965064 / +2.6%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	3.17168846 / 3.1969604 / -0.8%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	3.09894341 / 3.14476436 / -1.5%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	3.24426836 / 3.1268751 / +3.8%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	3.41291529 / 3.55348718 / -4.0%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	3.65566207 / 3.80986147 / -4.0%
vortex:parquet-zstd ratio decompress time/Arade	0.251465799 / 0.252271237 / -0.3%
vortex:parquet-zstd ratio decompress time/Bimbo	0.701097385 / 0.67603829 / +3.7%
vortex:parquet-zstd ratio decompress time/CMSprovider	0.371965466 / 0.371513036 / +0.1%
vortex:parquet-zstd ratio decompress time/Euro2016	0.0779741624 / 0.0797098273 / -2.2%
vortex:parquet-zstd ratio decompress time/Food	0.162020463 / 0.150713134 / +7.5%
vortex:parquet-zstd ratio decompress time/HashTags	0.0887398826 / 0.0838332529 / +5.9%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.167739505 / 0.165496619 / +1.4%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.167084505 / 0.165969254 / +0.7%
vortex:parquet-zstd ratio decompress time/taxi	0.155516603 / 0.152945824 / +1.7%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.607826625 / 0.593045528 / +2.5%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.616937973 / 0.591803563 / +4.2%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.527734073 / 0.498590258 / +5.8%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.522845903 / 0.49966453 / +4.6%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.526663588 / 0.514738879 / +2.3%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.528716904 / 0.510518034 / +3.6%
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33458157 / 1.33461969 / -0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (0.999x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	201130053 / 207154800 / -2.9%
parquet_rs-zstd compress time/Bimbo	999766638 / 997361993 / +0.2%
parquet_rs-zstd compress time/CMSprovider	488491464 / 489298301 / -0.2%
parquet_rs-zstd compress time/Euro2016	315334051 / 317377212 / -0.6%
parquet_rs-zstd compress time/Food	69434146 / 70783289 / -1.9%
parquet_rs-zstd compress time/HashTags	159617024 / 156362624 / +2.1%
parquet_rs-zstd compress time/TPC-H l_comment canonical	204664435 / 206247729 / -0.8%
parquet_rs-zstd compress time/TPC-H l_comment chunked	205317127 / 210954974 / -2.7%
parquet_rs-zstd compress time/taxi	97693299 / 89660756 / +9.0%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	1265348 / 1283205 / -1.4%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	1321906 / 1331491 / -0.7%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	13428544 / 13575413 / -1.1%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	13108513 / 13296502 / -1.4%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	141209911 / 145588240 / -3.0%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	142422419 / 137193228 / +3.8%
parquet_rs-zstd decompress time/Arade	49600757 / 49464244 / +0.3%
parquet_rs-zstd decompress time/Bimbo	90858797 / 91247444 / -0.4%
parquet_rs-zstd decompress time/CMSprovider	103390504 / 103755697 / -0.4%
parquet_rs-zstd decompress time/Euro2016	150311932 / 150686539 / -0.2%
parquet_rs-zstd decompress time/Food	25073839 / 25099717 / -0.1%
parquet_rs-zstd decompress time/HashTags	421453555 / 420259966 / +0.3%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	82566155 / 83010705 / -0.5%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	82554639 / 82972910 / -0.5%
parquet_rs-zstd decompress time/taxi	59248420 / 59354860 / -0.2%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2080488 / 2041334 / +1.9%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2053764 / 2057333 / -0.2%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	23534535 / 23781307 / -1.0%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23838651 / 23888880 / -0.2%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	255342283 / 253108932 / +0.9%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	255406069 / 254471998 / +0.4%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (1.011x ➖, 0↑ 2↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	99063348 / 100683844 / -1.6%
arrow-ipc compress time/Bimbo	618432992 / 631760254 / -2.1%
arrow-ipc compress time/CMSprovider	384087048 / 385208829 / -0.3%
arrow-ipc compress time/Euro2016	38252425 / 38642070 / -1.0%
arrow-ipc compress time/Food	33680624 / 34329598 / -1.9%
arrow-ipc compress time/HashTags	78770702 / 79471299 / -0.9%
arrow-ipc compress time/TPC-H l_comment canonical	949892516 / 951484876 / -0.2%
arrow-ipc compress time/TPC-H l_comment chunked	956638660 / 959731897 / -0.3%
arrow-ipc compress time/taxi	75079659 / 75614901 / -0.7%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	197741 / 203796 / -3.0%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	198347 / 202770 / -2.2%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2240418 / 2283324 / -1.9%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2215567 / 2258988 / -1.9%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25298893 / 25579765 / -1.1%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25466631 / 25718213 / -1.0%
arrow-ipc decompress time/Arade	181836173 / 140602792 / +29.3% 🔴
arrow-ipc decompress time/Bimbo	429491675 / 419836777 / +2.3%
arrow-ipc decompress time/CMSprovider	581816386 / 584423574 / -0.4%
arrow-ipc decompress time/Euro2016	74780366 / 74828253 / -0.1%
arrow-ipc decompress time/Food	58405454 / 48750052 / +19.8% 🔴
arrow-ipc decompress time/HashTags	142772805 / 143239785 / -0.3%
arrow-ipc decompress time/TPC-H l_comment canonical	627683462 / 633740620 / -1.0%
arrow-ipc decompress time/TPC-H l_comment chunked	627589670 / 635257193 / -1.2%
arrow-ipc decompress time/taxi	48520127 / 45264372 / +7.2%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	319625 / 320497 / -0.3%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	321932 / 320553 / +0.4%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3706289 / 3515429 / +5.4%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3612772 / 3581468 / +0.9%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	35857119 / 36217060 / -1.0%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36046543 / 36790000 / -2.0%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

claude and others added 30 commits October 15, 2026 17:29

docs(file): fix intra-doc links in segment_io's module docs

6ab900b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

chore(scan): clippy fixes and record the single-fork design in the ha…

7c1cb9b

…ndoff doc Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6

docs(scan): fix planner documentation links

7b15cfc

Keep private implementation names as code and remove the redundant v2 link target so the planner documentation builds with warnings denied. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

docs(scan): remove the planner decode-slice handoff doc

3b36128

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtNyxw9KerLDyt5eXnpnRE Signed-off-by: Claude <noreply@anthropic.com>

refactor(scan): use fixed V2 chunks and an execution adaptor

e4cfe34

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

perf(scan): bound V2 row chunks by serialized size

e39866a

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

test(scan): fix V2 chunk test lint and formatting

740e1ef

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Merge branch 'ji/segment-io' into ji/v2-scan-segment-io

d91f175

Signed-off-by: Claude <noreply@anthropic.com>

docs(io): fix grammar in the segments module docs

d56e68a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Merge branch 'ji/segment-io' into ji/v2-scan-segment-io

c793fe1

Signed-off-by: Claude <noreply@anthropic.com>

wip(scan): rename to IoKey/IoRequest/IoHandle

270a9e8

Signed-off-by: Claude <noreply@anthropic.com>

Merge branch 'ji/segment-io' into ji/v2-scan-segment-io

96a3f7b

Signed-off-by: Claude <noreply@anthropic.com>

claude added 2 commits October 16, 2026 16:44

Merge branch 'ji/segment-io-protocol' into ji/segment-io

6be6860

Signed-off-by: Claude <noreply@anthropic.com>

joseph-isaacs added changelog/skip Do not list PR in the changelog action/bench-compress Run only the compression benchmark on this PR labels Oct 16, 2026 — with Claude

joseph-isaacs added action/bench-compress Run only the compression benchmark on this PR and removed action/bench-compress Run only the compression benchmark on this PR labels Oct 16, 2026 — with Claude

github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 16, 2026

claude added 2 commits October 16, 2026 17:51

test(file): fix a typo in a segment_io test name

7002056

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Merge remote-tracking branch 'origin/ji/segment-io' into ji/v2-scan-s…

d3f81d8

…egment-io Signed-off-by: Claude <noreply@anthropic.com>

Navigation Menu

Repository navigation

bench: V2 scan decompression on the segment IO protocol - #10569

bench: V2 scan decompression on the segment IO protocol#10569
joseph-isaacs wants to merge 35 commits into
developfrom
ji/v2-scan-segment-io

joseph-isaacs commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

joseph-isaacs commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joseph-isaacs commented Oct 16, 2026

Summary

Changes on top of #10540 and #10527

Uh oh!

codspeed Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 7.26%

Performance Changes

Footnotes

Uh oh!

joseph-isaacs commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Oct 16, 2026

Benchmarks: Compression 📖

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed Bot commented Oct 16, 2026 •

edited

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading