Repository navigation
Compose and drive a V2 scan in DataFusion - #10708
Merged
Merged
Conversation
This was referenced Oct 23, 2026
Merging this PR will regress 1 benchmark4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-2-planners
branch
from
October 23, 2026 13:08
d091576 to
a6f15f1
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-3-drivers
branch
from
October 23, 2026 13:08
da8a7c8 to
a3ef12f
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-2-planners
branch
from
October 23, 2026 13:49
a6f15f1 to
1c7e1c8
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-3-drivers
branch
from
October 23, 2026 13:49
a3ef12f to
10ac271
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-2-planners
branch
from
October 23, 2026 13:56
1c7e1c8 to
26c4265
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-3-drivers
branch
2 times, most recently
from
October 23, 2026 14:07
f23da3a to
25458e3
Compare
joseph-isaacs
added
the
action/bench-sql
This was referenced Oct 26, 2026
joseph-isaacs
pushed a commit
that referenced
this pull request
Oct 26, 2026
The net change of #10708 (ji/scan-v2-chain-3-drivers at 25458e3, which includes the #10707 commits it stacks on) applied onto develop as one commit, for benchmarking only. Conflicts with develop are resolved by keeping both sides: the new Eval(pack)/Eval(Eval) rules sit beside the zoned-data rule, plan rules and `optimize` take the session, and the legacy-stats tests expect develop's packed zone conversion. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbaPHjo8XVLKjhGy9xoUMC
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-2-planners
branch
from
October 26, 2026 14:15
9f8bf1c to
d05e690
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-3-drivers
branch
from
October 26, 2026 14:16
25458e3 to
1ecdd22
Compare
A planner says what it needs, compute or reads. Where either runs is an engine's decision, so this crate stops driving: `ScanBuilder::prepare` lowers and plans a scan without IO into a `PreparedScan`, which exposes its rows, selection, filter plans and IO service, and `PrunePlanner` and `ChunkPlanner` are public for an engine to compose, the prune planner continuing into whatever it is built with and passing `None` when no proof ran. The builder's ordering, concurrency, mapping and streams go with the driver, and the layout's tests drive their scans on a run of their own. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener composes a scan's planners, a prune planner into a chunk planner when the scan has a filter and the chunk planner alone when it does not, and a driver of this crate runs them: the root run stepped on `spawn_cpu` tasks as DataFusion polls the stream, each spawned chunk on a task of its own up to the concurrency, or on the root run when the engine's partitions already fill the cores, with the concurrency derived from cores per partition unless configured. Batches are mapped to Arrow on the task that produced them. Every spawning decision of a V2 scan under DataFusion is in this crate. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`vortex_file::v2::ScanDriver` composes a scan's planners and runs them on the session's runtime, the way an engine's driver does, for the file API's callers: the compress and random-access benchmarks move to it. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener prepares the scan where compute runs, composes its planners itself, a prune planner continuing into a chunk planner when there is a filter and the chunk planner alone when there is not, and hands the driver the first planner. The driver only drives. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The prune planner skipped prefetching the zone maps of proofs an earlier scan of the file had proved. The segment list now comes from compiling the proof, and a proved proof compiles to its kept result without a segment scan, so the check repeated what compile already knew. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The scan code named the filter's zone-map plans proofs. They are the filter's pruning expressions evaluated over the file's zone map, so the layout, file and DataFusion code now says pruning, matching prune_zones, PrunedZones and the pruning field they already used. Behaviour is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
force-pushed
the
ji/scan-v2-chain-3-drivers
branch
from
October 26, 2026 14:28
1ecdd22 to
429e005
Compare
joseph-isaacs
added
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 2↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.924x ➖, 3↑ 2↓)
datafusion / parquet / ns (0.952x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.934x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.896x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.017x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.977x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.021x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.098x ➖, 0↑ 9↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 1↓)
No file size changes detected. |
joseph-isaacs
pushed a commit
that referenced
this pull request
Oct 26, 2026
…kDB V2 branch The DuckDB driver composes and drives the V2 planners the same way. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8NsCJMx4C45Wf2Df24qzZ
robert3005
approved these changes
Oct 26, 2026
joseph-isaacs
pushed a commit
that referenced
this pull request
Oct 26, 2026
With #10708 on develop, this branch's diff is only the DuckDB driver. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8NsCJMx4C45Wf2Df24qzZ
joseph-isaacs
pushed a commit
that referenced
this pull request
Oct 26, 2026
Resolve conflicts with #10708, which moved V2 scan driving out of ScanBuilder into ScanDrivers in vortex-datafusion and vortex-file, and renamed zone proofs to pruning plans. - Port per-root IO sessions and the inline-at-concurrency-one gate into both ScanDrivers. The DataFusion driver takes concurrency as the scan's total chunks in flight; the vortex-file driver keeps its per-worker-thread meaning. - Restore ScanBuilder::with_concurrency, used only to size filtered chunks, and set it from both drivers. - Keep the OnceCell scan cache, MaskScan and the is_zone_pruning dtype check under develop's pruning names. - Port dense_chunk_yields_a_batch_per_window onto PreparedScan; drop unordered_yields_before_the_split_finishes with the code develop removed. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJp2eSZRzHhv4cVCUNo6nV
This was referenced Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of a stack of three that lands the planner-driven V2 scan for DataFusion behind a switch, cut from #10694. Stacked on #10707 (the planner chain), itself on #10706. With this merged,
scan_v2runs TPC-H end to end; the perf and diagnostics commits of #10694 follow as separate PRs on top.Measured on its own
Against the V1 scan, locally (14 cores, interleaved A/B/B/A, 10 and 5 iterations a run) at the same code before the stack was cut (da8a7c8):
SF10 improves over #10707 (0.977 to 0.924) as the per-step task hops go; SF1 does not yet, and q13 at SF1 is far behind because every partition still plans the filter and projection itself (the plan-once and interning commits of #10694 fix that, taking the full branch to 0.963 at SF1). This PR's base is a branch, so CI's benchmark dispatch does not run on it.