Repository navigation
Bound V2 shared reads and partition DuckDB scan work - #10857
joseph-isaacs wants to merge 27 commits into
Conversation
Merging this PR will improve performance by 10.02%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing |
Polar Signals Profiling ResultsLatest Run
Previous Runs (100)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.022x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.019x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.016x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
File Size Changes (8 files changed, -0.0% overall, 0↑ 8↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.953x ➖, 4↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.019x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.986x ➖, 4↑ 2↓)
duckdb / parquet / ns (0.988x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.021x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 4↑ 6↓)
duckdb / parquet / ns (0.991x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.907x ➖, 4↑ 2↓)
datafusion / parquet / ns (1.033x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (1.053x ➖, 0↑ 3↓)
duckdb / parquet / ns (0.830x ➖, 5↑ 0↓)
|
|
We'd need to tackle ClickBench regression before merging |
…unks Fall back to the driver's normal partitioning when a lazy chunk's metadata cannot be initialized. Selection and pruning may avoid executing that chunk. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Only coalesce tiny dictionary regions below 8192 rows, allowing workers to balance dictionary expressions without cutting their cached value domains. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Run the PR and its develop base on the same ClickBench runner, with warmed ABBA timings, scan metrics, and native CPU profiles. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Avoid mixing current develop workflow actions with an older PR checkout during the controlled comparison. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The shared chunk planner had stopped emitting independent work, moving DataFusion scans onto the root task's step-by-step execution. Allow DataFusion to opt into independent chunks while DuckDB keeps its own work queue policy. Cover the independent planner's bounded selection scope. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The active benchmark run retains its pinned diagnostic workflow. Keep the PR change limited to DataFusion chunk scheduling and its regression coverage. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
V2 work ranges can overlap stored segments, duplicating reads and retaining buffers after their consumers finish. Drivers now share reads within one file execution and release them when owned row claims prove no unfinished work can read them again. DuckDB assigns work across its configured threads; DataFusion preserves independently scheduled scan chunks.
Changes
vortex-scan::planning::driver::FileReads. Queued planners, filtering, pruning, projection, and morsels transfer or splitRowClaimownership. Completed later parts can release their reads while an earlier part remains stalled.Spawninto ordinary child planning, moving chunk execution onto the root scan task. Restoring independent chunks recovers parallel compute and read scheduling. DuckDB retains ordinary child planners inside its own work assignments.Planner interfaces remain experimental; constructors and pruning continuations transfer row claims.
Performance
ClickBench CI and controlled diagnostics tested
aa2cf422. Final headdf3c42dfhas identical Rust sources; its only subsequent change removes temporary diagnostic workflows. The controls were built from developc56ee880and the preceding PR's Rust sources atd704b47a, with matching compiler flags.DataFusion Q38
Both controls and the candidate use V2 scanning and the same input files on one 96-core Graviton runner. Each version was warmed before two ABBA comparisons, with 30 iterations per trial and the first iteration excluded: 58 retained samples per version in each comparison.
The native CPU profiles corroborate the scheduling change: before, 22.72% of sampled cycles run through the root scan's
spawn_cpupath; after, 23.30% run through independentchunk_streamtasks and only 0.89% through that root path. The scan metrics report the same output rows and segment-cache misses, with individual I/O requests falling from 11 to 2. This is primarily an elapsed-time improvement from scheduling, not less total CPU work.The full 43-query report reduces DataFusion's attributed warm regression from +10.4% in the preceding rerun to +2.7%, with low confidence. Q20, Q21, Q23, Q26, Q28, and Q38 are now within 3.6% of the cached develop warm reference. This does not establish parity for every query: DataFusion Q24 remains +14.8%, and the combined Vortex cold geometric mean remains +13.4% in the report.
DuckDB Q28
52b50e21vs current52b50e21vs dictionary-aligned schedulerThe local comparison used 32 DuckDB/Tokio threads, 100 partitioned ClickBench files, and fresh connections. Both binaries were warmed; two eight-iteration trials per version retained 14 warm samples. Seven guard queries were checked, no benchmark/build workloads overlapped, and all 200 input-file size/mtime records remained unchanged. Existing benchmark processes and Cargo caches were preserved.
Current CI Q28 is 2.4% below the cached develop reference of 511.325 ms. Develop's DuckDB reference uses V1, so that comparison is not V2 versus V2. Earlier runs of the dictionary-aligned scheduler measured 500.956, 501.121, and 502.467 ms. Other DuckDB regressions remain, including Q20 (+14.3%) and Q38 (+24.8%) versus the cached reference.
Memory and retention
Restoring DataFusion Q38 parallelism increases median peak RSS across the paired 30-iteration trials from 1.95 to 2.32 GiB (+18.8%); develop measured 2.29 GiB. More simultaneous work is the tradeoff for the lower elapsed time.
The earlier local DuckDB Q28 comparison reduced peak RSS from 7.50–7.55 GiB to 4.95–5.84 GiB. These measurements compare V2 implementations and do not establish whole-process memory parity with V1. Decoded arrays and output buffers can retain memory independently of read-cache eviction.
The controlled 32-part retention test holds the first part stalled while later parts finish. Flat reads retained fall from 32 segments / 128 KiB under the former watermark policy to 2 / 8 KiB; dictionary reads fall from 33 / 144 KiB to 3 / 24 KiB. Each segment is fetched once, and the cache empties after the final claim drops.
Validation