Repository navigation
Add ScanThreading to stop double-counting scan concurrency under DataFusion - #10546
joseph-isaacs wants to merge 2 commits into
Conversation
ScanBuilder multiplied its per-thread concurrency by the available parallelism on every stream. DataFusion already opens one file stream per partition, so each stream kept concurrency x cores splits in flight and the total scaled with cores squared. ScanThreading::Internal keeps the existing behaviour. ScanThreading::External uses the concurrency as-is, and the DataFusion opener and DuckDB reader now select it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QV2TJ3vQXsPAXGPuLk3Qjt Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
With external threading each DataFusion file stream keeps a fixed number of splits in flight. When DataFusion plans fewer partitions than cores that leaves the machine underused, so split the machine-wide scan budget (DEFAULT_SCAN_CONCURRENCY x cores) evenly across the scan's partitions instead. An explicit scan_concurrency still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QV2TJ3vQXsPAXGPuLk3Qjt Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | random_i8[0.01] |
103.8 µs | 132.6 µs | -21.69% |
| ⚡ |
Simulation | density_sweep_dense_runs[0.001] |
46.9 µs | 29 µs | +62.06% |
| ⚡ |
Simulation | random_i128[0.01] |
5.9 µs | 4.9 µs | +21.24% |
| ⚡ ℹ️ | Simulation | test_scan_filter[int] |
3.5 ms | 3 ms | +15.11% |
| ⚡ | WallTime | bitpack_blocked_compress_avx2 |
7.6 µs | 6.7 µs | +13.75% |
| ⚡ | WallTime | threshold_avx2[80] |
3 µs | 2.7 µs | +11.24% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/laughing-gates-luy2rl (26cbb6a) with develop (c48fa1b)2
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(6eae823) during the generation of this report, so c48fa1b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.939x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 3↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 3↑ 2↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.035x ➖, 0↑ 5↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.003x ➖, 2↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.020x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.095x ➖, 2↑ 7↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.030x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.999x ➖, 0↑ 0↓)
File Size Changes (201 files changed, +0.0% overall, 98↑ 103↓)
Totals:
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.985x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.910x ➖, 3↑ 1↓)
datafusion / parquet / ns (1.073x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.007x ➖, 1↑ 2↓)
|
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 1↑ 4↓)
duckdb / vortex-compact / ns (1.005x ➖, 3↑ 4↓)
duckdb / parquet / ns (1.009x ➖, 2↑ 4↓)
duckdb / duckdb / ns (1.006x ➖, 2↑ 8↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.025x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.021x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.033x ➖, 0↑ 5↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.005x ➖, 1↑ 1↓)
duckdb / duckdb / ns (0.994x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.971x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.049x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 3↑ 3↓)
duckdb / vortex-compact / ns (0.956x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.965x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.014x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.965x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.011x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.999x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.000x ➖, 3↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 4↑ 2↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.056x ➖, 0↑ 6↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.995x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.027x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.053x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.934x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.191x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.030x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.681x ❌, 0↑ 8↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.036x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.975x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.150x ➖, 1↑ 8↓)
datafusion / parquet / ns (1.067x ➖, 1↑ 4↓)
duckdb / vortex-compact / ns (0.950x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.129x ➖, 0↑ 4↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.150x ➖, 1↑ 5↓)
datafusion / vortex-compact / ns (1.192x ➖, 0↑ 9↓)
datafusion / parquet / ns (0.897x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.374x ❌, 0↑ 11↓)
duckdb / vortex-compact / ns (1.266x ➖, 0↑ 10↓)
duckdb / parquet / ns (1.031x ➖, 0↑ 1↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.767x ✅, 2↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.974x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.165x ❌, 2↑ 7↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.958x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.981x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.1% overall, 35↑ 65↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.972x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.061x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.982x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.028x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.326x ❌, 0↑ 9↓)
datafusion / parquet / ns (1.086x ➖, 0↑ 3↓)
duckdb / vortex-compact / ns (1.241x ➖, 0↑ 10↓)
duckdb / parquet / ns (1.041x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.431x ❌, 0↑ 14↓)
datafusion / vortex-compact / ns (1.473x ❌, 0↑ 15↓)
datafusion / parquet / ns (1.144x ➖, 1↑ 4↓)
duckdb / vortex-file-compressed / ns (1.064x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (0.831x ➖, 4↑ 0↓)
duckdb / parquet / ns (1.289x ➖, 0↑ 11↓)
|
Summary
RepeatedScan::execute_streamsizes every stream to the whole machine:concurrency (4) × get_available_parallelism(). DataFusion'sVortexOpeneralready opens one file stream per partition, so on a 96-core box each of 96 streams kept 4 × 96 = 384 splits in flight. The cores were counted twice, so the total grew with cores².This PR lets the caller say whether the scan or an outside engine drives the threading.
Changes
ScanThreading(vortex-layout/src/scan/threading.rs):Internal(default, existing behaviour:concurrency × cores) orExternal(concurrencyas-is). It is set withScanBuilder::with_threadingand carried intoRepeatedScan. The per-thread default is now the publicDEFAULT_SCAN_CONCURRENCY = 4.persistent/opener.rs,persistent/source.rs):VortexOpenerscans withExternal. Whenscan_concurrencyis unset, each partition getsceil(4 × cores / partitions), so the total in flight equals one machine-wide scan however many partitions DataFusion plans. That is 4 per stream at one partition per core, and more when DataFusion plans fewer partitions than cores. An explicitscan_concurrencystill overrides it.file_reader.rs): markedExternal. DuckDB usesRepeatedScan::execute, which spawns nothing, and each DuckDB thread awaits one split at a time, so it was never double counted and runtime behaviour is unchanged.v2VortexDataSourcestaysInternal: it shows DataFusion a single partition and runs the splits itself.Tests: a unit test for
ScanThreading::splits_in_flightandrstestcases for the DataFusion per-partition sizing.Checks run:
cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warningscargo +nightly-2026-09-10 fmtcargo test -p vortex-datafusion -p vortex-layout --lib(all pass)vortex-duckdbcould not be built locally: the DuckDB source download was blocked. I have not run any benchmarks.API Changes
ScanThreading,ScanBuilder::with_threading/threading, andDEFAULT_SCAN_CONCURRENCY.RepeatedScan::newispub(crate), so it is unaffected.VortexSourcebehaviour change: the default per-file in-flight split count drops from4 × corestoceil(4 × cores / partitions), andscan_concurrencyis now per file rather than per thread.🤖 Generated with Claude Code
https://claude.ai/code/session_01QV2TJ3vQXsPAXGPuLk3Qjt
Generated by Claude Code