Repository navigation
perf(datafusion): avoid Arrow session lookup for ordinary scalars - #10541
Conversation
Fetch the Arrow session only in temporal conversions that need its type mapping. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will degrade performance by 16.55%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | filter_neon[8] |
867 ns | 1,382 ns | -37.26% |
| ❌ | WallTime | threshold_neon[8] |
1.5 µs | 1.9 µs | -22.13% |
| ❌ | WallTime | threshold_neon[80] |
3.7 µs | 4.5 µs | -17.64% |
| ❌ | WallTime | filter_neon[64] |
1.7 µs | 2.1 µs | -15.69% |
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.6 µs | -11.55% |
| ⚡ ℹ️ | Simulation | test_scan_filter[int] |
3.4 ms | 3 ms | +12.53% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/df-scalar-session (c9e7439) with develop (ad42859)
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.854x ✅, 5↑ 1↓)
duckdb / parquet / ns (0.990x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 4↑ 4↓)
duckdb / parquet / ns (0.989x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 5↑ 4↓)
duckdb / parquet / ns (1.005x ➖, 1↑ 3↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.365x ❌, 0↑ 13↓)
datafusion / parquet / ns (1.068x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.278x ➖, 0↑ 6↓)
duckdb / parquet / ns (1.124x ➖, 0↑ 0↓)
|
Summary
Converting ordinary DataFusion scalars currently retrieves an Arrow session even though those conversions do not use it. Fetch it only for the date and time conversions that need Arrow's type mapping.
Changes
Move the session lookup into the two temporal conversion branches. Integer, floating-point, boolean, string, binary, and other direct conversions keep their existing behavior while avoiding the unused session lookup and guard.
No isolated benchmark speedup is claimed. All 102 existing scalar-conversion tests pass on stock DataFusion.