[jackylee-ch]

What

Vortex's min, max and sum statistics skip NaN, but the DuckDB and DataFusion integrations report
them to the engine as exact bounds over every value. On a float column holding NaN, both engines
then answer from the statistics:

• SELECT max(x) returns the largest non-NaN value instead of NaN;
• WHERE x = 'NaN' returns no rows (DuckDB plans an EMPTY_RESULT, DataFusion prunes the file);
• in DataFusion, where a negative NaN sorts below every value, WHERE x < 0.5 drops it as well.

Fix

Check the footer's exact NaNCount. DuckDB sorts NaN above every value, so its max becomes NaN
when the count is non-zero and is left out when the count is unknown; min is already correct.
DataFusion sorts NaN by sign, so min, max and sum are reported only for a NaN-free column (this
also guards the v2 source, which reports no statistics yet). Other columns are unchanged; float
columns in files from before NaNCount (0.29 to 0.32) lose their bounds.

The new NaN cases fail on develop (max(x) returns 2, the NaN filters return no rows); two plan
checks pin that NaN-free float columns are still answered from statistics.

constant_distinct_count (from #10448) now reuses is_nan_free. #10265 and #9775 rewrite these
functions.

AI assistance

Written with agentic AI assistance; each failure above was reproduced on develop before the fix.

[codspeed]

Merging this PR will degrade performance by 12.92%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

• bench_compare_primitive[(10000, 2)]

⚠️ 34 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_scan_scalar_at[int_float]
• test_scan_to_arrow_array[int_float]
• pcodec_compress_f64
• test_repeated_scan_scalar_at[int]
• density_sweep_dense_runs[0.001]
• test_repeated_scan_scalar_at[int_str_float]
• test_repeated_scan_scalar_at[int_str]
• test_scan[int_str_float]
• test_scan[int_str]
• test_scan_filter[int_float]
• …and 24 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

• test_scan_scalar_at[int_float]
• test_scan_filter[int]
• test_repeated_scan[int]
• test_scan_to_arrow_array[int_float]
• test_repeated_scan_scalar_at[int_float]
• test_repeated_scan_scalar_at[int]
• test_repeated_scan_scalar_at[int_str_float]
• test_scan_scalar_at[int]
• test_repeated_scan_scalar_at[int_str]
• test_scan[int_str_float]
• …and 30 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 3 regressed benchmarks
✅ 2256 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌ ⚠️	Simulation	test_scan_scalar_at[int_float]	968.1 µs	1,150.6 µs	-15.86%
❌	WallTime	bitpack_blocked_compress_avx2	6.7 µs	7.6 µs	-11.6%
❌ ℹ️	Simulation	test_scan_filter[int]	3 ms	3.4 ms	-11.23%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing jackylee-ch:fix/nan-aware-float-stats (f2b1771) with develop (e8974d6)}