Repository navigation
Support Spark floating point semantics in bound comparisons #10064
Description
Vortex’s existing total ordering distinguishes -0.0 from +0.0 and compares NaNs by their bit patterns. Spark treats signed zeros as equal and all NaNs as equal. Spark currently pushes float and double predicates using Vortex’s existing comparison functions, so a pushed filter can return different rows than Spark evaluation.
Add an explicit comparison mode or separate bound scalar functions for Spark-compatible floating point equality and ordering. Preserve Vortex’s current behavior as the default. Until Spark-compatible operations exist, the Spark connector should decline affected pushdown.
Regression cases should cover =, !=, ranges, and IN with signed zeros and NaNs, including distinct NaN payloads.
Activity
jackylee-ch commentedon Oct 14, 2026
DuckDB has the same semantics (all NaNs are equal and greater than any other value,
-0.0 = 0.0), and the DuckDB integration pushes float comparisons too; the stats side is #10479. With duckdb 1.5.6 and the released extension,x = 0.0returns 1 of 2 rows,x > 100drops a negative NaN, and a hash join with 0.0 on the build side returns 0 of 100,000-0.0probe rows, as the build key reaches the scan asx = 0.0. An explicit option, like the SQL semantics #10057 added tolist_contains, can lowercol op literalinto existing expressions without losing pruning (float comparisons already prune only NaN-free zones):>/>=gainOR is_nan(x),</<=gainAND NOT is_nan(x), zero literals treat-0.0and0.0alike, and NaN literals reduce tois_nanor a constant. It has to reachlist_containstoo, since DuckDBINfilters, small-build-side join filters included, go through it (x IN (0.0, 'nan')returns 2 of 4 rows) and #10052 probes floats by their bits. If that works for you, I can start with the comparisons in vortex-array (column-to-column float comparisons would be declined) and sequencelist_containsand the DuckDB and Spark wiring behind #10052, #10054, #9141 and #9658.Investigated with AI assistance; the numbers are from local runs.
Keep float comparisons in Spark and test against Spark 4.2