You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
IR: set-op and aggregate nullability, and the SQL top-k rewrite's output shape #657
Found by the DataFusion differential check of #539 (573 SQL queries run through DataFusion 54 and through the lowering). These are in the IR's schema derivation and canonicalization (already on main), not in the SQL frontend.
Problem
Set operations take the left schema.SetOp derives its schema from left only (ir/non_asap.rs), so SELECT 1 AS x UNION ALL SELECT NULL is declared non-null but returns NULL. DataFusion declares it nullable. (feat(sql): lower SQL to unified operator and scalar IR #539 fixes the same issue for Concat, which its ROLLUP / CUBE / GROUPING SETS lowering uses.)
Some aggregate results are declared non-null.stddev / variance / median / percentile results, and ClickHouse argMax / argMin, are NULL over empty input (e.g. stddev(a) … WHERE id = 1 with one row, or an empty group), but the aggregate schema derivation (pre_asap/agg_intent.rs, around the result-field construction) marks them non-null. The nullability fix in non_asap.rs covers only Sum / Avg / Min / Max.
The top-k rewrite changes SQL output.canonicalize turns SELECT g, count(*) AS c … GROUP BY g ORDER BY c DESC LIMIT k into a one-column TopK aggregate instead of rows of (g, c), and extra projected columns (e.g. upper(g)) are dropped. This is the heavy-hitter rewrite and may be intended for planning, but the result shape no longer matches the SQL.
Possible fix
SetOp: a column is nullable if it is nullable on either side.
Aggregate schema derivation: mark these results nullable.
Top-k: keep the SQL output shape (rows of the group key and the count) above the TopK, or apply the rewrite only where the output shape is preserved.
Activity