Skip to content

IR: set-op and aggregate nullability, and the SQL top-k rewrite's output shape #657

Open

Description

@zzylol

Found by the DataFusion differential check of #539 (573 SQL queries run through DataFusion 54 and through the lowering). These are in the IR's schema derivation and canonicalization (already on main), not in the SQL frontend.

Problem

  • Set operations take the left schema. SetOp derives its schema from left only (ir/non_asap.rs), so SELECT 1 AS x UNION ALL SELECT NULL is declared non-null but returns NULL. DataFusion declares it nullable. (feat(sql): lower SQL to unified operator and scalar IR #539 fixes the same issue for Concat, which its ROLLUP / CUBE / GROUPING SETS lowering uses.)
  • Some aggregate results are declared non-null. stddev / variance / median / percentile results, and ClickHouse argMax / argMin, are NULL over empty input (e.g. stddev(a) … WHERE id = 1 with one row, or an empty group), but the aggregate schema derivation (pre_asap/agg_intent.rs, around the result-field construction) marks them non-null. The nullability fix in non_asap.rs covers only Sum / Avg / Min / Max.
  • The top-k rewrite changes SQL output. canonicalize turns SELECT g, count(*) AS c … GROUP BY g ORDER BY c DESC LIMIT k into a one-column TopK aggregate instead of rows of (g, c), and extra projected columns (e.g. upper(g)) are dropped. This is the heavy-hitter rewrite and may be intended for planning, but the result shape no longer matches the SQL.

Possible fix

  • SetOp: a column is nullable if it is nullable on either side.
  • Aggregate schema derivation: mark these results nullable.
  • Top-k: keep the SQL output shape (rows of the group key and the count) above the TopK, or apply the rewrite only where the output shape is preserved.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions