Skip to content

RANGE window frames over an array or struct key with a null element span the whole partition #6477

Description

@andygrove

Describe the bug

When an array or struct ORDER BY key holds a null element or field, an aggregate window function over the default RANGE frame returns the result for the whole partition on that row and on every row after it. DataFusion finds the end of a RANGE frame by comparing keys with ScalarValue::partial_cmp, which orders a null element above every other value (Postgres semantics), while the sort put that key first, as Spark does. Once the frame search passes that row, every later row's frame runs to the end of the partition.

RANK and DENSE_RANK over the same keys are correct.

Steps to reproduce

CREATE TABLE r(id INT, p INT, i INT) USING parquet;
INSERT INTO r VALUES (1, 1, NULL), (2, 1, 1), (3, 1, 1), (4, 1, 2);

SELECT id, SUM(id) OVER (ORDER BY array(i)) AS running FROM r;
-- Spark: (1, 1), (2, 6), (3, 6), (4, 10)   Comet: 10 on every row

SELECT id, SUM(id) OVER (PARTITION BY p ORDER BY named_struct('x', i)) AS running FROM r;
-- Spark: (1, 1), (2, 6), (3, 6), (4, 10)   Comet: 10 on every row

Expected behavior

The frame of each row ends at its last peer, as in Spark, so the running sums are 1, 6, 6 and 10.

Additional context

Found while fixing #5507 in #6475, whose fixture leaves the null row out of its running sums to stay clear of this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions