Describe the bug
When an array or struct ORDER BY key holds a null element or field, an aggregate window function over the default RANGE frame returns the result for the whole partition on that row and on every row after it. DataFusion finds the end of a RANGE frame by comparing keys with ScalarValue::partial_cmp, which orders a null element above every other value (Postgres semantics), while the sort put that key first, as Spark does. Once the frame search passes that row, every later row's frame runs to the end of the partition.
RANK and DENSE_RANK over the same keys are correct.
Steps to reproduce
CREATE TABLE r(id INT, p INT, i INT) USING parquet;
INSERT INTO r VALUES (1, 1, NULL), (2, 1, 1), (3, 1, 1), (4, 1, 2);
SELECT id, SUM(id) OVER (ORDER BY array(i)) AS running FROM r;
-- Spark: (1, 1), (2, 6), (3, 6), (4, 10) Comet: 10 on every row
SELECT id, SUM(id) OVER (PARTITION BY p ORDER BY named_struct('x', i)) AS running FROM r;
-- Spark: (1, 1), (2, 6), (3, 6), (4, 10) Comet: 10 on every row
Expected behavior
The frame of each row ends at its last peer, as in Spark, so the running sums are 1, 6, 6 and 10.
Additional context
Found while fixing #5507 in #6475, whose fixture leaves the null row out of its running sums to stay clear of this.
Describe the bug
When an array or struct
ORDER BYkey holds a null element or field, an aggregate window function over the defaultRANGEframe returns the result for the whole partition on that row and on every row after it. DataFusion finds the end of aRANGEframe by comparing keys withScalarValue::partial_cmp, which orders a null element above every other value (Postgres semantics), while the sort put that key first, as Spark does. Once the frame search passes that row, every later row's frame runs to the end of the partition.RANKandDENSE_RANKover the same keys are correct.Steps to reproduce
Expected behavior
The frame of each row ends at its last peer, as in Spark, so the running sums are 1, 6, 6 and 10.
Additional context
Found while fixing #5507 in #6475, whose fixture leaves the null row out of its running sums to stay clear of this.