Describe the bug
Spark orders a null element of an array key, and a null field of a struct key, below every other value, whatever the key's NULLS FIRST or NULLS LAST, which applies only to a null key itself. Comet's native sort takes the position of nested nulls from the key's null order instead. With the default null orders, ASC NULLS FIRST and DESC NULLS LAST, the two agree, but ASC NULLS LAST and DESC NULLS FIRST put a key that holds a null element at the other end from Spark.
Steps to reproduce
CREATE TABLE t(id INT, i INT) USING parquet;
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3);
SELECT id FROM t ORDER BY array(i) DESC NULLS FIRST, id;
-- Spark: 3, 1, 2 Comet: 2, 3, 1
SELECT id FROM t ORDER BY named_struct('x', i) NULLS LAST, id;
-- Spark: 2, 1, 3 Comet: 1, 3, 2
Expected behavior
A null element or field sorts below every other value, as in Spark, whatever the key's null order. The same applies to TopK and to window order keys.
Additional context
Found while fixing #5507 in #6475, whose fixture keeps the default null orders to stay clear of this.
Describe the bug
Spark orders a null element of an array key, and a null field of a struct key, below every other value, whatever the key's
NULLS FIRSTorNULLS LAST, which applies only to a null key itself. Comet's native sort takes the position of nested nulls from the key's null order instead. With the default null orders,ASC NULLS FIRSTandDESC NULLS LAST, the two agree, butASC NULLS LASTandDESC NULLS FIRSTput a key that holds a null element at the other end from Spark.Steps to reproduce
Expected behavior
A null element or field sorts below every other value, as in Spark, whatever the key's null order. The same applies to TopK and to window order keys.
Additional context
Found while fixing #5507 in #6475, whose fixture keeps the default null orders to stay clear of this.