Skip to content

Native sort orders null elements of array and struct keys by the key's null order, unlike Spark #6476

Description

@andygrove

Describe the bug

Spark orders a null element of an array key, and a null field of a struct key, below every other value, whatever the key's NULLS FIRST or NULLS LAST, which applies only to a null key itself. Comet's native sort takes the position of nested nulls from the key's null order instead. With the default null orders, ASC NULLS FIRST and DESC NULLS LAST, the two agree, but ASC NULLS LAST and DESC NULLS FIRST put a key that holds a null element at the other end from Spark.

Steps to reproduce

CREATE TABLE t(id INT, i INT) USING parquet;
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3);

SELECT id FROM t ORDER BY array(i) DESC NULLS FIRST, id;
-- Spark: 3, 1, 2   Comet: 2, 3, 1

SELECT id FROM t ORDER BY named_struct('x', i) NULLS LAST, id;
-- Spark: 2, 1, 3   Comet: 1, 3, 2

Expected behavior

A null element or field sorts below every other value, as in Spark, whatever the key's null order. The same applies to TopK and to window order keys.

Additional context

Found while fixing #5507 in #6475, whose fixture keeps the default null orders to stay clear of this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions