CometSort sorts a multi-column key containing a collated string by raw bytes

[andygrove]

Describe the bug

QueryPlanSerde.supportedSortType only type-checks a sort that has a single sort order. A multi-column sort whose key includes a string with a non-default collation (Spark 4.0+) therefore converts to CometSortExec, which sorts it by raw bytes rather than by the collation.

The collation check in CometShuffleExchangeExec.columnarShuffleFailureReasons keeps the stage off Comet for hash and range partitioning, which #6110 found is the only thing stopping this there. SinglePartition and round-robin exchanges have no such check, so the sort still reaches native.

Steps to reproduce

On main at cccc08b, default Spark 4.1 profile:

CREATE TABLE repro_coll_sort (_1 INT, _2 STRING) USING parquet;
INSERT INTO repro_coll_sort VALUES (1, 'b'), (2, 'A'), (3, 'a'), (4, 'B');

SELECT _1, row_number() OVER (ORDER BY c, _1) AS rn
FROM (SELECT _1, _2 COLLATE UTF8_LCASE AS c FROM repro_coll_sort);

The executed plan has CometWindowExec over CometSort [c ASC NULLS FIRST, _1 ASC NULLS FIRST] over a SinglePartition CometColumnarExchange, and the row numbers differ:

_1	Spark rn	Comet rn
1	3	4
2	1	1
3	2	3
4	4	2

Comet orders the rows A, B, a, b, which is byte order. Spark orders them A, a, b, B, which is UTF8_LCASE order with _1 breaking the ties.

Expected behavior

CometSortExec should decline a sort key that contains a non-UTF8_BINARY collated string at any position or nesting depth, or route it through the codegen dispatcher, so the result matches Spark.

Additional context

Found while reviewing #6110. There, removing the shuffle collation check makes listagg(DISTINCT ...) under utf8_lcase return aabb instead of ab, because CometSort stays on a two-column collated key. The #5302 author reported the same limitation, including the window order-spec variant.

[LinSimon-901101]

take