fix: fall back when struct field names collide case-insensitively

[comphead]

Which issue does this PR close?

Closes #6136.

Rationale for this change

Under spark.sql.caseSensitive=false, Spark's Parquet reader (ParquetReadSupport.clipParquetGroupFields) matches each requested field to the file fields sharing its toLowerCase(Locale.ROOT) name, and raises Found duplicate field(s) "x": [x, X] in case-insensitive mode when more than one answers. The native scan returned rows instead.

Comet resolves nested names only while casting a column whose file type differs from the requested type. check_conversion and CometCastColumnExpr both go through match_struct_fields. When the two types are equal, nothing resolves the names:

• DataFusion's DefaultPhysicalExprAdapter leaves a column as a bare Column whenever its logical and physical Fields compare equal.
• The Parquet opener skips the adapter altogether when the file schema equals the logical schema and no predicate is pushed. That is the metadata-free case the issue describes.

So s struct<x, X> read from a file with the same shape came back positionally.

One correction to the issue: Spark's analyzer does check nested names. On 3.4 through 4.1, DataSource.resolveRelation runs SchemaUtils.checkSchemaColumnNameDuplication recursively under the session resolver, so spark.read.schema("s struct<x: bigint, X: bigint>") fails with COLUMN_ALREADY_EXISTS under the case-insensitive resolver. The shape still reaches the scan when a DataFrame is analyzed under the case-sensitive resolver and planned under the case-insensitive one. The same flip on top-level columns never reaches Comet, because FileSourceStrategy fails with AMBIGUOUS_REFERENCE while resolving the relation's output.

The colliding names are in the requested schema, so this is decidable from the plan, as #6004 did for repeated field ids.

What changes are included in this PR?

1. DataTypeSupport.hasCaseInsensitiveDuplicateFieldNames is true when two sibling fields anywhere in a type fold to one name under toLowerCase(Locale.ROOT), the fold Spark's reader uses. It walks struct children at every depth, including array elements and map keys and values.
2. CometNativeScan.isSupported declines the scan when spark.sql.caseSensitive=false and the required schema trips that predicate, so Spark's reader resolves it and reports the ambiguity. Only the V1 native scan is gated. The Iceberg scan resolves by field id and is unchanged.
3. The Parquet compatibility guide lists the new fallback.

Like #6004, the check reads the requested schema, not the file. It can decline a read Spark would accept, such as a file without the colliding sibling, or siblings that resolve by field id. That costs native execution but not correctness, and such schemas only reach the scan past the analyzer check above. No Parquet decoding changes.

The rest of this family was already covered. Byte-identical duplicate names (#5866) and repeated field ids (#6004) in the requested schema are declined at planning time. A duplicate present only in the file makes the file type differ from the requested type, so the native checks run.

How are these changes tested?

• CometNativeReaderSuite: end to end, with the issue's reproduction. A file written without key-value metadata, with schema s struct<x, X>, is read back with that same schema. The DataFrame is analyzed under the case-sensitive resolver, then planned and run under the case-insensitive one. The scan carries the new fallback reason and raises Spark's Found duplicate field(s) "x": [x, X] in case-insensitive mode. The case-sensitive side is already covered by the existing test duplicate Parquet field names - distinct siblings and repeated names in separate groups, where a struct<dup, Dup> read stays native and matches Spark.
• CometScanRuleSuite: the predicate detects a collision at the root, in a nested struct, in array elements, and in map keys and values. It does not flag a parent and child, or cousins, that share a folded name.

Neither test can be a SQL file test. SQL file configs apply to the whole file, a catalog table read under the case-insensitive resolver fails analysis on both engines, and the fixture needs a file written without key-value metadata.

[sunchao]

Summary

• Prior state and problem: Matching file and requested schemas could bypass native name resolution, returning nested x/X fields where Spark raises an ambiguity error.
• Design approach: Decline V1 native scans when the required schema contains case-insensitive sibling-name collisions and spark.sql.caseSensitive=false.
• Correctness / compatibility analysis: Verified Spark’s Locale.ROOT folding, recursive schema checks and field-ID precedence against sources for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0. The fallback preserves Spark’s behavior.
• Key design decisions: Checking the requested schema avoids file inspection. It conservatively falls back even when field IDs or an absent physical sibling would permit reading. Case-sensitive scans and Iceberg remain unaffected.
• Implementation sketch: A recursive DataTypeSupport helper traverses structs, arrays and maps. CometNativeScan.isSupported reuses existing fallback tagging. Regression tests and compatibility documentation cover the change. This adds schema traversal during planning without adding per-row work or a new abstraction.
• Behavioral changes worth calling out: A DataFrame analyzed case-sensitively and then planned case-insensitively now falls back so Spark reports the duplicate-field error.
• Suggested improvements: No introduced P1/P2 issues found within this review.

Reviewed the entire five-file diff from base 65a0cda1cf62877a37ec1a1f5f9ebc9dd1405ebd to head a2b114d7316eb3546b9fde74f184d0de88cc6c1b. The PR is not a draft. Routed skills: review-comet-pr and review-comet-expression-pr. Snapshot and live checks showed no existing reviews, comments or review threads.

Exact-head CI: 8 checks passed, 9 build/test checks were queued and 13 checks were skipped, including Spark SQL suites. No failures were reported, but CI validation remains pending.

Validation: Compiled exact-head DataTypeSupport.scala and passed 30 schema cases across ROOT/Turkish default locales, plus duplicate-name, analyzer, field-ID and missing-sibling checks. A separate Spark 4.1.3 metadata-free Parquet reproduction passed. Full Comet/native regression suites and Spark SQL suites were not run locally. No built native library was available, so end-to-end Comet execution remains unverified.

[andygrove]

I ran the new tests on 3.4, 3.5, 4.1 and 4.2 and they pass, and the end-to-end test fails with the gate removed. Nice fix.

[andygrove]

The comment says a DataFrame analyzed under the case-sensitive resolver is how this schema gets past the analyzer, but plain SQL reaches here too. Spark caches the resolved relation for a catalog table, and later reads reuse it without running checkSchemaColumnNameDuplication again. So CREATE TABLE t (id bigint, s struct<x: bigint, X: bigint>) USING parquet, one SELECT * FROM t with spark.sql.caseSensitive=true, and then the same query under the default all land on this gate. A temp view created under the case-sensitive resolver does as well. The gate handles both correctly, so this is only about the comment. Could it mention the cached-table path? It's plain SQL, so it's the more likely way in, and whoever eventually swaps this gate for a native check will want to test it.

[comphead]

Thanks @sunchao @andygrove