docs: add release notes for 1.1.0 with known regressions and workarounds

docs/source/index.md

@@ -139,6 +139,7 @@ Runs your existing Spark queries on the Apache DataFusion native engine, no code
 
 User Guide <user-guide/index>
 Contributor Guide <contributor-guide/index>
+Release Notes <release-notes/index>
 Changelog <changelog/index>
 About <about/index>
 ```

docs/source/release-notes/1.1.0.md

@@ -0,0 +1,82 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet 1.1.0 Release Notes
+
+Comet 1.1.0 fixes about 90 bugs that shipped in 1.0.0, about 40 of which returned results that
+differed from Spark. Before upgrading, read the
+[upgrade guide](../user-guide/latest/migration-guide.md#upgrading-to-comet-110) for the behavior
+changes in this release, and the known regressions below.
+
+## Known Regressions
+
+After the first release candidate was cut, an audit of every pull request in 1.1.0 against 1.0.0
+found the regressions below ([#6399](https://github.com/apache/datafusion-comet/issues/6399)). Each
+one is a case that 1.0.0 handled correctly and 1.1.0 doesn't. Each entry says how to avoid the
+problem, and each setting or query change it names has been checked against the regression's
+reproducer on 1.1.0. Most of them make the affected expression or operator fall back to Spark, so
+queries that use it can run more slowly. Fixes are planned for 1.1.1, and
+[#6402](https://github.com/apache/datafusion-comet/issues/6402) has the current list, including
+the regressions that were fixed before the release.
+
+### Wrong Results
+
+- **`array_distinct` and `array_union` with `-0.0`**
+  ([#5701](https://github.com/apache/datafusion-comet/issues/5701)). On Spark versions without
+  SPARK-54918, which are 3.4, 3.5, 4.0 before 4.0.5 and 4.1 before 4.1.4, these treat `-0.0` and
+  `0.0` as the same value and return `0.0` for it, where Spark keeps them apart. To avoid it, set
+  both `spark.comet.expression.ArrayDistinct.enabled=false` and
+  `spark.comet.expression.ArrayUnion.enabled=false`, which run the projections that use them in
+  Spark.
+- **Iceberg reads of a renamed nested field**
+  ([#6546](https://github.com/apache/datafusion-comet/issues/6546)). On an Iceberg table where a
+  field inside a struct column, or inside the struct of an array, was renamed after some data files
+  were written, the native Iceberg scan can return NULL for that field in rows from the older
+  files. 1.0.0 returned the values. To avoid it, set `spark.comet.scan.icebergNative.enabled=false`,
+  which reads every Iceberg table with Spark's reader.
+
+### Errors
+
+- **Iceberg reads of a column that gained a nested field**
+  ([#6504](https://github.com/apache/datafusion-comet/issues/6504)). On an Iceberg table where a
+  struct column, or the struct inside an array or map, gained a field after some data files were
+  written, the native Iceberg scan fails with `Incorrect number of arrays for StructArray fields`
+  when it reads one of the older files. Plain reads of such a column already failed in 1.0.0. In
+  1.1.0, `IS NULL` and `IS NOT NULL` filters on the column, non-outer `explode` of such an array,
+  and joins on such a struct fail too, where 1.0.0 ran them in Spark. To avoid it, set
+  `spark.comet.scan.icebergNative.enabled=false`, which reads every Iceberg table with Spark's
+  reader.
+- **A spilled native aggregate under memory pressure**
+  ([#6254](https://github.com/apache/datafusion-comet/issues/6254)). After a native final hash
+  aggregate spills, it reads the spilled data back with no way to spill again. 1.0.0 ignored a
+  refused memory request at that step, but since the upgrade to DataFusion 55 the refusal fails the
+  task. So a task near its memory limit can fail with `Failed to acquire N bytes` in
+  `FinalHashAggregateStream`. To avoid it, give executors more off-heap memory with
+  `spark.memory.offHeap.size`. Twice the size that failed was enough in our tests, but the margin
+  depends on the workload. Setting `spark.comet.exec.aggregate.enabled=false` for the affected job
+  also avoids it, by running its aggregates in Spark.
+
+### Performance
+
+- **Partial aggregates without the Comet shuffle manager.** When the final aggregate runs in Spark,
+  which happens for every aggregate if the Comet shuffle manager isn't installed, the partial `avg`,
+  decimal `sum`, `stddev`, `variance`, `corr`, `first`, `last` and a few others now run in Spark
+  too. In 1.0.0 they ran natively, and `avg` could return NULL in this plan
+  ([#5419](https://github.com/apache/datafusion-comet/issues/5419)). Installing the Comet shuffle
+  manager keeps them native.

docs/source/release-notes/index.md

@@ -0,0 +1,31 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Release Notes
+
+The release notes describe what users of each Comet release should know that the change log and the
+[upgrade guide](../user-guide/latest/migration-guide.md) don't cover, such as known regressions and
+the settings that avoid them. The [change log](../changelog/index.md) lists every pull request in a
+release.
+
+```{toctree}
+:maxdepth: 1
+
+1.1.0 <1.1.0>
+```

docs/source/user-guide/latest/migration-guide.md

@@ -62,6 +62,9 @@ key is removed.
 Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below
 need none either, but check whether any of them applies to your deployment.
 
+Comet `1.1.0` also has known regressions, each with a setting that avoids it. They are listed in
+the [1.1.0 release notes](../../release-notes/1.1.0.md).
+
 Comet `1.1.0` requires JDK 17 or later. JDK 11 is no longer supported. See
 [Installing Comet](installation.md) for the supported Java, Scala, and Spark versions.
 
@@ -124,6 +127,19 @@ session's `spark.shuffle.manager`. A session that named `CometShuffleManager` af
 had started with a different shuffle manager used to plan Comet shuffles that failed with a
 `ClassCastException`. Such a session now runs without Comet, with a warning.
 
+### Native Iceberg Reads on EKS with IRSA
+
+On EKS with IAM Roles for Service Accounts (IRSA), when `AWS_WEB_IDENTITY_TOKEN_FILE`,
+`AWS_ROLE_ARN` and a region are set and the catalog configures no credentials, Comet `1.1.0`'s
+native Iceberg scan takes its S3 credentials only from the web-identity role. If that fails, it no
+longer falls back to the node role or Pod Identity, as Comet `1.0.0` did. So a cluster whose IRSA
+setup is broken, and that was reading S3 as the node role without anyone noticing, now fails native
+Iceberg reads with `failed to load signing credential`. The "EKS / IRSA" section of
+[S3 Credential Providers](s3-credential-providers.md) explains the change. To go back to the old
+credential chain for a catalog, set
+`spark.sql.catalog.<catalog>.s3.comet.credential.webIdentity.enabled=false`. A table loaded by path
+has no catalog to set that on, so for it set `spark.comet.scan.icebergNative.enabled=false`.
+
 ### Deprecated and Removed Settings
 
 `spark.comet.exec.memoryPool.fraction` is deprecated and will be removed in a future major release.