[EPIC] Simplify memory-limited (spilling) execution #26711

Describe the problem

Collects the existing issues so the special cases can be revisited together rather than one at a time.

Memory-limited execution has grown by accretion and is hard to reason about as @2010YOUY01 and @andygrove point out in #25853 (comment).

Each spilling operator (sort, aggregate, sort-merge join, nested loop join, repartition) has fixed its own out-of-memory failures with a local special case, and the shared pieces such as StreamingMergeBuilder, MultiLevelMergeBuilder and SpillManager have picked up knobs that only one caller sets.

For example, the spill merge now has three options that only aggregate spilling uses (with_replay_headroom, with_intermediate_merge_sizing and with_max_batch_bytes)

The result is hard to reason about locally

Issues

Shared spill and merge infrastructure

• Memory accounting inaccuracies in multi level merge #23760
• Add separate spill_batch_size configuration #17595
• Move the spill pre-processing steps into SpillManager #18261
• Replace record_batch.get_array_memory_size() in spilling operators #13430
• Limit the max merge degree during re-spill in external sort #16908

Aggregate

• Unify the spilling aggregate streams behind one spill-replay driver #25537
• Remove the legacy GroupedHashAggregateStream and enable_migration_aggregate #25902
• Hash aggregate with a few large groups fails after spilling: spill batches are sized by rows, not bytes #25851
• Final hash aggregate fails with ResourcesExhausted when its materialized output is larger than its table #26701
• Final and Single aggregates over fully sorted input fail with ResourcesExhausted instead of emitting completed groups #26628

Sort

• ExternalSorter memory accounting: fixes missing from branch-55 and remaining gaps on main #25804
• Clarify memory-limit guarantees for external-sort spilling #25183
• Revisit ExternalSorter sort strategy and sort_in_place_threshold_bytes #21543
• SortExec merge lookahead can retain batches without memory reservations #26411

Testing

• Improve test coverage for spilling (memory-limited) sort/aggregation/sort-merge-join #13431
• Extend aggregation fuzz testing with memory-limit tests #23658
• Run the sqllogictest corpus across a matrix of batch sizes #26700

Related discussions and epics

• [DISCUSSION] Memory accounting model discussion #16841 [DISCUSSION] Memory accounting model discussion
• [EPIC] Refactor Operators for Maintainability #26225 [EPIC] Refactor Operators for Maintainability
• [EPIC] Reduce complexity and duplication hotspots in DataFusion #26309 [EPIC] Reduce complexity and duplication hotspots in DataFusion
• [EPIC] Spilling coverage gaps vs Spark and DuckDB: window, cross join, and pool-driven reclaim #25637 [EPIC] Spilling coverage gaps vs Spark and DuckDB: window, cross join, and pool-driven reclaim
• [EPIC] Spilling Hash Join — run any join in a bounded memory budget #24768 [EPIC] Spilling Hash Join
• [EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation #13123 [EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation
• Fix Memory Backpressure/Spilling Coordination #20715 Fix Memory Backpressure/Spilling Coordination