Repository navigation
Release DataFusion 56.0.0 (Oct/Nov 2026) #24461
Description
Is your feature request related to a problem or challenge?
Previous release is #22393 (Jul / Aug 2026) so next major release would be approximately Sep / Oct 2026.
Steps:
- Create a
branch-56branch: https://github.com/apache/datafusion/tree/branch-56 - Update version and changelog: [branch-56] Prepare for 56.0.0 release: Version and Changelog #27098
- Test with DataFusion Python
- Test with DataFusion Comet - deps: upgrade to DataFusion 56.0.0 datafusion-comet#7249
- Test with DataFusion Ballista
- Test with DataFusion Federated - Update DataFusion to 56.0.0 and Arrow to 60.1 datafusion-contrib/datafusion-federation#191 (the build passes locally)
- Test with DataFusion Table Providers - Update DataFusion to 56.0 and Arrow to 60.1 datafusion-contrib/datafusion-table-providers#773 (the build passes locally)
- Test with delta.rs
- Test vortex - [do not merge] feat: prepare DataFusion 56 upgrade vortex-data/vortex#10309
- Test with iceberg-rust
- Test with LakeSail
- Test with parquet viewer
- Test with datafusion-materialized-views: Upgrade to DataFusion branch-56 datafusion-contrib/datafusion-materialized-views#124
- Voting Thread
- Publish to crates.io
- Create ticket for next release: Release DataFusion
56.1.0(minor/patch) Release (Nov 2026) #24463 / Release DataFusion57.0.0(Dec 2026 / Jan 2027) #25901 - Create blog post: Blog post for the DataFusion
56.0.0release #27201
Features to mention in the blog (if they make it)
-
ASOFjoins from @Xuanwo : ASOF join support / Specialize Range Joins #318 - Push down topk through join #21621
- Add write support to Avro format #24925 from @pepijnve
- Add support for SQL based registration of external catalogs #25111 from @pepijnve
- Versioned documentation #17071
- Enable TopK per Partition (
datafusion.optimizer.enable_window_topn) by default #24608 - [EPIC] A Collection of Wrong Results Bugs - Q4 2026 #26672 from @alamb
- Add support for reading/writing ALP data
Bugs/issues that should be fixed
Before branch cut
- Side effects from
serde_json/preserve_orderfeature impact user applications #25883 @adriangb - [dbolympics] fix: cast outer projection expr to declared field type in SingleDistinctToGroupBy #26211 from @jayzhan211
- Rename
LogicalPlan::DescribeTabletoLogicalPlan::Showmove statement implementations out ofsqlmodule #26057 from @pepijnve (to reduce API churn) - refactor: register proto decoders with
T::register_protoinstead of a free function #26565 from @adriangb (so it is not a breaking change later) -
DISTINCT ONwith nested payloads can fail memory reservation after spilling #26744 from @shehabgamin - Regression:
ORDER BY col, (scalar subquery) LIMITover aRIGHT JOINfails inSanityCheckPlan#26572
After branch cut (requires backport to branch-56)
- TPC-H Q18 regression #27108 from @devanbenz (regression; revert feat: estimate Decimal filter selectivity with interval analysis #25790); Backport PR: [branch-56] Revert #25790 (Decimal filter selectivity) due to TPC-H Q18 regression (#27145) #27147
- Reusing a
limit=1FTS plan concurrently can return an empty result #27130 from @alamb (regression, reported by @Xuanwo); Backport PR: [branch-56] fix: give each SortExec reset_state copy its own TopK dynamic filter (backport #27142) #27174 (@raghavyadav01) - REGRESSION: Multi-key null-aware join silently defaults to one NOT IN value key #26569 from @jayzhan211 (regression); Backport PR: [branch-56] fix: reject a multi-key null-aware join without a NOT IN value key count (backport #27111) #27176
- SortExec panics in sort_run on FFI-imported sliced batches: assertion reservation.size() + shared.size() >= reserved_bytes #27069 from @andygrove (regression); Backport PR: [branch-56] fix: count the largest capacity seen for a shared buffer address (backport #27097) #27175
- REGRESSION: UNION: DataFrame::schema() reports the first input's type, but execution returns the coerced type #27023 from @mikewolfd (regression); Backport PR: [branch-56] fix: infer UNION output types before analysis (backport #27139) #27178
- REGRESSION: SanityCheckPlan error with sort-merge join when a filter makes a sort or join key constant (constant lost in
EquivalenceGroupprojection) #26624 from @alpaylan (regression); Fix PR: fix: preserve constant equivalence properties through projection #27123; Backport PR: [branch-56] fix: preserve constant equivalence properties through projection (backport #27123) #27179 - Regression: short floating-point IN lists mishandle signed zero #27121 from @lonless9 (regression); Fix PR: IN LIST: treat signed zeros as equal #25186 (@geoffreyclaude); Backport PR: [branch-56] IN LIST: treat signed zeros as equal (backport #25186) #27204
- REGRESSION: Window ordered by (total DESC, k) fails SanityCheckPlan when k is a function of a filtered constant column #26762 from @adriangb (regression); Fix PR: fix: skip a leading per-partition constant when finding ordering heads #27140
- ExternalSorter memory accounting: fixes missing from branch-55 and remaining gaps on main #25804 (finding 9) from @andygrove (reported by @Xuanwo); Fix PR: fix: leave memory for others when a spilled sort merges its runs #26264
- Incorrect regex-to-LIKE simplification for literal backslashes #27120 from @lonless9
- REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193) #27065 from @andygrove (regression); Fix PR: fix: keep nested Parquet binary fields out of the binary-to-string coercion #27067
Community Wishlist
Before branch cut
- Further improve performance of IN list evaluation #19241
- datafusion-physical-expr: add
DynamicFilterPhysicalExpr::update_from(Self)method #26534 @jayshrivastava - dev: Allow new release version to be automatically updated #26007 from @nuno-faria
- docs: add version picker and manual release snapshots #24792 from @yiaany
- fix: keep field metadata in the result of min and max #26939 from @cblp (requested by @alamb)
- fix: keep extension types when skip_metadata clears Parquet field metadata #26854 from @cblp (requested by @alamb)
After branch cut (requires backport to branch-56)
- feat: show extension type names in DESCRIBE and information_schema.columns #26893 from @cblp (requested by @alamb); Backport PR: [branch-56] feat: show extension type names in DESCRIBE and information_schema.columns (backport #26893) #27057
- perf: prune oversized dictionaries before IPC spilling #27015 from @brancz; Backport PR: [branch-56] perf: prune oversized dictionaries before IPC spilling (backport #27015) #27107
- feat: support ALP parquet encoding in writer options #27036 from @Surajsuthar (requested by @alamb); Backport PR: [branch-56] feat: support ALP parquet encoding in writer options (backport #27036) #27093
- fix: read Hive's null partition placeholder as NULL #26310 f4om @xanderbailey; Backport PR: [branch-56] fix: read Hive's null partition placeholder as NULL (backport #26310) #27171
- Avoid eager merging of expanding-key ingestion spills #27043 from @brancz
- feat: store start/end bounds in
Partitioning::Range#26593 @jayshrivastava
Reacted by Adam Gutglick, Jens Holdgaard Pedersen, Nathan, Kosta Tarasov, Kumar Ujjawal, Matt Butrovich and Namgung Chan
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Release DataFusion `56.0.0` (Sep / Oct 2026)[/-][+]Release DataFusion `56.0.0` (Oct 2026)[/+]xudong963 commentedon Aug 26, 2026
I can help with this release!
Xuanwo commentedon Sep 9, 2026
Really hope #318 can deliever in v56.0, let's rock!
alamb commentedon Sep 9, 2026
318 -- perhaps the lowest issue number left open that tracks something useful. A long standing request
Xuanwo commentedon Sep 14, 2026
One more question: what testing needs to be done for releases? Perhaps we could also include
lance. I'm willing to take on all the related work.alamb commentedon Sep 15, 2026
Thanks @Xuanwo -- I think what we typically do is to use the release ticket to coordinate testing with downstream crates that wish to help as we get close to making a release candidate. So for example, @AdamGS often tests with Vortex as part of that process.
Here is an example in the 55 release ticket where we were disucssing it
55.0.0(Jul / Aug 2026) #22393 (comment)alamb commentedon Sep 15, 2026
Since this release will have an update to arrow/parquet to version 60, perhaps this would be a good candidate to try out "LTS" / Long term support: #16622
kumarUjjawal commentedon Sep 15, 2026
Datafusion 56.0 will be in a good stable condition for downstream consumers and should be good starting point for new projects. So LTS makes sense.
alamb commentedon Sep 16, 2026
FYI @hareshkh who was asking about a more stable branch / backports in the call today
[-]Release DataFusion `56.0.0` (Oct 2026)[/-][+]Release DataFusion `56.0.0` (Oct/Nov 2026)[/+]alamb commentedon Sep 28, 2026
Adjusting date from Oct --> Oct/Nov as we are also working on a 55.2:
55.2.0(patch) Release (Oct 2026) #25758adriangb commentedon Sep 30, 2026
I would like to land a fix for #25883 before next release
[branch-55] fix: frame IPC messages when reading spill files so batch…
jayshrivastava commentedon Oct 22, 2026
👋🏽 Flagging a couple changes I would like to see in df56 :)
DynamicFilterPhysicalExpr::update_from(Self)method #26534Partitioning::Range#2659359 remaining items
raghavyadav01 commentedon Nov 6, 2026
Backport PR for #27130 (the TopK
reset_stateregression, fixed on main by #27142): #27174. It's a clean cherry-pick; sort tests and sqllogictest pass onbranch-56.[branch-56] fix: give each SortExec reset_state copy its own TopK dyn…
alamb commentedon Nov 6, 2026
I spent some more time today working/ triaging / backporting. Thank you @raghavyadav01 and @xudong963 for the help so far
Here is a list of issues that don't have a PR ready for review
Here is a list of open fix PRs waiting to merge:
Here is a list of open backport PRs:
Here are issues I suggest we DO NOT treat as release blockers (see individual issues for rationale)
[branch-56] fix: reject a multi-key null-aware join without a NOT IN …
[branch-56] fix: infer UNION output types before analysis (backport #…
fix: preserve constant equivalence properties through projection (apa…
[branch-56] fix: preserve constant equivalence properties through pro…
shehabgamin commentedon Nov 7, 2026
+1 for Sail
shehabgamin commentedon Nov 7, 2026
Probably too late, but figured I'd share just in case
#27207
[branch-56] fix: count the largest capacity seen for a shared buffer …
[branch-56] Prepare for 56.0.0 release: Version and Changelog (#27098)
jayzhan211 commentedon Nov 8, 2026
#27219:
ORDER BY ... LIMITabove aWITH ... AS MATERIALIZEDCTE can return wrong rows (none, in the repro), because theTopKAggregationrule limits the CTE body. The feature (#25580) is onbranch-56, and the repro gives the same wrong rows on abranch-56build. Mentioning it here so it's visible while 56.0.0 is being prepared.alamb commentedon Nov 8, 2026
Yeah I don't know why it hasn't merged yet.
andygrove commentedon Nov 8, 2026
It is in the merge queue now. I will create backport PR once it has merged.
andygrove commentedon Nov 8, 2026
back-port PR #27243