Repository navigation
fix(substrait): dedupe repeated GROUP BY keys in the producer - #27073
Open
jonasHanhan wants to merge 2 commits into
Open
jonasHanhan wants to merge 2 commits into
jonasHanhan wants to merge 2 commits into
Conversation
DataFusion's Aggregate schema keeps each grouping key once, but the Substrait producer was emitting a repeated expression_reference for GROUP BY k, k. The consumer then expanded those refs into extra columns, so projections built from the DataFusion plan indexed the wrong fields and a round trip swapped the grouping and measure columns. Skip duplicate grouping expressions within each grouping so the AggregateRel output layout matches the DataFusion schema. Closes apache#27058.
bvolpato
approved these changes
Nov 6, 2026
bvolpato
left a comment
Contributor
There was a problem hiding this comment.
LGTM. The per-group deduplication matches the ordering used by Aggregate's schema, and the regression checks the unoptimized plan where the duplicate key is still present. I ran the Substrait crate tests at this head, including the round-trip regression.
Could you also enable Actions on your fork and push again? Rust (fork) currently reports no Rust workflow run for this commit, so CI still needs to run before merge. Thanks for fixing this.
Author
|
Thanks for the review! I'll enable Actions on the fork and push again so Rust (fork) can run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
GROUP BY k, kgives a plan whose projection reads the aggregate's columns shifted by one #27058.Rationale for this change
GROUP BY k, kis legal SQL and does not change the result, but producing Substrait from that plan and consuming it again swapped the grouping and measure columns.DataFusion's
Aggregateschema keeps each grouping key once (grouping_set_to_exprlist). The producer already shared identical expressions ingrouping_expressions, but it still wrote a repeatedexpression_referencefor each occurrence ([0, 0]). The consumer expands each reference into its own output column, so the AggregateRel effectively gained an extra column. Projections built from the DataFusion plan still indexed fields as if that column were not there, and a round trip returned(sum, k)under the names(k, s).What changes are included in this PR?
In
parse_flat_grouping_exprs, skip grouping expressions that are already present in the current grouping. The AggregateRel then declares and references the repeated key once, matching the DataFusion Aggregate schema that projections are built against.What is the testing strategy for this PR?
Added
aggregate_duplicate_group_by_key_roundtripinroundtrip_logical_plan.rs. It uses an unoptimized plan soeliminate_duplicated_exprdoes not hide the bug, checks that the producer emits a single grouping expression reference, and asserts that collecting the plan before and after a Substrait round trip returns the same batches. It also covers the optimized path where the duplicate key is already removed.Locally:
cargo test -p datafusion-substraitpasses, andcargo clippy -p datafusion-substrait --all-targets -- -D warningsandcargo fmtare clean.Are there any user-facing changes?
Substrait plans produced from logical plans that still contain repeated
GROUP BYkeys (for example unoptimized plans) now round-trip with the correct column order. No API changes.