Skip to content

fix(substrait): dedupe repeated GROUP BY keys in the producer - #27073

Open
jonasHanhan wants to merge 2 commits into
apache:mainfrom
jonasHanhan:fix-substrait-duplicate-groupby
Open

jonasHanhan wants to merge 2 commits into
apache:mainfrom
jonasHanhan:fix-substrait-duplicate-groupby

Conversation

@jonasHanhan

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

GROUP BY k, k is legal SQL and does not change the result, but producing Substrait from that plan and consuming it again swapped the grouping and measure columns.

DataFusion's Aggregate schema keeps each grouping key once (grouping_set_to_exprlist). The producer already shared identical expressions in grouping_expressions, but it still wrote a repeated expression_reference for each occurrence ([0, 0]). The consumer expands each reference into its own output column, so the AggregateRel effectively gained an extra column. Projections built from the DataFusion plan still indexed fields as if that column were not there, and a round trip returned (sum, k) under the names (k, s).

What changes are included in this PR?

In parse_flat_grouping_exprs, skip grouping expressions that are already present in the current grouping. The AggregateRel then declares and references the repeated key once, matching the DataFusion Aggregate schema that projections are built against.

What is the testing strategy for this PR?

Added aggregate_duplicate_group_by_key_roundtrip in roundtrip_logical_plan.rs. It uses an unoptimized plan so eliminate_duplicated_expr does not hide the bug, checks that the producer emits a single grouping expression reference, and asserts that collecting the plan before and after a Substrait round trip returns the same batches. It also covers the optimized path where the duplicate key is already removed.

Locally: cargo test -p datafusion-substrait passes, and cargo clippy -p datafusion-substrait --all-targets -- -D warnings and cargo fmt are clean.

Are there any user-facing changes?

Substrait plans produced from logical plans that still contain repeated GROUP BY keys (for example unoptimized plans) now round-trip with the correct column order. No API changes.

DataFusion's Aggregate schema keeps each grouping key once, but the
Substrait producer was emitting a repeated expression_reference for
GROUP BY k, k. The consumer then expanded those refs into extra columns,
so projections built from the DataFusion plan indexed the wrong fields
and a round trip swapped the grouping and measure columns.

Skip duplicate grouping expressions within each grouping so the
AggregateRel output layout matches the DataFusion schema.

Closes apache#27058.
@github-actions github-actions Bot added the substrait Changes to the substrait crate label Nov 4, 2026

@bvolpato bvolpato left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The per-group deduplication matches the ordering used by Aggregate's schema, and the regression checks the unoptimized plan where the duplicate key is still present. I ran the Substrait crate tests at this head, including the round-trip regression.

Could you also enable Actions on your fork and push again? Rust (fork) currently reports no Rust workflow run for this commit, so CI still needs to run before merge. Thanks for fixing this.

@jonasHanhan

Copy link
Copy Markdown
Author

Thanks for the review! I'll enable Actions on the fork and push again so Rust (fork) can run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait producer: GROUP BY k, k gives a plan whose projection reads the aggregate's columns shifted by one

2 participants