Repository navigation
fix: apply SingleDistinctToGroupBy to aggregates that carry an alias - #27080
Open
fangchenli wants to merge 3 commits into
Open
fangchenli wants to merge 3 commits into
fangchenli wants to merge 3 commits into
Conversation
The rule only matched a bare Expr::AggregateFunction, so an alias directly on the aggregate, as the DataFrame API produces, turned the rewrite off and gave the same query a different plan than SQL. Look through aliases when matching and strip them in the rewrite; the outer projection restores output names from the schema. Also correct the can_strip_every_distinct doc comment, which said this rule does not rewrite aliased aggregates. Its behavior is unchanged. Closes apache#23401 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fangchenli
marked this pull request as draft
November 4, 2026 06:31
Stripping aliases from the rewritten outer aggregates turned aggregates that differed only in their alias, such as `count_distinct(b)` aliased as `n1` and `n2`, into the same `count(alias1)`. `Aggregate` rejected the repeated output name, so optimization failed with a duplicate field error on queries that succeed without the rewrite. Compute each outer aggregate once and let the projection select it under every original name, which is the plan SQL already produces. The cast check in that projection read types by position, so it now maps each original aggregate to its deduplicated position. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fangchenli
marked this pull request as ready for review
November 5, 2026 06:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
single_distinct_to_groupbynot applied to DataFrame API count(DISTINCT) #23401.This picks up #23403, which the stale bot closed. It has been reworked on current
main; see the note on the constant-argument guard below.Rationale for this change
The same query gets a different plan depending on how it is written:
SingleDistinctToGroupByonly matches a bareExpr::AggregateFunction, so an alias on the aggregate turns it off. DataFrame users therefore miss the rewrite that SQL users get, including the parallelism it provides.Measured on 4M rows and 2,000 groups, using the DataFrame API (median of 3 rounds, each the median of 9 runs). Peak RSS includes the in-memory table:
count(DISTINCT int)sum(DISTINCT int)count(DISTINCT varchar)The main point is that both APIs now reach the same plan. The result is not always an improvement:
count(DISTINCT int)has a specialized accumulator and gains nothing, andsum(DISTINCT int)trades memory for speed, which is the tradeoff described in #25932. Any policy on when the rewrite should apply, such as #26000, is orthogonal and would now apply to both APIs equally.What changes are included in this PR?
is_single_distinct_agglooks through aliases on the aggregate, and the rewrite strips them. Output names are unaffected because the outer projection restores them from the schema.count_distinct(b)aliased as bothn1andn2. AnAggregatecannot output a name twice, so the rewrite computes each one once and the outer projection selects it under every original name. SQL already produces this plan for the same query.can_strip_every_distinctsaid this rule does not rewrite an aliased aggregate. That is no longer true, so the comment now gives the reason that still applies. Its behavior does not change.An earlier version of this PR also skipped the rewrite for constant
DISTINCTarguments, because the rewrite exposed a wrong result on empty input inEliminateGroupByConstant. #22132 has since fixed that, so the guard is gone.What is the testing strategy for this PR?
aliased_single_distinct_aggregate_is_rewritten, checks the optimized plan and the results end to end.single_distinct_to_groupby.rscover aliased aggregates with and withoutGROUP BY, nested aliases, partly aliased aggregates, the case that must stay unrewritten (two distinct fields), and thecountrollup path.identical_aliased_single_distinct_aggregates_are_computed_once, covers identical aggregates with different aliases, checking the plan and results end to end. Unit tests cover the same case, including one where only some of the duplicates are aliased, and confirm non-distinct duplicates still work.Are there any user-facing changes?
DataFrame queries with an aliased single
DISTINCTaggregate now get the same plan as the equivalent SQL, so their performance and memory use can change as shown above. Results and output names do not change. There are no public API changes.🤖 Generated with Claude Code