Skip to content

fix: preserve constant equivalence properties through projection - #27123

Merged
alamb merged 6 commits into
apache:mainfrom
kumarUjjawal:kumarUjjawal/datafusion-26624-fix-plan
Nov 6, 2026
Merged

alamb merged 6 commits into
apache:mainfrom
kumarUjjawal:kumarUjjawal/datafusion-26624-fix-plan

Conversation

@kumarUjjawal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #26624.

Rationale for this change

With prefer_hash_join = false, a query can fail physical plan validation when a filter makes a computed sort or join key constant. Embedding a projection into the sort-merge join loses the constant property after sorting has already relied on it.

What changes are included in this PR?

Preserve each projected equivalence class's constant metadata. A projected member has the same value as its source member on every row, and projection keeps rows in the same partitions. Its known value and uniform or partition-local status therefore still hold. Outer joins clear constants on the NULL-padded side before projecting their output. Partition merges clear Heterogeneous constants.

Also preserve scalar-subquery leaves during indirect projection. They read a shared subquery result without depending on the input schema. The arithmetic ORDER BY reproduction needs the metadata copy; the scalar-subquery join reproduction needs both changes.

What is the testing strategy for this PR?

SQL cases in joins.slt cover arithmetic, upper(), left joins, and the scalar-subquery join. A NULL-padded left join checks that matched values sort before the NULL rows. The metadata unit test covers known, unknown, and partition-local constants, dropped arguments, and partition merging.

The original SQL cases and metadata test fail before the fix and pass afterward. Validation includes the extended workspace tests, full lint suite, and strict all-features Clippy.

Are there any user-facing changes?

Affected queries plan and execute successfully. No public API changes.

Needs a backport to branch-56 for the 56.0.0 regression tracker (#24461).

@kumarUjjawal
kumarUjjawal requested a review from alamb November 5, 2026 07:30
@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Nov 5, 2026

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kumarUjjawal -- this looks correct, and a nice minimal fix to me

I pushed a commit to tweak a comment and will merge. Thank you again

@alamb
alamb added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026
@kumarUjjawal
kumarUjjawal added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026
kumarUjjawal and others added 3 commits November 6, 2026 10:27
With batch_size = 10 the 100 input rows are spread across 4 partitions,
and the partial states are merged in whichever order the partitions
finish, so the two differing percentiles can be reported as either
"0.5 and 0.9" or "0.9 and 0.5".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@alamb

alamb commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Updated test to get a passinc CI

@alamb
alamb added this pull request to the merge queue Nov 6, 2026
@alamb alamb mentioned this pull request Nov 6, 2026
28 of 53 tasks
Merged via the queue into apache:main with commit ad8578a Nov 6, 2026
43 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REGRESSION: SanityCheckPlan error with sort-merge join when a filter makes a sort or join key constant (constant lost in EquivalenceGroup projection)

2 participants