Repository navigation
fix(substrait): accept standard shift_left / shift_right function names in the consumer - #27072
Open
jonasHanhan wants to merge 2 commits into
Open
jonasHanhan wants to merge 2 commits into
jonasHanhan wants to merge 2 commits into
Conversation
…onsumer Substrait's functions_arithmetic.yaml defines the shift functions as shift_left and shift_right, but the consumer only knew the names bitwise_shift_left and bitwise_shift_right, so plans from other producers were rejected with "Unsupported function name". Map the standard names to the shift operators and keep the old names for existing plans. Closes apache#27059
bvolpato
approved these changes
Nov 6, 2026
bvolpato
left a comment
Contributor
There was a problem hiding this comment.
LGTM. The standard shift names map to the matching DataFusion operators, and keeping the bitwise_shift_* aliases preserves compatibility with existing plans. The full Substrait crate tests pass locally at this head.
Could you also enable Actions on your fork and push again before merge? Rust (fork) is currently neutral because it has no run for this commit. Thanks for fixing this.
Author
|
Thanks for the review! I'll enable Actions on the fork and push again so Rust (fork) can run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
shift_left/shift_rightfrom functions_arithmetic.yaml #27059.Rationale for this change
Substrait's
functions_arithmetic.yamldefinesshift_leftandshift_right, but the consumer only recognisedbitwise_shift_left/bitwise_shift_right. A plan from another producer that uses the standard names fails withUnsupported function name: "shift_left".What changes are included in this PR?
name_to_opmapsshift_left->BitwiseShiftLeftandshift_right->BitwiseShiftRight.bitwise_shift_left/bitwise_shift_rightnames still work, so plans written by current and older DataFusion versions keep loading.Things I deliberately left out, happy to follow up if wanted:
bitwise_shift_*. Switching it to the standard names would make plans from a new DataFusion unreadable by older DataFusion consumers, so it seemed better as a separate change.shift_right_unsignedis not mapped. DataFusion's>>is an arithmetic shift for signed integers (matching Substrait'sshift_right), and I don't think there is an existing logical-shift operator to map it to.What is the testing strategy for this PR?
Added
test_shift_function_namesinconsumer/expr/scalar_function.rs, which consumes a scalar function for each of the four names (standard and legacy) and checks the resulting expression is the expected<</>>binary expression.Locally:
cargo test -p datafusion-substraitpasses, andcargo clippy -p datafusion-substrait --all-targets -- -D warningsandcargo fmt --checkare clean.Are there any user-facing changes?
The Substrait consumer now accepts plans that use the standard shift function names. No API changes.