Skip to content

fix(substrait): accept standard shift_left / shift_right function names in the consumer - #27072

Open
jonasHanhan wants to merge 2 commits into
apache:mainfrom
jonasHanhan:substrait-standard-shift-names
Open

jonasHanhan wants to merge 2 commits into
apache:mainfrom
jonasHanhan:substrait-standard-shift-names

Conversation

@jonasHanhan

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Substrait's functions_arithmetic.yaml defines shift_left and shift_right, but the consumer only recognised bitwise_shift_left / bitwise_shift_right. A plan from another producer that uses the standard names fails with Unsupported function name: "shift_left".

What changes are included in this PR?

  • name_to_op maps shift_left -> BitwiseShiftLeft and shift_right -> BitwiseShiftRight.
  • The existing bitwise_shift_left / bitwise_shift_right names still work, so plans written by current and older DataFusion versions keep loading.

Things I deliberately left out, happy to follow up if wanted:

  • The producer still writes bitwise_shift_*. Switching it to the standard names would make plans from a new DataFusion unreadable by older DataFusion consumers, so it seemed better as a separate change.
  • shift_right_unsigned is not mapped. DataFusion's >> is an arithmetic shift for signed integers (matching Substrait's shift_right), and I don't think there is an existing logical-shift operator to map it to.
  • One thing I'm not fully sure about: Substrait leaves the result of a shift amount outside the bit width up to the implementation, and DataFusion wraps it. I didn't change that behaviour here.

What is the testing strategy for this PR?

Added test_shift_function_names in consumer/expr/scalar_function.rs, which consumes a scalar function for each of the four names (standard and legacy) and checks the resulting expression is the expected << / >> binary expression.

Locally: cargo test -p datafusion-substrait passes, and cargo clippy -p datafusion-substrait --all-targets -- -D warnings and cargo fmt --check are clean.

Are there any user-facing changes?

The Substrait consumer now accepts plans that use the standard shift function names. No API changes.

…onsumer

Substrait's functions_arithmetic.yaml defines the shift functions as
shift_left and shift_right, but the consumer only knew the names
bitwise_shift_left and bitwise_shift_right, so plans from other producers
were rejected with "Unsupported function name". Map the standard names to
the shift operators and keep the old names for existing plans.

Closes apache#27059
@github-actions github-actions Bot added the substrait Changes to the substrait crate label Nov 4, 2026

@bvolpato bvolpato left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The standard shift names map to the matching DataFusion operators, and keeping the bitwise_shift_* aliases preserves compatibility with existing plans. The full Substrait crate tests pass locally at this head.

Could you also enable Actions on your fork and push again before merge? Rust (fork) is currently neutral because it has no run for this commit. Thanks for fixing this.

@jonasHanhan

Copy link
Copy Markdown
Author

Thanks for the review! I'll enable Actions on the fork and push again so Rust (fork) can run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait consumer doesn't recognise shift_left / shift_right from functions_arithmetic.yaml

2 participants