Skip to content

Move BundleLifter from Iceberg to sdk/util - #40978

Merged
ahmedabu98 merged 1 commit into
apache:masterfrom
tejasiyer-dev:move-bundle-lifter
Nov 10, 2026
Merged

ahmedabu98 merged 1 commit into
apache:masterfrom
tejasiyer-dev:move-bundle-lifter

Conversation

@tejasiyer-dev

Copy link
Copy Markdown
Contributor

Part of #40976

Follow-up to #40543.

Moves BundleLifter from sdks/java/io/iceberg to sdks/java/core/src/main/java/org/apache/beam/sdk/util and marks it @Internal, so it can be reused outside Iceberg (e.g. by WriteFiles direct writes).

Changes

  • Moved BundleLifter and BundleLifterTest into org.apache.beam.sdk.util (@Internal).
  • Updated the import in Iceberg's WriteToDestinations.
  • No behavior changes.

Update compatibility

The class name, the BundleLiftDoFn step name, and the output tags are unchanged, so Iceberg pipeline step names stay the same and streaming --update is unaffected. Only the Java package changed, which Dataflow does not use for update matching.

Testing

  • BundleLifterTest (core)
  • IcebergIOWriteTest

@github-actions github-actions Bot added java io core iceberg Iceberg IO connector size/s S: 10-49 lines changed labels Nov 3, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly relocates BundleLifter and BundleLifterTest from org.apache.beam.sdk.io.iceberg to org.apache.beam.sdk.util and annotates BundleLifter with @Internal for reuse across other SDK components (such as WriteFiles). The refactor preserves step names and output tags, maintaining pipeline update compatibility.

🔍 General Feedback

  • The package move, visibility annotations, and import updates in Iceberg IO are clean and have no behavioral regressions.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label iceberg.
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@ahmedabu98 ahmedabu98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arunpandianp

Copy link
Copy Markdown
Contributor

R: @scwhittle

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

+ " into a static factory method may fix it.")
public class BundleLifter<T> extends PTransform<PCollection<T>, PCollectionTuple> {

final TupleTag<T> smallBatchTag;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated so can be done separate PR, but I think that it could be improved to stream the elements through. Once we've exceeded threshold we transition to large output and

  1. flush the existing queue
  2. stop sizing/queuing further process elements and just outputting them directly.

@tejasiyer-dev

Copy link
Copy Markdown
Contributor Author

The beam_PreCommit_Java failure on #40978 seems to be unrelated to the change.

@ahmedabu98
ahmedabu98 merged commit 1f52d36 into apache:master Nov 10, 2026
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core iceberg Iceberg IO connector io java Next Action: Reviewers size/s S: 10-49 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants