Repository navigation
Move BundleLifter from Iceberg to sdk/util - #40978
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly relocates BundleLifter and BundleLifterTest from org.apache.beam.sdk.io.iceberg to org.apache.beam.sdk.util and annotates BundleLifter with @Internal for reuse across other SDK components (such as WriteFiles). The refactor preserves step names and output tags, maintaining pipeline update compatibility.
🔍 General Feedback
- The package move, visibility annotations, and import updates in Iceberg IO are clean and have no behavioral regressions.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @ahmedabu98 for label iceberg. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @scwhittle |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
| + " into a static factory method may fix it.") | ||
| public class BundleLifter<T> extends PTransform<PCollection<T>, PCollectionTuple> { | ||
|
|
||
| final TupleTag<T> smallBatchTag; |
There was a problem hiding this comment.
Unrelated so can be done separate PR, but I think that it could be improved to stream the elements through. Once we've exceeded threshold we transition to large output and
- flush the existing queue
- stop sizing/queuing further process elements and just outputting them directly.
|
The beam_PreCommit_Java failure on #40978 seems to be unrelated to the change. |
Part of #40976
Follow-up to #40543.
Moves
BundleLifterfromsdks/java/io/icebergtosdks/java/core/src/main/java/org/apache/beam/sdk/utiland marks it@Internal, so it can be reused outside Iceberg (e.g. byWriteFilesdirect writes).Changes
BundleLifterandBundleLifterTestintoorg.apache.beam.sdk.util(@Internal).WriteToDestinations.Update compatibility
The class name, the
BundleLiftDoFnstep name, and the output tags are unchanged, so Iceberg pipeline step names stay the same and streaming--updateis unaffected. Only the Java package changed, which Dataflow does not use for update matching.Testing
BundleLifterTest(core)IcebergIOWriteTest