Search before asking
Paimon version
master (1.5-SNAPSHOT)
Compute Engine
Flink (cross-partition upsert / global dynamic-bucket sink) with unaligned checkpoints.
Minimal reproduce step
- Run a cross-partition upsert sink (global index assigner) with
execution.checkpointing.unaligned.enabled = true.
- Have a checkpoint land during the bootstrap-drain window, so a
KEY_PART record is still queued when the barrier triggers.
What doesn't meet your expectations?
The assigner crashes. GlobalIndexAssigner.bootstrapKey has checkArgument(inBoostrap()), which throws (and bootstrapKeys is already null, so it would NPE) when a KEY_PART record arrives after the bootstrap phase has ended. Under unaligned checkpoints the barrier overtakes buffered records, so prepareSnapshotPreBarrier runs endBoostrap (setting bootstrap=false, nulling bootstrapKeys) while KEY_PART records are still queued; those late records then hit bootstrapKey with bootstrap == false and fail the job, which can re-fail on restart in the same window. Aligned checkpoints never hit this, because barrier alignment processes every pre-barrier KEY_PART before endBoostrap.
Anything else?
No response
Are you willing to submit a PR?
Search before asking
Paimon version
master (1.5-SNAPSHOT)
Compute Engine
Flink (cross-partition upsert / global dynamic-bucket sink) with unaligned checkpoints.
Minimal reproduce step
execution.checkpointing.unaligned.enabled = true.KEY_PARTrecord is still queued when the barrier triggers.What doesn't meet your expectations?
The assigner crashes.
GlobalIndexAssigner.bootstrapKeyhascheckArgument(inBoostrap()), which throws (andbootstrapKeysis already null, so it would NPE) when aKEY_PARTrecord arrives after the bootstrap phase has ended. Under unaligned checkpoints the barrier overtakes buffered records, soprepareSnapshotPreBarrierrunsendBoostrap(settingbootstrap=false, nullingbootstrapKeys) whileKEY_PARTrecords are still queued; those late records then hitbootstrapKeywithbootstrap == falseand fail the job, which can re-fail on restart in the same window. Aligned checkpoints never hit this, because barrier alignment processes every pre-barrierKEY_PARTbeforeendBoostrap.Anything else?
No response
Are you willing to submit a PR?