[core] Register late bootstrap keys after unaligned checkpoints - #10295
Open
LuciferYang wants to merge 3 commits into
Open
LuciferYang wants to merge 3 commits into
LuciferYang wants to merge 3 commits into
Conversation
With execution.checkpointing.unaligned=true the checkpoint barrier can run the assigner's endBootstrap while KEY_PART records from slower channels are still queued. The first late bootstrapKey then failed the in-bootstrap assertion, failing the task and likely restart-looping under the same backpressure that motivated unaligned checkpoints. A late KEY_PART record's purpose is only to register its key in the index, which by then is bulk-loaded: put the key directly instead of crashing. Assisted-by: GLM-5.3
Guard the late-KEY_PART put with keyIndex.get(key)==null so a KEY_PART arriving after endBoostrap under unaligned checkpoints cannot overwrite an index entry an input record already assigned. Add tests pinning both the no-overwrite guard and the late-key registration (cross-partition retraction).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Under unaligned checkpoints the barrier can overtake buffered
KEY_PARTrecords, soGlobalIndexAssigner.prepareSnapshotPreBarrierrunsendBoostrap(settingbootstrap=falseand nullingbootstrapKeys) whileKEY_PARTrecords are still queued. Those late records then reachbootstrapKeywithbootstrap == false, wherecheckArgument(inBoostrap())throws (and the now-nullbootstrapKeyswould NPE), crashing the assigner and re-failing on restart in the same window. Aligned checkpoints never hit this, since barrier alignment processes every pre-barrierKEY_PARTbeforeendBoostrap.This registers a late bootstrap key directly into
keyIndexinstead of assertinginBoostrap(). It uses first-writer-wins (register only when the key is absent), so a lateKEY_PARTcannot overwrite an entry the running assigner already assigned to a live input row, which would otherwise leavekeyIndexpointing at a bucket the data no longer lives in and misroute later same-key records into a cross-bucket duplicate primary key. The in-bootstrap buffering path and the aligned-checkpoint path are byte-for-byte unchanged.Limitation: under unaligned-checkpoint reordering, if a data row for the same key is processed before the late
KEY_PART, the record in the previously-assigned partition is not retracted. Preserving that ordering fully is an operator/checkpoint-layer concern beyond this assigner; this change keepskeyIndexself-consistent and removes the crash.This closes #10294.
Tests
testLateBootstrapKeyAfterEndRegistersKeypins that aKEY_PARTafterendBoostrapregisters instead of crashing.testLateBootstrapKeyDoesNotOverwriteAssignedKeypins first-writer-wins: a lateKEY_PARTdoes not overwrite an already-assigned key.testLateBootstrapKeyRegistersAndRetractsOldPartitionpins the cross-partition registration path.API and Format
No.
Documentation
No.