Dedupe batch completion across both code paths#5
Open
lovitt wants to merge 1 commit into
Open
Conversation
97db1da to
2eb52ae
Compare
2eb52ae to
89e80db
Compare
89e80db to
187c78e
Compare
on_batch_complete can be triggered from two code paths that may observe a batch as complete at the same time: the last child (on_child_complete) and the parent's own post-setup backstop (complete). Only on_child_complete claimed the batch_completion_gid SETNX token before firing on_complete; complete fired it directly. When both observe complete? == true concurrently, on_batch_complete could fire twice. Extract the guarded fire into trigger_completion (SETNX -> on_complete, clearing the key on error so completion can be reattempted) and route both paths through it, so a concurrent backstop and child no longer both fire. on_batch_node_complete stays outside the guard: it is a distinct per-node event that complete fires once on its own.
187c78e to
15449f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
on_batch_completecan be triggered from two code paths that may observe a batch as complete at the same time:on_child_complete— the last child to finish notifies the parent.complete— the parent's own post-setupbackstop, run at the end ofexecute.Only
on_child_completeclaimed thebatch_completion_gidSETNX token before firingon_complete;completefired it directly. When both observecomplete? == trueconcurrently,on_batch_completecould fire twice.The fix
Extract the guarded fire into
trigger_completion(status)— SETNX the token, runon_complete, clear the token on error so completion can be reattempted — and route both paths through it, so a concurrent backstop and child no longer both fire.on_batch_node_completestays outside the guard: it's a distinct per-node event thatcompletefires once on its own.Testing
#trigger_completion: fires when unclaimed, forwards a non-:completedstatus, no-ops when already claimed, clears the key and re-raises on error.#complete: claims the token when complete; defers (noon_complete) when the token is already held, whileon_batch_node_completestill propagates.Scope and limitations
The SETNX guard dedupes the common case: two paths racing to complete a batch that's still in flight. It does not handle Cloud Tasks delivering the same job twice — an uncommon at-least-once redelivery. A job that already completed can arrive after
cleanuphas wiped the batch state and the token, re-create a phantom slot, and re-fireon_batch_complete. Cloudtasker runs a job every time Cloud Tasks delivers it and never checks whether that job already ran, so workers needing exactly-once completion should keepperformandon_batch_completeidempotent.