Skip to content

Dedupe batch completion across both code paths#5

Open
lovitt wants to merge 1 commit into
masterfrom
fix/batch-completion-double-fire
Open

Dedupe batch completion across both code paths#5
lovitt wants to merge 1 commit into
masterfrom
fix/batch-completion-double-fire

Conversation

@lovitt

@lovitt lovitt commented Jul 10, 2026

Copy link
Copy Markdown
Member

The bug

on_batch_complete can be triggered from two code paths that may observe a batch as complete at the same time:

  • on_child_complete — the last child to finish notifies the parent.
  • complete — the parent's own post-setup backstop, run at the end of execute.

Only on_child_complete claimed the batch_completion_gid SETNX token before firing on_complete; complete fired it directly. When both observe complete? == true concurrently, on_batch_complete could fire twice.

The fix

Extract the guarded fire into trigger_completion(status) — SETNX the token, run on_complete, clear the token on error so completion can be reattempted — and route both paths through it, so a concurrent backstop and child no longer both fire.

on_batch_node_complete stays outside the guard: it's a distinct per-node event that complete fires once on its own.

Testing

  • #trigger_completion: fires when unclaimed, forwards a non-:completed status, no-ops when already claimed, clears the key and re-raises on error.
  • #complete: claims the token when complete; defers (no on_complete) when the token is already held, while on_batch_node_complete still propagates.

Scope and limitations

The SETNX guard dedupes the common case: two paths racing to complete a batch that's still in flight. It does not handle Cloud Tasks delivering the same job twice — an uncommon at-least-once redelivery. A job that already completed can arrive after cleanup has wiped the batch state and the token, re-create a phantom slot, and re-fire on_batch_complete. Cloudtasker runs a job every time Cloud Tasks delivers it and never checks whether that job already ran, so workers needing exactly-once completion should keep perform and on_batch_complete idempotent.

@lovitt lovitt force-pushed the fix/batch-completion-double-fire branch from 97db1da to 2eb52ae Compare July 10, 2026 18:38
@lovitt lovitt changed the title Dedupe batch completion across both finisher paths Dedupe batch completion across both finisher paths, and outlast cleanup Jul 11, 2026
@lovitt lovitt changed the title Dedupe batch completion across both finisher paths, and outlast cleanup Dedupe batch completion across both finisher paths Jul 11, 2026
@lovitt lovitt force-pushed the fix/batch-completion-double-fire branch 2 times, most recently from 2eb52ae to 89e80db Compare July 11, 2026 16:06
@lovitt lovitt changed the title Dedupe batch completion across both finisher paths Dedupe batch completion across both code paths Jul 11, 2026
@lovitt lovitt force-pushed the fix/batch-completion-double-fire branch from 89e80db to 187c78e Compare July 11, 2026 16:10
on_batch_complete can be triggered from two code paths that may observe a
batch as complete at the same time: the last child (on_child_complete) and
the parent's own post-setup backstop (complete). Only on_child_complete
claimed the batch_completion_gid SETNX token before firing on_complete;
complete fired it directly. When both observe complete? == true
concurrently, on_batch_complete could fire twice.

Extract the guarded fire into trigger_completion (SETNX -> on_complete,
clearing the key on error so completion can be reattempted) and route
both paths through it, so a concurrent backstop and child no longer both
fire. on_batch_node_complete stays outside the guard: it is a distinct
per-node event that complete fires once on its own.
@lovitt lovitt force-pushed the fix/batch-completion-double-fire branch from 187c78e to 15449f4 Compare July 11, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant