Skip to content

Prevent on_batch_complete from firing before all children are enqueued#175

Open
lovitt wants to merge 3 commits into
keypup-io:masterfrom
castolonco:fix/batch-completion-premature-firing
Open

Prevent on_batch_complete from firing before all children are enqueued#175
lovitt wants to merge 3 commits into
keypup-io:masterfrom
castolonco:fix/batch-completion-premature-firing

Conversation

@lovitt

@lovitt lovitt commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

The bug

on_batch_complete can fire before the batch has finished enqueuing its children.

The parent enqueues children one at a time, and complete? only checks the children already registered in the batch state — it has no notion of how many there will ultimately be. On a large fan-out, an early child can finish before the parent has registered the rest; at that point, if every child the batch knows about is complete, complete? returns true and completion fires while later children are still unscheduled.

We hit this in production: a ~300-child batch fired on_batch_complete while still enqueuing, and the summary it produced was missing the results the not-yet-run children went on to write.

The fix

Mark the batch while setup is enqueuing children, and require enqueuing to be finished before a child can trigger completion:

  • setup marks batch_setup_gid in_progress before scheduling and done after.
  • on_child_complete returns early unless setup_complete?.
  • cleanup removes the key.

Only the child-driven path is gated. A batch whose children all finish during enqueuing still completes via the parent's own complete at the end of execute, so no completion is lost. This is separate from the existing batch_completion_gid guard, which dedupes concurrent finishers on an already-enqueued batch.

Progressive batch expansion is unaffected: the marker is written during the initial setup, and an expanding child sits in the batch state as processing, so complete? stays false while it runs.

One edge case: if the root fails setup on every retry until it's marked dead, the batch has registered children but its marker never reaches done, so it never resolves. This only affects a batch whose root died before it could finish enqueuing; on master the same case instead fires on_batch_complete off the partial set, reporting a never-assembled batch as complete.

Backward compatibility

setup marks a batch in_progress while it enqueues children and done once finished; the gate only holds a batch back while it's in_progress. A batch enqueued before this change has no marker, so it's treated as fully enqueued and completes normally.

Testing

Added a spec reproducing the premature fire (written to fail on master), plus coverage for setup_complete?, the setup marker, cleanup removal, and a batch enqueued before this change (upgrade in flight).

lovitt added 2 commits July 9, 2026 13:54
`schedule_pending_jobs` dispatches each child to Cloud Tasks and only
then registers it in the batch state, one child at a time. `complete?`
considers a batch done as soon as every *currently-registered* child is
complete — it has no notion of the parent having finished enqueuing.

On a large fan-out with uneven child durations, fast early children can
complete while the parent is still scheduling slower siblings. At that
point every registered child is complete, so `on_child_complete` fires
`on_batch_complete` prematurely — before the slow children have even been
scheduled. The one-shot completion guard makes this worse: it fires
exactly once, so the early (wrong) result is never corrected.

Flag the batch as fully enqueued at the end of `setup`, and require that
flag before a child may trigger completion. The parent's own post-`setup`
`complete` call remains the backstop for the case where every child
finishes before the flag is set, so no batch can hang. Children still
register after dispatch, so a partial setup cannot orphan state.
Progressive expansion (a running child adding more jobs to an
already-enqueued batch) is a supported workflow. It is unaffected by the
premature-completion race — and by the seal — because the expanding child
sits in the parent's batch state as `processing` until after it has
registered its additions, so the batch cannot be judged complete mid-
expansion. Pin that down so the seal can't regress it.
@alachaum alachaum self-requested a review July 10, 2026 16:18
Track setup as in_progress/done instead of a single end-of-setup flag, and
treat a marker-less batch (enqueued before this change) as fully enqueued so
batches in flight during an upgrade are not stranded.
@lovitt

lovitt commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

I just pushed a follow-up commit (e964835) for backwards compatibility — otherwise a batch already in flight during an upgrade would be held open and never fire on_batch_complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant