Skip to content

Fix resumable job progress across runtimes - #1521

Draft
bgentry wants to merge 1 commit into
masterfrom
bg/fix-resumable-progress
Draft

bgentry wants to merge 1 commit into
masterfrom
bg/fix-resumable-progress

Conversation

@bgentry

@bgentry bgentry commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Resumable jobs save completed steps and cursors so retries can skip finished work. This PR fixes three bugs that made retries skip work that never committed or repeat work already done.

1. Retries can skip work that rolled back. River normally saves progress from memory when an attempt ends. Transactional helpers write a checkpoint immediately, inside the worker’s transaction. That checkpoint must commit or roll back with the application changes.

In Go, ResumableSetStepTx and ResumableSetStepCursorTx also added the checkpoint to memory before their database write succeeded. Their Rust equivalents and JavaScript’s resumable.checkpoint had the same bug. A worker could update a customer row’s address and save the customer’s ID as a cursor in the job’s metadata, all in one transaction. If the transaction rolled back, River still saved the cursor with the failed attempt. The retry then skipped that customer even though the address update never committed. A failed checkpoint write could cause the same bug.

The fix has to work without knowing how the transaction ends: River cannot see whether a caller-owned transaction later commits or rolls back. If the attempt's final update saved the checkpoint, it could restore one that rolled back. If it saved only the earlier progress, it could overwrite one that committed. So after a successful checkpoint write, the final update saves neither. It saves only progress recorded after that write. A failed checkpoint write leaves pending progress unchanged.

The cost is that a rollback can make the retry repeat earlier steps whose progress existed only in memory. Ruby already works this way. Suppose prepare completes before import checkpoints cursor 1, with no previously saved progress:

Checkpoint outcome before the attempt fails Next attempt
Write fails Skips prepare; starts import without a cursor
Write succeeds; transaction commits Skips prepare; resumes import at cursor 1
Write succeeds; transaction rolls back Repeats prepare; starts import without a cursor

2. A cursor in the first step is lost. A cursor records progress within an unfinished step. Go, Rust, JavaScript, and Java saved cursors only when at least one step had completed.

If the first step processed items 1–7, recorded cursor 7, and failed, the retry received no cursor and repeated those items.

River now saves cursors independently of step completion. The retry receives cursor 7 so it can continue with item 8. This also preserves cursors on snooze and cooperative shutdown.

3. Snoozing drops progress in Rust and JavaScript. Snoozing ends the attempt and schedules another run. Both runtimes treated snooze as a successful worker return, but saved progress only on error. A worker that completed download and snoozed before transform therefore downloaded everything again.

Both runtimes now save progress unless the job completed, matching Go and Ruby. Their worker test helpers follow the same rule.

Shared conformance cases cover these behaviors, including failed checkpoint writes and progress recorded after a checkpoint.

Not addressed here:

  • Java still has no transactional checkpoint helper.
  • Checkpoint updates match only on job ID. An execution that continues after timing out can still overwrite a newer execution's progress.

@bgentry bgentry changed the title fix resumable progress across runtimes Fix resumable job progress across runtimes Oct 20, 2026
Keep transactional checkpoints out of deferred attempt metadata after
successful writes so a later failure cannot persist progress whose
application transaction rolls back. Preserve earlier deferred progress
when the checkpoint write itself fails. Follow Ruby's rule that the
transaction owns a successful write, allowing earlier unpersisted steps
to run again after rollback.

Persist cursor changes independently of completed steps so a failed first
step can resume from its cursor. Retain Rust and JavaScript progress on
snooze and other incomplete outcomes, including in worker helpers, and
align Java's cursor behavior with the other runtimes.

Cover rollback, commit followed by failure, failed writes, subsequent
cursor changes, first-step retries, snooze, and cooperative shutdown with
regressions and shared conformance scenarios.
@bgentry
bgentry force-pushed the bg/fix-resumable-progress branch from 74e3217 to d761ae5 Compare October 20, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant