Skip to content

fix(sql): preserve unacked batch messages when ack deadline expires - #63

Open
AdamMagued wants to merge 1 commit into
ThreeDotsLabs:masterfrom
AdamMagued:fix-issue-700-ack-deadline-batch-loss
Open

AdamMagued wants to merge 1 commit into
ThreeDotsLabs:masterfrom
AdamMagued:fix-issue-700-ack-deadline-batch-loss

Conversation

@AdamMagued

Copy link
Copy Markdown

Motivation / Background

Fixes ThreeDotsLabs/watermill#700.

When using queue offsets adapters (such as PostgreSQLQueueOffsetsAdapter and MySQLQueueOffsetsAdapter), Subscriber.query fetches a batch of rows and iterates through them. If an earlier message in the batch misses its AckDeadline, Subscriber.query breaks the iteration loop. Previously, it passed the entire slice of selected rows (Rows: messageRows) to AckMessageQuery. Because queue adapters delete or mark as acknowledged every row present in params.Rows, all subsequent unprocessed rows in the batch, along with the timed-out message, were deleted or marked as acknowledged.

Details

  • In Subscriber.query, track successfully acknowledged rows in a dedicated ackedRows slice and supply only ackedRows to AckMessageQueryParams.Rows.
  • Guard AckMessageQuery execution against empty queries with !ackQuery.IsZero().
  • In PostgreSQLQueueOffsetsAdapter.AckMessageQuery and MySQLQueueOffsetsAdapter.AckMessageQuery, return an empty Query without error when params.Rows is empty.
  • Add focused unit tests in subscriber_test.go verifying that a batch timeout on an unacknowledged message acknowledges only preceding acknowledged rows and retains unacknowledged rows.
  • Add unit tests in queue_offsets_adapter_test.go covering DeleteOnAck branches and empty row handling for PostgreSQL and MySQL queue adapters.

Alternative approaches considered (if applicable)

Filtering rows by offset inside individual offsets adapters was considered, but passing only acknowledged rows directly from Subscriber.query solves the problem at the caller level for all queue adapters and prevents unnecessary row allocations downstream.

Checklist

  • I wrote tests for the changes.
  • All tests are passing.
  • Code has no breaking changes.
  • (If applicable) documentation on watermill.io is updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[watermill-sql] A missed ack deadline deletes the unprocessed rows of the batch (PostgreSQLQueueOffsetsAdapter)

1 participant