Repository navigation
refactor(queues): claim email and webhook deliveries by lease on the database clock - #79
Merged
Merged
Conversation
The lease vocabulary of the outbox (database clock, claim increment, settle guard on id and attempts, release) moves to packages/drizzle/src/lease.ts so the email and webhook queues can claim and settle the same way instead of copying it. The retry decision (dead at the last attempt, a dead row stays dead on the slow cadence, jittered delay otherwise) becomes the pure failureSchedule in shared/jitter.ts, with retryAfterSeconds promoted from the relay. The due predicates (isDue, isDeadDue) and the dead share of a claim (deadShareOf) are shared too; a generic claimDue over the CTE union does not type-check on drizzle 0.45 without casts, so each queue still writes its own claim. The check scripts share their step runner, span recorder, probe claims and shifted clock in check-harness.ts. Outbox behaviour is unchanged; check:outbox-lease stays green.
…ad state The email queue now follows the outbox lease model. A claim increments attempts and leases the row until now() plus EMAIL_LEASE_SECONDS on the database clock, in one statement that unions due pending rows with due dead rows (at most a quarter of the claim). Every settle is guarded by id, attempts and a non-terminal status and reports what it wrote, so a worker that lost its lease no longer overwrites the row of the one that holds it. Retryable failures follow failureSchedule: pending with a jittered delay, then dead on the 4th failed send as before (attempts: 4 in the event, the ceiling is 4 now that the claim counts the attempt), replayed every EMAIL_DEAD_RETRY_HOURS while the message is younger than EMAIL_DEAD_REPLAY_WINDOW_HOURS so a stale sign-in or reset link is never sent late. failed stays the never replayed terminal of a permanent rejection. EMAIL_DELIVERY_EXHAUSTED is emitted only when the settle wrote and only on the first transition to dead or failed, in the settle's unit of work. On stop, the worker releases the chunks it had not started (attempt given back, due at once). The failed sweep also purges dead rows, and the boot refuses a replay window that closes before the first replay. The Resend batch call aborts after 30 s as a retryable failure, the claim size is derived from the lease (3 chunks of 100 at 120 s), and before each chunk after the first the worker checks that its remaining lease covers one more send plus a 15 s margin, releasing the rest otherwise; the boot refuses EMAIL_LEASE_SECONDS under one send timeout plus that margin. A row that fails to render is settled failed on its own, with its event, and the rest of the chunk is still sent. check:email-lease proves the claim, lease expiry, database clock, stale guard, dead replay window and share, release and terminal states against Postgres; check:marksent is rewritten for the lease guard.
…dead_letter The webhook worker claimed rows inside a transaction held open with SET LOCAL, stamped the lease from the process clock and settled blindly, so a worker whose lease had expired could overwrite the outcome of the one that reclaimed the row. Deliveries now follow the outbox and email model: one claim statement (live and replayable dead_letter CTEs, skip locked, dead share capped at a quarter) bumps attempts and leases the row on the database clock, and markSuccess, markFailed and release are guarded by the claimed attempts and report whether they wrote. The public statuses are unchanged. A dead_letter with next_attempt_at set is retried every WEBHOOK_DEAD_RETRY_HOURS; one with NULL (missing or disabled endpoint, non-public destination, every existing row) is terminal. Counters, the endpoint reset or bump and WEBHOOK_DELIVERY_EXHAUSTED only follow a settle that wrote, the event only on the first death, in the settle's unit of work. The attempt row is still written when a real request went out. Unstarted deliveries are released on stop. enqueueReplay stamps the database clock. A delivery dies on its 4th failed send as before. The host lookup of the SSRF guard is bounded to 5 s and a timeout is retryable. WEBHOOK_LEASE_SECONDS (default 330) must cover the lookup and 30 s request timeouts plus a 30 s margin, checked at boot; a claim takes as many deliveries as the lease covers (8 at 330 s, capped at 10), and before each delivery after the first the worker checks its remaining lease and releases the rest otherwise. Fan-out schedules new deliveries on the database clock. Known quirk kept: a thrown processDelivery marks the delivery failed with no organization, so its exhaustion emits no event and bumps no counter. During a rolling deploy an old worker may settle blindly for one lease. Proven against Postgres by check:webhook-lease, wired in CI.
nextAttemptAt stamped retries from the process clock and isDeadLetter with JITTER_MAX_ATTEMPTS carried a dead threshold shared by every queue. Every queue now schedules through failureSchedule with its own max attempts and writes now() + interval on the database clock, so the helpers have no caller left. The header now says how the jitter math reaches the queues.
Generalise the delivery section of EVENTS.md to the three queues and record the knobs, the dead states, the double delivery caveat and the db:push partial index caveat. Extend root rule 6 to delivery queue bookkeeping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FOR UPDATE SKIP LOCKED), like the outboxdeadstate replayed every 6 h within a 24 h window;failedstays the permanent rejectiondead_letterwith a next attempt is replayed every 6 h, without one it is terminal (existing rows stay terminal)@packages/drizzleused by all three queues; process-clock retry helpers removed fromjitter.tscheck:email-leaseandcheck:webhook-leasein CI,check:marksentrewritten; 1000 api tests green