Repository navigation
refactor(outbox): claim events by lease and deliver per named handler - #78
Merged
Merged
Conversation
The outbox gains dead_at and a deliveries jsonb map (handler name to delivery time, checked to be an object) so a lease relay can retry an event per handler and park it as dead. The pending index now follows the claim order (occurred_at, id) over live rows only, with partial indexes for the dead retry and the dispatched retention sweep. The migration also drops the notify trigger: the notification moves into the enqueue transaction.
The event dispatcher now subscribes handlers under a persisted kebab-case name, unique per event type, and delivers an event only to the handlers not yet delivered, reporting one outcome per handler so a relay can retry just the ones that failed. A throwing handler is a failure and never stops the others. The unused dispatch, dispatchAll, unsubscribe and clearHandlers go away. onEvent takes the handler name. The three shipped handlers are named after their module and reaction; the two that send an email pass an idempotency key derived from the handler name and the event id, so a redelivery no longer sends a second email. The legacy dispatcher passes the row id as the event id.
The outbox port drops its transaction parameter and the pending batch read under a long lock. claim takes due live rows and due dead rows in one statement (two skip locked CTEs, a union and an update from it), bumps attempts and leases them with the SQL clock, so concurrent relays never share a row and a crashed one only delays it until the lease ends. markDelivered and markFailed are guarded by the claimed attempt count, so a settle after the lease expired writes nothing, and merge the handler deliveries server side. pendingState reports the next due delay and the oldest live row. Failures surface as typed results after capture. enqueue writes through the ambient executor, keys each row by the event id with on conflict do nothing, and notifies on the same executor, so listeners wake only once the run commits. emitEvent stamps the id it returns on the event. The legacy dispatcher drains through claim and settles each event after its subscribers; it no longer recreates the notify trigger. Hand-written outbox fakes give way to mockPort. check:outbox-lease proves the lease, the settle guard, the merge, the dead replay, the type filter and the rollback against Postgres, and runs in CI.
… state OutboxRelay claims a batch sized to what the lease can process, delivers each record to every handler not yet in its deliveries, each one raced against its budget and outside any transaction, then settles the lease: markDelivered when all succeed, markFailed with jittered retry otherwise. After OUTBOX_MAX_ATTEMPTS the event is dead, announced once through the new OUTBOX_EVENT_DEAD catalog type and replayed on the dead retry delay. OutboxDispatcher keeps its name and becomes the loop around the relay: LISTEN wake, the pending due delay, a 30 s poll, a 50 ms floor after an empty drain and a 2 s retry after a failed one. stop() halts claiming, waits for the running drain and abandons handlers still running once the shutdown grace is spent, so the workers step keeps its budget. The built-in subscribers still run in their own transaction through a temporary bridge until they become named handlers. The lease must cover the handler budget, checked at boot.
The LISTEN connection now retries from 250 ms doubling up to 15 s, sends select 1 every 15 s and replaces the client when the probe fails, and sets a connect and query timeout so a half-open socket cannot hang the probe. A NOTIFY sent while the connection was down is lost, so a reconnection calls onReconnect once: the outbox dispatcher wakes for a catch-up drain, the notification hub passes none.
…d handlers The three built-ins become handlers named audit-log, webhook-fanout and notification-fanout, subscribed to every catalog type by buildEventDispatcher next to the user handlers found in the container. Each takes the event rebuilt from its row (eventId, organizationId, requestId and aggregateType now ride on IDomainEvent) and runs its own short unit of work instead of a transaction handed to it; the audit keeps its chain lock inside it. A redelivery writes nothing twice: the audit row id and the webhook idempotency key already derive from the event id, and a notification without a dedup window now gets the key event:<eventId>, so the unique index on (user_id, dedup_key) drops the second insert. Proven by the new fanout and outbox-lease steps. outbox-subscriber.ts and the temporary transaction bridge are gone, the layering allowlists are empty.
…eping emitEvent already returns the eventId it hands to the outbox; a test now pins that this id is the row id the repository writes, so callers can correlate the audit entry and the webhook deliveries with it. Rule 6 states that delivery bookkeeping is not a state change and that a dead event is announced once.
The guides described a dispatcher that held row locks while built-in subscribers ran inside its transaction and dropped user handlers on a throw. They now describe the lease claim, named handlers settled per attempt, deliveries, jittered retry, the dead state replayed every OUTBOX_DEAD_RETRY_HOURS, the idempotence argument of each of the six handlers and why the audit chain follows completion order. The outbox sweep keeps purging on dispatched_at alone; check-outbox-lease now proves on Postgres that a dead undelivered row survives it and a delivered one past the cutoff does not, and a unit test pins the rendered filter. Catalog counts move to 86 / 35 / 51 and Phase I moves to the shipped phases, with the email queue and webhook delivery lease alignment left as an open follow-up.
|
🎉 This PR is included in version 2.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FOR UPDATE SKIP LOCKED, SQL clock) and settled per attempt with a guard, no transaction held around handlersdeliveriesrecords who succeeded so a retry only reruns the failed onesOUTBOX_EVENT_DEADin the same transaction (scrubbed error text)Resultso they are retried (token revocation, notification emails with idempotency keys)select 1keepalive, query timeout, catch-up drain on reconnect; notify trigger replaced bypg_notifyat enqueuecheck:outbox-lease(19 steps) in CI; migration 0024 must run with old pods stopped (see DEPLOY-RAILWAY)