Repository navigation
fix(cron): run dynamic schedules with their payload and keep one chain per schedule - #102
Merged
Merged
Conversation
…n per schedule Successors of a cron occurrence now copy its payload and priority, and occurrences of a dynamic schedule run as pgconductor.invoke events with their payload, which the worker validates like an invocation. ctx.schedule() replaces a schedule in _private_schedule(), which holds the queue row lock like _private_unschedule(). A running occurrence of the replaced schedule inserts no successor once a later occurrence of its schedule is live, so only the new chain survives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
psteinroe
added a commit
that referenced
this pull request
Oct 26, 2026
Settlement now plans for one partition, so its cost no longer grows with the number of queues (finding P10). Before, `buildReturnExecutions` referenced the partitioned executions table about 25 times and none of those references could be pruned. At 106 queues that cost ~520 ms of planning for each of the first 6 calls on every connection and ~30 ms per call after that. With Postgres' default `jit = on`, the statement's estimated cost also crossed `jit_above_cost`, so every settle JIT-compiled ~10.7k functions and took **~14 s**. **One queue per statement** `returnExecutions` now takes the worker's `queueName`, and results no longer carry a `queue` each. Every reference to the executions table in the statement is narrowed to that queue. The queue is inlined as an escaped `E''` literal rather than bound as a parameter, because a cached generic plan prunes partitions only by constants. So each queue gets its own prepared statement, planned a few times per connection and then served from the plan cache. With a parameter, Postgres kept choosing custom plans: ~3.5 ms of planning on every call, plus one ~250 ms generic plan per connection at 106 queues. A test covers queue names with a quote and a backslash, and the escaping round-trips under both `standard_conforming_strings` settings. **Parent paths move into functions** Waiting parents can be in any queue, so their lookups now run in two plpgsql functions that settlement calls only when a settled execution has a parent: - `_private_complete_children()` stores results for waiting parents, wakes them, and returns orphaned children (children no parent waits on any more). - `_private_fail_waiting_parents()` fails the chain of waiting ancestors and returns them for dead-lettering, result waits and the cancel cascade. Their statements can't be pruned, so they use `plan_cache_mode = force_generic_plan` (planned once per connection) and look rows up by arrays of ids to stay on indexes: `jsonb_to_recordset` is always estimated at 100 rows, which otherwise led to a seq scan of every partition. Both functions read with fresh snapshots, so they run first in `writes`, before the statement changes any row they read. That keeps their view identical to the old single snapshot. The old existence check on the parent in `inserted_children` is dropped because `valid_results` already holds that row locked. Behaviour is unchanged otherwise: the full suite passes unchanged apart from the call shape. Two new tests cover the moved paths with parents in other queues: an orphaned child, and a three-level failure that dead-letters the middle ancestor. Both new tests also pass on `main`. **Measurements** The harness seeds ~1M executions: 93% completed, 3% failed, 3% pending, 1% suspended parents. It runs Postgres 15 with `pg_stat_statements.track_planning`. Each settle is one completion plus one retry, sent through `DatabaseClient` (postgres.js prepared statements). Times below are DB-side from `pg_stat_statements`; wall times on this machine were noisy. | | before | after | |---|---|---| | 6 queues, first 6 calls per connection | 18–29 ms planning each | 3.5–4.5 ms planning each | | 6 queues, steady state | 1.0–1.1 ms exec (cached generic plan) | 0.5–0.6 ms exec, no planning | | 106 queues, first 6 calls per connection | ~520 ms planning each (~3.1 s per connection) | 3.5–7.7 ms planning each | | 106 queues, steady state | 5.7 ms exec, ~30 ms wall (a generic plan locks every partition) | 0.5–0.7 ms exec, no planning | | 106 queues, `jit = on` | ~14 s per settle | 0.8 ms exec | When every settled child wakes a parent in another queue (an `invokeAndWait` scenario), median wall time per settle drops from 39 ms to 9 ms at 106 queues and from 3.7 ms to 2.8 ms at 6 queues. At 106 queues the first call per connection takes ~170 ms (the functions' generic plans), against ~600–900 ms for each of the first 3 calls before. Trade-off: each settled queue keeps its own cached plan of about 1 MB per connection. Before, there was one plan per connection, but it grew with the partition count: 2 MB at 6 queues, 16 MB at 106. **Follow-ups (out of scope)** - Foreign-key checks into executions from steps and signals are still planned against every partition (P12, F11). - Claim planning still grows with queue count (P11, F12). The claim's estimated cost should be checked against `jit_above_cost` as well. - `cron-scheduling.test.ts` › *cron task with failure still reschedules next execution* timed out once in CI after the rebase onto #102 (no future occurrence for ~12 s) and passed on re-run and in every local run. Settlement is not involved; it may be a timing flake in #102's successor check (F13).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes follow-ups 1, 12 and 19 from the handover (originally noted in #89 and #90).
What
Dynamic schedule payloads (fix(scheduling): overnight windows, unschedule races, maintenance jitter #89, fix(retries): make retried writes from running executions replay-safe #90).
ctx.schedule(task, name, { cron }, { userId })lost its payload in two ways:_private_schedule_next_occurrence()inserted successors withoutpayloadorpriority, so from the second occurrence on the payload wasnulland the priority was0.taskEventForgave cron executions no payload at all, so even the first occurrence's payload never reached the handler. A dynamic occurrence arrived as{ name: "<schedule name>" }, which matches none of the handler's event types.Successors now copy
payloadandpriority, as they already didgroupandmetadata. Occurrences of a dynamic schedule (dynamic::dedupe keys) now run as{ name: "pgconductor.invoke", payload }. That matches the typectx.schedulechecks the payload against, and the worker validates the payload against the task's schema, as it does for every invocation. Static cron triggers are unchanged:{ name: "<trigger name>" }, no payload.Two schedule chains (fix(scheduling): overnight windows, unschedule races, maintenance jitter #89). If
ctx.schedule()replaced a schedule while a running occurrence of the old schedule was inserting its successor, both chains survived. There were two ways this happened:The fix also has two parts:
_private_schedule(), a plpgsql function next to_private_unschedule(). It takes the queue row lockfor update, then deletes and upserts in later statements. The successor path already takes that lockfor share, so the two are now ordered._private_schedule_next_occurrence()inserts no successor while a live occurrence of the same schedule with a later scheduled time exists. It stores that occurrence's id as the step instead. It compares the dedupe key's timestamp, notrun_at, because retries and sleeps moverun_at.Docs (fix(retries): make retried writes from running executions replay-safe #90).
testing.mdsaid the next occurrence is scheduled after each run. It is scheduled when an occurrence starts, before its handler.cron.mdnow describes the event a dynamic occurrence receives and what replacing a running schedule does.AGENTS.md"Cron Schedules" is updated.Behaviour change: handlers that told dynamic occurrences apart by
event.name === "<schedule name>"now see"pgconductor.invoke". That shape was never in the handler's type, and the payload was never delivered.Verification
Test first. All three new tests in
cron-scheduling.test.tsfailed onmain:{ name: "user-u1", payload: undefined }, and the successor had no payload or priority.The existing test cron task with failure still reschedules next execution caught a first version that compared
run_at; the dedupe key timestamp fixes that.just lint(oxlint plus typecheck) andjust formatare clean.pnpm test: 614 of 615 pass. The one failure is inwait-deadlines.test.ts(… settled after the deadline > times out), which fails the same way onmainat9f450dawhen run alone (2–3 of its 11 tests). See the follow-ups.Follow-ups (out of scope)
wait-deadlines.test.tsis flaky onmain. Since fix(testing): keep fake time running so event waits match under a set test clock #98, fake time keeps running, so the waiter's 1 s fake deadline (12:00:00 → 12:00:01) can pass during setup before the test sets the clock forward. The test then sees a completion instead of a timeout. This fits F13 (test suite).trace_context, so each occurrence starts a new trace. I left that unchanged on purpose.🤖 Generated with Claude Code