Skip to content

fix(cron): run dynamic schedules with their payload and keep one chain per schedule - #102

Merged
psteinroe merged 1 commit into
mainfrom
t3/fix-cron-payload-priority
Oct 21, 2026
Merged

psteinroe merged 1 commit into
mainfrom
t3/fix-cron-payload-priority

Conversation

@psteinroe

Copy link
Copy Markdown
Owner

Fixes follow-ups 1, 12 and 19 from the handover (originally noted in #89 and #90).

What

  1. Dynamic schedule payloads (fix(scheduling): overnight windows, unschedule races, maintenance jitter #89, fix(retries): make retried writes from running executions replay-safe #90). ctx.schedule(task, name, { cron }, { userId }) lost its payload in two ways:

    • _private_schedule_next_occurrence() inserted successors without payload or priority, so from the second occurrence on the payload was null and the priority was 0.
    • taskEventFor gave cron executions no payload at all, so even the first occurrence's payload never reached the handler. A dynamic occurrence arrived as { name: "<schedule name>" }, which matches none of the handler's event types.

    Successors now copy payload and priority, as they already did group and metadata. Occurrences of a dynamic schedule (dynamic:: dedupe keys) now run as { name: "pgconductor.invoke", payload }. That matches the type ctx.schedule checks the payload against, and the worker validates the payload against the task's schema, as it does for every invocation. Static cron triggers are unchanged: { name: "<trigger name>" }, no payload.

  2. Two schedule chains (fix(scheduling): overnight windows, unschedule races, maintenance jitter #89). If ctx.schedule() replaced a schedule while a running occurrence of the old schedule was inserting its successor, both chains survived. There were two ways this happened:

    • Sequential: the replacement commits between the occurrence's claim and its successor insert, and the successor is inserted anyway with the old cron expression.
    • Concurrent: neither statement sees the other's uncommitted row.

    The fix also has two parts:

    • The scheduling statement moved into _private_schedule(), a plpgsql function next to _private_unschedule(). It takes the queue row lock for update, then deletes and upserts in later statements. The successor path already takes that lock for share, so the two are now ordered.
    • _private_schedule_next_occurrence() inserts no successor while a live occurrence of the same schedule with a later scheduled time exists. It stores that occurrence's id as the step instead. It compares the dedupe key's timestamp, not run_at, because retries and sleeps move run_at.
  3. Docs (fix(retries): make retried writes from running executions replay-safe #90). testing.md said the next occurrence is scheduled after each run. It is scheduled when an occurrence starts, before its handler. cron.md now describes the event a dynamic occurrence receives and what replacing a running schedule does. AGENTS.md "Cron Schedules" is updated.

Behaviour change: handlers that told dynamic occurrences apart by event.name === "<schedule name>" now see "pgconductor.invoke". That shape was never in the handler's type, and the payload was never delivered.

Verification

Test first. All three new tests in cron-scheduling.test.ts failed on main:

  • every occurrence of a dynamic schedule gets its payload and priority: fake clock, two drained occurrences. Got { name: "user-u1", payload: undefined }, and the successor had no payload or priority.
  • a successor scheduled after the replacement is not inserted: left two pending occurrences.
  • a replacement concurrent with the successor removes it: holds the successor's transaction open and waits until the replacement either finishes or blocks on a lock. Left two pending occurrences.

The existing test cron task with failure still reschedules next execution caught a first version that compared run_at; the dedupe key timestamp fixes that.

just lint (oxlint plus typecheck) and just format are clean. pnpm test: 614 of 615 pass. The one failure is in wait-deadlines.test.ts (… settled after the deadline > times out), which fails the same way on main at 9f450da when run alone (2–3 of its 11 tests). See the follow-ups.

Follow-ups (out of scope)

  • wait-deadlines.test.ts is flaky on main. Since fix(testing): keep fake time running so event waits match under a set test clock #98, fake time keeps running, so the waiter's 1 s fake deadline (12:00:00 → 12:00:01) can pass during setup before the test sets the clock forward. The test then sees a completion instead of a timeout. This fits F13 (test suite).
  • Cron successors still don't carry trace_context, so each occurrence starts a new trace. I left that unchanged on purpose.

🤖 Generated with Claude Code

…n per schedule

Successors of a cron occurrence now copy its payload and priority, and occurrences
of a dynamic schedule run as pgconductor.invoke events with their payload, which
the worker validates like an invocation.

ctx.schedule() replaces a schedule in _private_schedule(), which holds the queue
row lock like _private_unschedule(). A running occurrence of the replaced schedule
inserts no successor once a later occurrence of its schedule is live, so only the
new chain survives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@psteinroe
psteinroe merged commit 78d7a9e into main Oct 21, 2026
11 checks passed
psteinroe added a commit that referenced this pull request Oct 26, 2026
Settlement now plans for one partition, so its cost no longer grows with
the number of queues (finding P10). Before, `buildReturnExecutions`
referenced the partitioned executions table about 25 times and none of
those references could be pruned. At 106 queues that cost ~520 ms of
planning for each of the first 6 calls on every connection and ~30 ms
per call after that. With Postgres' default `jit = on`, the statement's
estimated cost also crossed `jit_above_cost`, so every settle
JIT-compiled ~10.7k functions and took **~14 s**.

**One queue per statement**

`returnExecutions` now takes the worker's `queueName`, and results no
longer carry a `queue` each. Every reference to the executions table in
the statement is narrowed to that queue. The queue is inlined as an
escaped `E''` literal rather than bound as a parameter, because a cached
generic plan prunes partitions only by constants. So each queue gets its
own prepared statement, planned a few times per connection and then
served from the plan cache. With a parameter, Postgres kept choosing
custom plans: ~3.5 ms of planning on every call, plus one ~250 ms
generic plan per connection at 106 queues. A test covers queue names
with a quote and a backslash, and the escaping round-trips under both
`standard_conforming_strings` settings.

**Parent paths move into functions**

Waiting parents can be in any queue, so their lookups now run in two
plpgsql functions that settlement calls only when a settled execution
has a parent:
- `_private_complete_children()` stores results for waiting parents,
wakes them, and returns orphaned children (children no parent waits on
any more).
- `_private_fail_waiting_parents()` fails the chain of waiting ancestors
and returns them for dead-lettering, result waits and the cancel
cascade.

Their statements can't be pruned, so they use `plan_cache_mode =
force_generic_plan` (planned once per connection) and look rows up by
arrays of ids to stay on indexes: `jsonb_to_recordset` is always
estimated at 100 rows, which otherwise led to a seq scan of every
partition. Both functions read with fresh snapshots, so they run first
in `writes`, before the statement changes any row they read. That keeps
their view identical to the old single snapshot. The old existence check
on the parent in `inserted_children` is dropped because `valid_results`
already holds that row locked. Behaviour is unchanged otherwise: the
full suite passes unchanged apart from the call shape. Two new tests
cover the moved paths with parents in other queues: an orphaned child,
and a three-level failure that dead-letters the middle ancestor. Both
new tests also pass on `main`.

**Measurements**

The harness seeds ~1M executions: 93% completed, 3% failed, 3% pending,
1% suspended parents. It runs Postgres 15 with
`pg_stat_statements.track_planning`. Each settle is one completion plus
one retry, sent through `DatabaseClient` (postgres.js prepared
statements). Times below are DB-side from `pg_stat_statements`; wall
times on this machine were noisy.

| | before | after |
|---|---|---|
| 6 queues, first 6 calls per connection | 18–29 ms planning each |
3.5–4.5 ms planning each |
| 6 queues, steady state | 1.0–1.1 ms exec (cached generic plan) |
0.5–0.6 ms exec, no planning |
| 106 queues, first 6 calls per connection | ~520 ms planning each (~3.1
s per connection) | 3.5–7.7 ms planning each |
| 106 queues, steady state | 5.7 ms exec, ~30 ms wall (a generic plan
locks every partition) | 0.5–0.7 ms exec, no planning |
| 106 queues, `jit = on` | ~14 s per settle | 0.8 ms exec |

When every settled child wakes a parent in another queue (an
`invokeAndWait` scenario), median wall time per settle drops from 39 ms
to 9 ms at 106 queues and from 3.7 ms to 2.8 ms at 6 queues. At 106
queues the first call per connection takes ~170 ms (the functions'
generic plans), against ~600–900 ms for each of the first 3 calls
before.

Trade-off: each settled queue keeps its own cached plan of about 1 MB
per connection. Before, there was one plan per connection, but it grew
with the partition count: 2 MB at 6 queues, 16 MB at 106.

**Follow-ups (out of scope)**

- Foreign-key checks into executions from steps and signals are still
planned against every partition (P12, F11).
- Claim planning still grows with queue count (P11, F12). The claim's
estimated cost should be checked against `jit_above_cost` as well.
- `cron-scheduling.test.ts` › *cron task with failure still reschedules
next execution* timed out once in CI after the rebase onto #102 (no
future occurrence for ~12 s) and passed on re-run and in every local
run. Settlement is not involved; it may be a timing flake in #102's
successor check (F13).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant