Skip to content

perf(settlement): plan settlement for one partition - #103

Merged
psteinroe merged 1 commit into
mainfrom
perf/settlement-pruning
Oct 26, 2026
Merged

psteinroe merged 1 commit into
mainfrom
perf/settlement-pruning

Conversation

@psteinroe

@psteinroe psteinroe commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Settlement now plans for one partition, so its cost no longer grows with the number of queues (finding P10). Before, buildReturnExecutions referenced the partitioned executions table about 25 times and none of those references could be pruned. At 106 queues that cost ~520 ms of planning for each of the first 6 calls on every connection and ~30 ms per call after that. With Postgres' default jit = on, the statement's estimated cost also crossed jit_above_cost, so every settle JIT-compiled ~10.7k functions and took ~14 s.

One queue per statement

returnExecutions now takes the worker's queueName, and results no longer carry a queue each. Every reference to the executions table in the statement is narrowed to that queue. The queue is inlined as an escaped E'' literal rather than bound as a parameter, because a cached generic plan prunes partitions only by constants. So each queue gets its own prepared statement, planned a few times per connection and then served from the plan cache. With a parameter, Postgres kept choosing custom plans: ~3.5 ms of planning on every call, plus one ~250 ms generic plan per connection at 106 queues. A test covers queue names with a quote and a backslash, and the escaping round-trips under both standard_conforming_strings settings.

Parent paths move into functions

Waiting parents can be in any queue, so their lookups now run in two plpgsql functions that settlement calls only when a settled execution has a parent:

  • _private_complete_children() stores results for waiting parents, wakes them, and returns orphaned children (children no parent waits on any more).
  • _private_fail_waiting_parents() fails the chain of waiting ancestors and returns them for dead-lettering, result waits and the cancel cascade.

Their statements can't be pruned, so they use plan_cache_mode = force_generic_plan (planned once per connection) and look rows up by arrays of ids to stay on indexes: jsonb_to_recordset is always estimated at 100 rows, which otherwise led to a seq scan of every partition. Both functions read with fresh snapshots, so they run first in writes, before the statement changes any row they read. That keeps their view identical to the old single snapshot. The old existence check on the parent in inserted_children is dropped because valid_results already holds that row locked. Behaviour is unchanged otherwise: the full suite passes unchanged apart from the call shape. Two new tests cover the moved paths with parents in other queues: an orphaned child, and a three-level failure that dead-letters the middle ancestor. Both new tests also pass on main.

Measurements

The harness seeds ~1M executions: 93% completed, 3% failed, 3% pending, 1% suspended parents. It runs Postgres 15 with pg_stat_statements.track_planning. Each settle is one completion plus one retry, sent through DatabaseClient (postgres.js prepared statements). Times below are DB-side from pg_stat_statements; wall times on this machine were noisy.

before after
6 queues, first 6 calls per connection 18–29 ms planning each 3.5–4.5 ms planning each
6 queues, steady state 1.0–1.1 ms exec (cached generic plan) 0.5–0.6 ms exec, no planning
106 queues, first 6 calls per connection ~520 ms planning each (~3.1 s per connection) 3.5–7.7 ms planning each
106 queues, steady state 5.7 ms exec, ~30 ms wall (a generic plan locks every partition) 0.5–0.7 ms exec, no planning
106 queues, jit = on ~14 s per settle 0.8 ms exec

When every settled child wakes a parent in another queue (an invokeAndWait scenario), median wall time per settle drops from 39 ms to 9 ms at 106 queues and from 3.7 ms to 2.8 ms at 6 queues. At 106 queues the first call per connection takes ~170 ms (the functions' generic plans), against ~600–900 ms for each of the first 3 calls before.

Trade-off: each settled queue keeps its own cached plan of about 1 MB per connection. Before, there was one plan per connection, but it grew with the partition count: 2 MB at 6 queues, 16 MB at 106.

Follow-ups (out of scope)

@psteinroe
psteinroe force-pushed the perf/settlement-pruning branch from 5fb3c46 to 97fd369 Compare October 26, 2026 09:16
@psteinroe
psteinroe merged commit 9cbbd76 into main Oct 26, 2026
21 of 22 checks passed
@psteinroe
psteinroe deleted the perf/settlement-pruning branch October 26, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant