Repository navigation
Conversation
…s, set-based invoke_batch - steps, signals and wait subscriptions no longer have foreign keys into executions; every path that deletes executions deletes them with _private_delete_execution_data() - drop the per-partition (task_key) index and the unused event wait match index - invoke_batch inserts a batch without dedupe in one statement - completion compares signal_count with its snapshot value instead of counting all signals - dispatch probes prefix lengths between the field's shortest and longest prefix term
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes per-insert and per-row work from the hot tables. This covers review findings P12, P13, P14/S3, P5 and P6, plus the unused event wait index that #98 flagged.
P12: no foreign keys into executions. Steps, signals and wait subscriptions no longer reference
_private_executions; that RI check ran on every step save, signal and wait registration. Every path that deletes executions now calls the new_private_delete_execution_data(ids)after the delete. That covers retention, settlement withremoveOnComplete/removeOnFail0,cancel_execution, superseding ininvoke(),_private_fail_waiting_parents, the three schedule-removal paths, and the queue drop, which deletes by queue and now drops the partition without detaching it. The helper is a volatile SQL function, so each of its statements takes a fresh snapshot. That means it also finds rows committed while the deleting statement waited for the row lock, which is the guarantee the cascade gave. Every writer of those rows already locks the execution row and writes nothing once it is gone. The one exception wasclearWaitingState, which stored the timeout step without a lock; it now inserts the step only after its update has locked the parent. AGENTS.md documents the invariant.P14/S3: dropped the per-partition
(task_key)index andidx_custom_event_subscription_wait_match. I re-checked this against #91. Every query that filters executions bytask_keyuses another index: the PK, the schedule index, the partial dedupe index (all of those queries filtersingleton_on is null) or the key index. On base, thetask_keyindex was only picked on the internal queue, because of stale stats. With real stats, dispatch uses the PK. Dispatch joins subscriptions by id or(event_key, required_field_count), soidx_custom_event_subscription_wait_matchhad 0 scans.P13:
invoke_batchwithout dedupe runs as one insert. When no spec hasdedupe_keyordedupe_seconds, the batch is oneinsert … selectfromunnest(specs) with ordinality. The ids are generated in a materialized CTE and returned in input order, and_private_check_metadataruns per spec first. Batches with dedupe keep the loop.P5 / P6. On completion, settlement now compares the locked row's
signal_countwith an unlocked read of the same row (its snapshot value) instead of counting the execution's entire signal history. Each signal insert increments the counter in the same transaction, so the two are equivalent. Prefix matching in dispatch probes only the lengths between the field's shortest and longest prefix term (two index probes for the bounds) instead of collecting distinct lengths over all of its terms.S2 (O2-04, the redundant parent check in
inserted_children) was already removed by #103.Measurements
Postgres 17 database with 1M executions in 6 queues, 200k steps, 300k signals, 50k event waits and 102k subscriptions, including 100k prefix terms on one field. The numbers are mean execution ms from
pg_stat_statements, as the median of 3 base/new runs interleaved on a shared, noisy machine. The harness is outside the repo, in the task state dir.invoke_batchof 100, no dedupesignal_executioninvokeThe
(task_key)index drop did not reproduce the review's −38% invoke insert: a 100k-row bulk insert took about 2.5 s with and without it, dominated by the column defaults. It still removes one btree per queue from every insert and delete. A sparse prefix field pays for one probe per length in the range, so it is slightly slower; a dense one no longer scans its whole term population.Tests
execution-data.test.tscovers every deletion path: the deleted execution's steps, signals and waits are gone and another execution's rows are kept. Two concurrency tests use deterministic lock barriers, and both fail against naive implementations (verified by swapping them in). One commits a step while settlement waits for the row lock, and the step must still be deleted. The other hasclearWaitingStaterun against an execution being superseded, and it must store no step. Newinvoke_batchtests cover field mapping, id order and per-spec metadata checks.just lint,just format,pnpm typecheckand the fullpnpm test(631 tests) pass.