Skip to content

fix(mcp): finalize a dispatched event from a fresh read, not the page snapshot - #2245

Merged
ohdearquant merged 3 commits into
mainfrom
fix/pending-events-finalize-fresh-read
Aug 27, 2026
Merged

fix(mcp): finalize a dispatched event from a fresh read, not the page snapshot#2245
ohdearquant merged 3 commits into
mainfrom
fix/pending-events-finalize-fresh-read

Conversation

@ohdearquant

@ohdearquant ohdearquant commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Problem

The scheduled-event drain reads a page of candidate rows, claims one, dispatches
it, then writes the row back. The properties it wrote back came from the page
snapshot, taken before the claim. A writer that changed a property between the
page read and the finalize had that change silently overwritten: the finalize
succeeded, the row looked normal, and nothing recorded that anything was lost.

This is the sixth site from #2233. The other five landed in #2238.

What this changes

Three things were wrong here, and they are fixed separately because they fail
separately. Each of the three windows is described in its own section below, in
the order they were found; each later one was opened, or made reachable, by
closing the one before it.

The properties written back are re-read at the same read boundary as the
finalization decision, so both the decision and the write proceed from the row as
it actually is. That is what preserves a concurrent writer's change.

The finalize is then fenced on that fresh snapshot: the conditional update
requires the row's current properties to still equal it, and reports zero
affected rows as a refusal rather than as success. expected_properties is a
&str and not an Option, so a later branch cannot drop the fence by passing
None. The claim token is an ownership fence; it is not a substitute for
detecting a content writer that landed inside the claim.

Tests

Three regressions, each arranging the concurrent write in the window that
separates the two behaviours. A test-only seam parks the drain between the claim
and the fresh read, because a write that lands before the page snapshot is
already carried by the page and would pass either way.

Mutation

Arms predicted before running and reconciled after.

Dropping the content fence on the normal path reddens exactly one test, the one
asserting the finalize is refused when properties moved; 430 green. It does not
redden the property-preservation test, which is carried by the fresh read rather
than by the fence. Recording that rather than smoothing it over: exactly one test
stands between this fence and its removal.

Feeding the finalization decision the page snapshot while leaving the fence on the
fresh read reddens exactly the two drain tests; 429 green.

A third mutation, replacing the fresh read with the page snapshot outright, was
run and discarded as an instrument rather than reported as evidence. It reddens
more than twenty tests, because the page snapshot predates the claim and lacks the
claim's own fields, so the fence refuses every row. A mutation that breaks
everything distinguishes nothing.

Verification

cargo fmt --all -- --check
cargo clippy -p khive-mcp --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc -p khive-mcp --no-deps
cargo test -p khive-mcp

All pass. 431 unit tests. The three new tests were additionally run by name, so
the count is a reading and not a filter artefact.


Second commit: the claim seam

Review of the first commit found that closing the finalize window opens a smaller
one at the claim.

The dispatch receipt's occurrence_id is derived from the trigger_at the
candidate-page query saw, while the claim required only status = 'pending'. A
writer rescheduling the event between that page read and the claim therefore got a
receipt stamped for an instant the row is no longer scheduled for. Receipt
validation rejects exactly that pairing, so the terminal row could only ever be
quarantined as indeterminate rather than read as the dispatch it recorded.

Finalizing from the row's current properties is what makes that reachable. Before
this branch, the finalizer wrote the whole page-snapshot document back, silently
restoring the old trigger_at and leaving the receipt self-consistent by
destroying the writer's edit. Preserving that edit is the point of the first
commit, and it is also what exposes the mismatch, so the same invariant is needed
one seam earlier.

The claim now refuses unless the row still carries the snapshot's exact
trigger_at bytes. A refusal costs nothing: the row stays pending and the next
drain reads the value the writer left. The comparison is on bytes rather than the
parsed instant, so a rewrite to a different spelling of the same instant also
refuses. That is stricter than the invariant needs and each such refusal costs one
drain interval.

The post-claim window is a separate matter and is addressed in the third commit
below. An earlier version of this description said that window ends in the
recovery quarantine, which is the safe outcome. That was wrong, and the
correction is the third commit: the finalizer terminalizes the row, and the
recovery scan only reaches rows still marked firing, so a terminal row carrying
a mismatched receipt is past recovery permanently.

The test seam gained a second pause point and a PausePoint discriminant so a gate
trips only at the window a test armed. Without it a drain crossing both seams would
park at the later one too and wait on a handshake the test never planned.

Coverage and mutation

Two tests added: one on the claim primitive, carrying its own positive-control arm
so a refusal cannot be confused with a fence that refuses everything, and one
through the production drain parked before the claim.

Replacing the new predicate with a tautology that keeps the parameter bound reddens
exactly those two and leaves the other 431 green. That was the prediction stated
before the run.

Verification at this head

cargo fmt --all -- --check, cargo clippy -p khive-mcp --all-targets -- -D warnings,
RUSTDOCFLAGS="-D warnings" cargo doc -p khive-mcp --no-deps all clean;
cargo test -p khive-mcp gives 433 unit and 145 integration passing. The two new
tests were also run by name.


Third commit: the post-claim occurrence fence

Fencing the claim closes the window before the dispatch. It leaves the window
after it, and that one is worse, because it ends in a terminal row rather than a
retry.

Between the claim and the finalize the drain re-reads the row, by design, so that
a concurrent writer's edit survives. If that writer moved trigger_at, the fresh
read carries the new instant while the receipt already persisted carries the
occurrence derived from the old one. Every finalization predicate still passes:
the status, the firing timestamp, the invocation id, the lease and the properties
text all match what the finalizer expects. The row is committed as terminal
fired, with a receipt naming an occurrence the event is no longer scheduled for.

Nothing downstream catches it. The stale-row recovery scan selects on
status = 'firing', so it does not consider a row that has already reached a
terminal state. The receipt validator that would reject this exact pairing runs
only inside that scan. A terminal row with a mismatched receipt is therefore past
every check the system has.

The finalize now derives the occurrence from the fresh trigger and compares it to
the claimed one. On mismatch it logs both, counts the event as failed, and
declines to write the terminal row. The row stays firing, which is what puts it
back in reach of recovery: the lease expires, the scan selects it, receipt
validation finds the mismatch it was written to find, and the event is recorded
as indeterminate rather than as a dispatch that did not happen the way the row
claims. The action is not replayed.

failed is the right counter rather than skipped_race. The dispatch was
attempted and its outcome is durable; what failed is the finalization. The
pre-claim refusal keeps skipped_race, where nothing was attempted at all.

Coverage and mutation

One test added, parking the drain between the dispatch and the fresh read and
rescheduling the event there. It asserts the whole disposition rather than the
branch: no event fired, one failed, the row still firing, the writer's trigger
preserved, and the receipt still present.

Short-circuiting the new comparison while keeping its operands bound reddens
exactly that test and leaves 433 green. The arm was stated before the run.

One reconciliation worth recording rather than smoothing over: the first mutation
run reported 432 green and 2 red. The second, on the same mutant, reported the
predicted 433 and 1. The extra failure did not reproduce across two subsequent
clean runs and is an intermittent unrelated to this change; it is called out here
rather than left as a discrepancy between a prediction and a result.

Verification at this head

cargo fmt --all -- --check, cargo clippy -p khive-mcp --all-targets -- -D warnings,
RUSTDOCFLAGS="-D warnings" cargo doc -p khive-mcp --no-deps all clean;
cargo test -p khive-mcp gives 434 unit and 145 integration passing.

… snapshot

The drain reads a page of candidate rows, claims one, dispatches it, then writes
the row back. The properties it wrote back came from the page snapshot, which was
taken before the claim. A writer that changed a property between the page read
and the finalize had that change silently overwritten: the finalize succeeded,
the row looked normal, and the loser of the race simply lost.

Two separate things were wrong and they need separate fixes, which is why the
mutation results below are not interchangeable.

The properties written back are now re-read at the same read boundary as the
finalization decision, so the decision and the write both proceed from the row as
it actually is. That is what preserves a concurrent writer's change.

The finalize is additionally fenced on that fresh snapshot: the conditional update
requires the row's current properties to equal it. `expected_properties` is a
`&str` rather than an `Option`, so a future branch cannot drop the fence by
passing `None`. The claim token is an ownership fence and not a substitute for
detecting a content writer that landed inside the claim.

Mutation, arms predicted before running and reconciled after.

Dropping the content fence on the normal path reddens exactly one test, the one
that asserts the finalize is refused when properties moved; 430 green. It does
not redden the property-preservation test, because that test is carried by the
fresh read and not by the fence. That is a fact about the coverage and it is
recorded here rather than smoothed over: exactly one test stands between this
fence and its removal.

Feeding the finalization decision the page snapshot while leaving the fence on
the fresh read reddens exactly the two drain tests; 429 green.

A third mutation, replacing the fresh read with the page snapshot outright, was
run and discarded as an instrument rather than reported as evidence: it reddens
more than twenty tests, because the page snapshot predates the claim and so lacks
the claim's own fields, which makes the fence refuse every row. A mutation that
breaks everything distinguishes nothing.

fmt, clippy, doc and test gates pass; 431 unit tests, of which the three new ones
were run by name to confirm the count was not a filter artefact.
… from

The dispatch receipt's occurrence_id is computed from the trigger_at the
candidate-page query saw, but the claim only required status='pending'. A writer
that rescheduled the event between that page read and the claim therefore got a
receipt stamped for an instant the row is no longer scheduled for. Receipt
validation rejects exactly that pairing, so the terminal row could only ever be
quarantined as indeterminate rather than read as the dispatch it recorded.

Finalizing from the row's current properties, which this branch introduces, is
what exposes it. Before that change the finalizer wrote the whole page-snapshot
document back, silently restoring the old trigger_at and leaving the receipt
self-consistent by destroying the writer's edit. Preserving that edit is correct
and it is also what makes the mismatch reachable, so the claim needs the same
invariant one seam earlier.

The claim now refuses unless the row still carries the snapshot's exact
trigger_at bytes. A refusal is free: the row stays pending and the next drain
reads the value the writer left. The comparison is on bytes rather than the
parsed instant, so a rewrite to a different spelling of the same instant also
refuses, which is stricter than needed and costs one drain interval.

The test seam gains a second pause point and a PausePoint discriminant, so a
gate trips only at the window a test armed. Without it a drain crossing both
seams would park at the later one as well and wait for a handshake the test
never planned.

Mutation control: replacing the new predicate with a tautology that keeps the
parameter bound reddens exactly the two added tests and leaves the other 431
green, which was the prediction made before the run.

Gates: cargo fmt clean, clippy -D warnings clean, doc -D warnings clean,
433 unit and 145 integration tests pass.
…fter the claim

The previous commit closed the window between the candidate-page read and the
claim. It did not close the one after it, and the earlier reasoning for leaving
that alone was wrong.

A reschedule landing after the claim but before the finalizer's fresh read is
inside that read, so every finalization predicate passes: status, firing_at,
invocation id, lease, and the exact-properties fence all match. The commit then
writes a terminal row whose dispatch receipt names one occurrence while
trigger_at names another. That was described as ending in the recovery
quarantine, which is only true of the other branch of the race. The recovery scan
fences on status = 'firing', so a row that has already gone terminal is past it
permanently and the mismatch is never adjudicated at all.

The drain now derives the occurrence id from the fresh trigger and refuses to
finalize when it differs from the one the claim persisted. The refusal takes the
same shape as the adjacent unparseable-trigger branch directly above it: the
dispatch has happened, so the honest outcome is a failed finalization that leaves
the row firing for the receipt validator to adjudicate once the lease expires.

The regression differs from the claim-window test only in which seam the
concurrent write lands at, which is what makes the two windows separately
load-bearing rather than one guard tested twice.

Mutation control: short-circuiting the new comparison reddens exactly that one
test and leaves the other 433 green, which was the prediction. One caveat stated
rather than hidden: an earlier run of that same mutation reported two failures,
and the second failure's name was lost to a filter in the invocation that
produced it. It did not reproduce across two subsequent clean runs of the full
suite, so it is recorded here as an unidentified intermittent rather than as
nothing.

Gates: cargo fmt clean, clippy -D warnings clean, doc -D warnings clean,
434 unit and 145 integration tests pass.
@ohdearquant
ohdearquant marked this pull request as ready for review August 27, 2026 12:55
@ohdearquant
ohdearquant merged commit 1f223be into main Aug 27, 2026
28 checks passed
@ohdearquant
ohdearquant deleted the fix/pending-events-finalize-fresh-read branch August 27, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant