fix(mcp): finalize a dispatched event from a fresh read, not the page snapshot - #2245
Merged
Merged
Conversation
… snapshot The drain reads a page of candidate rows, claims one, dispatches it, then writes the row back. The properties it wrote back came from the page snapshot, which was taken before the claim. A writer that changed a property between the page read and the finalize had that change silently overwritten: the finalize succeeded, the row looked normal, and the loser of the race simply lost. Two separate things were wrong and they need separate fixes, which is why the mutation results below are not interchangeable. The properties written back are now re-read at the same read boundary as the finalization decision, so the decision and the write both proceed from the row as it actually is. That is what preserves a concurrent writer's change. The finalize is additionally fenced on that fresh snapshot: the conditional update requires the row's current properties to equal it. `expected_properties` is a `&str` rather than an `Option`, so a future branch cannot drop the fence by passing `None`. The claim token is an ownership fence and not a substitute for detecting a content writer that landed inside the claim. Mutation, arms predicted before running and reconciled after. Dropping the content fence on the normal path reddens exactly one test, the one that asserts the finalize is refused when properties moved; 430 green. It does not redden the property-preservation test, because that test is carried by the fresh read and not by the fence. That is a fact about the coverage and it is recorded here rather than smoothed over: exactly one test stands between this fence and its removal. Feeding the finalization decision the page snapshot while leaving the fence on the fresh read reddens exactly the two drain tests; 429 green. A third mutation, replacing the fresh read with the page snapshot outright, was run and discarded as an instrument rather than reported as evidence: it reddens more than twenty tests, because the page snapshot predates the claim and so lacks the claim's own fields, which makes the fence refuse every row. A mutation that breaks everything distinguishes nothing. fmt, clippy, doc and test gates pass; 431 unit tests, of which the three new ones were run by name to confirm the count was not a filter artefact.
… from The dispatch receipt's occurrence_id is computed from the trigger_at the candidate-page query saw, but the claim only required status='pending'. A writer that rescheduled the event between that page read and the claim therefore got a receipt stamped for an instant the row is no longer scheduled for. Receipt validation rejects exactly that pairing, so the terminal row could only ever be quarantined as indeterminate rather than read as the dispatch it recorded. Finalizing from the row's current properties, which this branch introduces, is what exposes it. Before that change the finalizer wrote the whole page-snapshot document back, silently restoring the old trigger_at and leaving the receipt self-consistent by destroying the writer's edit. Preserving that edit is correct and it is also what makes the mismatch reachable, so the claim needs the same invariant one seam earlier. The claim now refuses unless the row still carries the snapshot's exact trigger_at bytes. A refusal is free: the row stays pending and the next drain reads the value the writer left. The comparison is on bytes rather than the parsed instant, so a rewrite to a different spelling of the same instant also refuses, which is stricter than needed and costs one drain interval. The test seam gains a second pause point and a PausePoint discriminant, so a gate trips only at the window a test armed. Without it a drain crossing both seams would park at the later one as well and wait for a handshake the test never planned. Mutation control: replacing the new predicate with a tautology that keeps the parameter bound reddens exactly the two added tests and leaves the other 431 green, which was the prediction made before the run. Gates: cargo fmt clean, clippy -D warnings clean, doc -D warnings clean, 433 unit and 145 integration tests pass.
…fter the claim The previous commit closed the window between the candidate-page read and the claim. It did not close the one after it, and the earlier reasoning for leaving that alone was wrong. A reschedule landing after the claim but before the finalizer's fresh read is inside that read, so every finalization predicate passes: status, firing_at, invocation id, lease, and the exact-properties fence all match. The commit then writes a terminal row whose dispatch receipt names one occurrence while trigger_at names another. That was described as ending in the recovery quarantine, which is only true of the other branch of the race. The recovery scan fences on status = 'firing', so a row that has already gone terminal is past it permanently and the mismatch is never adjudicated at all. The drain now derives the occurrence id from the fresh trigger and refuses to finalize when it differs from the one the claim persisted. The refusal takes the same shape as the adjacent unparseable-trigger branch directly above it: the dispatch has happened, so the honest outcome is a failed finalization that leaves the row firing for the receipt validator to adjudicate once the lease expires. The regression differs from the claim-window test only in which seam the concurrent write lands at, which is what makes the two windows separately load-bearing rather than one guard tested twice. Mutation control: short-circuiting the new comparison reddens exactly that one test and leaves the other 433 green, which was the prediction. One caveat stated rather than hidden: an earlier run of that same mutation reported two failures, and the second failure's name was lost to a filter in the invocation that produced it. It did not reproduce across two subsequent clean runs of the full suite, so it is recorded here as an unidentified intermittent rather than as nothing. Gates: cargo fmt clean, clippy -D warnings clean, doc -D warnings clean, 434 unit and 145 integration tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The scheduled-event drain reads a page of candidate rows, claims one, dispatches
it, then writes the row back. The properties it wrote back came from the page
snapshot, taken before the claim. A writer that changed a property between the
page read and the finalize had that change silently overwritten: the finalize
succeeded, the row looked normal, and nothing recorded that anything was lost.
This is the sixth site from #2233. The other five landed in #2238.
What this changes
Three things were wrong here, and they are fixed separately because they fail
separately. Each of the three windows is described in its own section below, in
the order they were found; each later one was opened, or made reachable, by
closing the one before it.
The properties written back are re-read at the same read boundary as the
finalization decision, so both the decision and the write proceed from the row as
it actually is. That is what preserves a concurrent writer's change.
The finalize is then fenced on that fresh snapshot: the conditional update
requires the row's current properties to still equal it, and reports zero
affected rows as a refusal rather than as success.
expected_propertiesis a&strand not anOption, so a later branch cannot drop the fence by passingNone. The claim token is an ownership fence; it is not a substitute fordetecting a content writer that landed inside the claim.
Tests
Three regressions, each arranging the concurrent write in the window that
separates the two behaviours. A test-only seam parks the drain between the claim
and the fresh read, because a write that lands before the page snapshot is
already carried by the page and would pass either way.
Mutation
Arms predicted before running and reconciled after.
Dropping the content fence on the normal path reddens exactly one test, the one
asserting the finalize is refused when properties moved; 430 green. It does not
redden the property-preservation test, which is carried by the fresh read rather
than by the fence. Recording that rather than smoothing it over: exactly one test
stands between this fence and its removal.
Feeding the finalization decision the page snapshot while leaving the fence on the
fresh read reddens exactly the two drain tests; 429 green.
A third mutation, replacing the fresh read with the page snapshot outright, was
run and discarded as an instrument rather than reported as evidence. It reddens
more than twenty tests, because the page snapshot predates the claim and lacks the
claim's own fields, so the fence refuses every row. A mutation that breaks
everything distinguishes nothing.
Verification
All pass. 431 unit tests. The three new tests were additionally run by name, so
the count is a reading and not a filter artefact.
Second commit: the claim seam
Review of the first commit found that closing the finalize window opens a smaller
one at the claim.
The dispatch receipt's
occurrence_idis derived from thetrigger_atthecandidate-page query saw, while the claim required only
status = 'pending'. Awriter rescheduling the event between that page read and the claim therefore got a
receipt stamped for an instant the row is no longer scheduled for. Receipt
validation rejects exactly that pairing, so the terminal row could only ever be
quarantined as indeterminate rather than read as the dispatch it recorded.
Finalizing from the row's current properties is what makes that reachable. Before
this branch, the finalizer wrote the whole page-snapshot document back, silently
restoring the old
trigger_atand leaving the receipt self-consistent bydestroying the writer's edit. Preserving that edit is the point of the first
commit, and it is also what exposes the mismatch, so the same invariant is needed
one seam earlier.
The claim now refuses unless the row still carries the snapshot's exact
trigger_atbytes. A refusal costs nothing: the row stays pending and the nextdrain reads the value the writer left. The comparison is on bytes rather than the
parsed instant, so a rewrite to a different spelling of the same instant also
refuses. That is stricter than the invariant needs and each such refusal costs one
drain interval.
The post-claim window is a separate matter and is addressed in the third commit
below. An earlier version of this description said that window ends in the
recovery quarantine, which is the safe outcome. That was wrong, and the
correction is the third commit: the finalizer terminalizes the row, and the
recovery scan only reaches rows still marked
firing, so a terminal row carryinga mismatched receipt is past recovery permanently.
The test seam gained a second pause point and a
PausePointdiscriminant so a gatetrips only at the window a test armed. Without it a drain crossing both seams would
park at the later one too and wait on a handshake the test never planned.
Coverage and mutation
Two tests added: one on the claim primitive, carrying its own positive-control arm
so a refusal cannot be confused with a fence that refuses everything, and one
through the production drain parked before the claim.
Replacing the new predicate with a tautology that keeps the parameter bound reddens
exactly those two and leaves the other 431 green. That was the prediction stated
before the run.
Verification at this head
cargo fmt --all -- --check,cargo clippy -p khive-mcp --all-targets -- -D warnings,RUSTDOCFLAGS="-D warnings" cargo doc -p khive-mcp --no-depsall clean;cargo test -p khive-mcpgives 433 unit and 145 integration passing. The two newtests were also run by name.
Third commit: the post-claim occurrence fence
Fencing the claim closes the window before the dispatch. It leaves the window
after it, and that one is worse, because it ends in a terminal row rather than a
retry.
Between the claim and the finalize the drain re-reads the row, by design, so that
a concurrent writer's edit survives. If that writer moved
trigger_at, the freshread carries the new instant while the receipt already persisted carries the
occurrence derived from the old one. Every finalization predicate still passes:
the status, the firing timestamp, the invocation id, the lease and the properties
text all match what the finalizer expects. The row is committed as terminal
fired, with a receipt naming an occurrence the event is no longer scheduled for.Nothing downstream catches it. The stale-row recovery scan selects on
status = 'firing', so it does not consider a row that has already reached aterminal state. The receipt validator that would reject this exact pairing runs
only inside that scan. A terminal row with a mismatched receipt is therefore past
every check the system has.
The finalize now derives the occurrence from the fresh trigger and compares it to
the claimed one. On mismatch it logs both, counts the event as failed, and
declines to write the terminal row. The row stays
firing, which is what puts itback in reach of recovery: the lease expires, the scan selects it, receipt
validation finds the mismatch it was written to find, and the event is recorded
as indeterminate rather than as a dispatch that did not happen the way the row
claims. The action is not replayed.
failedis the right counter rather thanskipped_race. The dispatch wasattempted and its outcome is durable; what failed is the finalization. The
pre-claim refusal keeps
skipped_race, where nothing was attempted at all.Coverage and mutation
One test added, parking the drain between the dispatch and the fresh read and
rescheduling the event there. It asserts the whole disposition rather than the
branch: no event fired, one failed, the row still
firing, the writer's triggerpreserved, and the receipt still present.
Short-circuiting the new comparison while keeping its operands bound reddens
exactly that test and leaves 433 green. The arm was stated before the run.
One reconciliation worth recording rather than smoothing over: the first mutation
run reported 432 green and 2 red. The second, on the same mutant, reported the
predicted 433 and 1. The extra failure did not reproduce across two subsequent
clean runs and is an intermittent unrelated to this change; it is called out here
rather than left as a discrepancy between a prediction and a result.
Verification at this head
cargo fmt --all -- --check,cargo clippy -p khive-mcp --all-targets -- -D warnings,RUSTDOCFLAGS="-D warnings" cargo doc -p khive-mcp --no-depsall clean;cargo test -p khive-mcpgives 434 unit and 145 integration passing.