docs(ha): correct split-brain claims the fence does not actually deliver - #129
Merged
Conversation
An HA re-check of the leadership-lease construct confirmed the lease algebra, the staged-queue handoffs and the store finalizer are correct as designed, but found prose across code and docs asserting guarantees the fence does not provide. This corrects the statements only -- the four .py files are AST-identical (docstrings and comments), ruff format/check clean. - The H1 epoch fence guards the CLAIM and nothing else. Post-claim disposition writes (mark_done/mark_failed/dead_letter_now/complete_with_response) resolve by id with no epoch, owner or status precondition, and the in-claim stranded-lease reclaim is a separate UNGUARDED statement that commits even when the guarded claim matches zero rows. Scoped the token accordingly. - "Fenced" does not imply "stopped processing". _check_fence flips a boolean and cancels nothing; graph teardown is not budgeted against the fence-to-expiry margin, and that margin is itself short by the renew round trip plus up to one fence tick. The validator checks ordering, never margin. - recover_inflight_on_promotion and the SQL Server reset_stale_inflight were justified by "the prior leader has stopped processing" -- a compensating control resting on a false premise (CLAUDE.md 11). Replaced with the sound directional argument: re-pending risks a DUPLICATE, which at-least-once permits and idempotent outbounds absorb; not re-pending risks a STRAND, which the count-and-log invariant forbids. - The row lease is stamped once at claim and is NEVER renewed. settings.py, postgres.py (twice) and the sweep's own no-theft argument all cited a renew timer with no implementation anywhere in the store, and the sizing rule derived from it was wrong: size lease_ttl_seconds against the longest legitimate claim-to-terminal-write hold, not a renew interval. - The derived leader_node_id is a one-sided cross-node wall-clock comparison with no lower bound, so a fast-clocked node wins the freshness pick even after it dies -- including masking the console's "cluster has no leader" health check, which keys off that field being absent. ADR 0008 and ADR 0056 carry inline CORRECTION errata rather than rewritten reasoning; ADR 0056's VIP release budget was derived from the uncorrected margin. Also replaces the exhaustive "leadership is DB-clock, row leases are wall-clock" pairing with an "at least" formulation (CLAUDE.md 11) -- the nodes.last_seen heartbeat was the omitted third.
bc9ccd7 corrected two of the three places justifying an on-promotion recovery write with "the prior leader has stopped processing" -- postgres.py's recover_inflight_on_promotion and engine.py's SQL Server reset_stale_inflight call. It missed the third, ~100 lines above the second: the comment on the leader-maintenance branch still read "the old leader self-fenced before its lease expired, so re-pending its in-flight rows can't steal from a live processor." That enumeration being incomplete is itself the defect class the original commit was fixing (CLAUDE.md 11 -- a completeness claim is a liability), so the replacement states the reason positively rather than listing sites: the fence stops the node REPORTING leader, not working, so the reset is correct because it errs toward a duplicate (permitted) rather than a strand (forbidden). Also records what the re-check turned up while verifying this site: SQL Server has NO periodic in-flight recovery at all. reclaim_expired_leases is defined on the Postgres store alone (0 definitions in sqlserver.py) and the sweep is gated on hasattr() at engine.py:1062, so a row left INFLIGHT outside a promotion -- including by the deliberate teardown path in stage_dispatcher, which leaves a cancelled prefix INFLIGHT for exactly this recovery -- has nothing periodic to reclaim it and is stranded until the next promotion or restart. Postgres bounds the same case at roughly reclaim_interval + lease_ttl. Tracked in ADR 0157. Comment-only: AST-identical with docstrings stripped; ruff clean.
wshallwshall
enabled auto-merge (squash)
August 2, 2026 02:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
An HA re-check of the active-passive leadership-lease construct. The construct held — the lease
algebra, the staged-queue handoffs and the store finalizer are correct as designed, and the classic
node-clock split-brain hole is closed (expiry is evaluated on the DB clock on both backends). What the
re-check found instead was prose across code and docs asserting guarantees the fence does not
provide. This PR corrects the statements. No behavior changes.
Five classes of defect, each corrected at every site:
The H1 epoch fence guards the CLAIM and nothing else. Post-claim disposition writes
(
mark_done/mark_failed/dead_letter_now/complete_with_response) resolve their row byidalone — no epoch, no owner, no status precondition — and the cross-owner stranded-lease reclaiminside the FIFO claim transaction is a separate, unguarded statement that commits even when the
guarded claim matches zero rows.
current_epoch's docstring andCLUSTERING.mdnow scope the tokenexplicitly instead of implying a general write fence.
"Fenced" does not imply "stopped processing."
_check_fenceflips an in-memory boolean andcancels nothing. Nothing budgets graph teardown against the fence-to-expiry margin, and that margin
is itself smaller than
ttl - fence: the fence baseline is stamped after the renew round tripreturns while the expiry is stamped on the DB clock at statement execution, plus up to one
_fence_tickof detection lag._fence_orderingvalidates the ordering only, never a margin.Two compensating controls rested on a false premise (CLAUDE.md §11).
recover_inflight_on_promotionand the SQL Server on-promotionreset_stale_inflightwere bothjustified by "the prior leader has stopped processing," which (2) shows does not follow. Replaced
with the argument that actually holds and is directional: re-pending a row whose sender is still
winding down risks a duplicate, which at-least-once permits and idempotent outbounds absorb;
not re-pending it risks a strand, which the count-and-log invariant forbids. Erring toward
duplication is the invariant-safe direction.
The row lease is stamped once at claim and is NEVER renewed.
settings.py,postgres.py(twice) and the sweep's own no-theft argument all cited a "renew timer" that has no implementation
anywhere in the store (
grep -i renew messagefoundry/store/returns two docstring lines and nocode). The sizing rule derived from it was wrong: size
lease_ttl_secondsagainst the longestlegitimate claim-to-terminal-write hold, not against a renew interval.
The derived
leader_node_idis a one-sided cross-node wall-clock comparison. The freshness testhas an upper bound only, so a row stamped by a fast-clocked node has a negative age, passes, and —
being the largest
last_seen— wins the pick, including after that node hard-crashes. Documentedthat it is observability and the lease is authoritative, and that the web console's "cluster has no
leader" health check keys off this field being absent and can therefore be masked by it.
Also replaces the exhaustive "leadership is DB-clock, row leases are wall-clock" pairing with an
"at least" formulation (CLAUDE.md §11 — a completeness claim is a liability);
nodes.last_seenwasthe omitted third clock-sensitive value.
ADR 0008 and ADR 0056 carry inline
⚠️ CORRECTIONerrata in the established repo style rather thanrewritten reasoning — ADR 0056's VIP release budget was derived from the uncorrected margin, so that
derivation needed flagging where a reader will meet it.
Type of change
docs/adr/is included or linked)Reviewer notes
Verified zero behavior change, not merely asserted. The four
.pyfiles are AST-identical totheir parents with docstrings stripped — checked by parsing both revisions, removing every module /
class / function docstring, and comparing
ast.dumpoutput. Every changed line is a comment ordocstring.
On the "gates pass locally" checkbox. Now genuinely green, after a correction worth recording. This
worktree had been created without a
.venv, so at first submission onlyruff check/ruff format --checkand the pre-commit suite (ledger gate, customer/PHI leak guard, secret detection,bandit) had run, and the box was left unticked rather than assumed. Since bootstrapped:
ruff check .— All checks passedruff format --check .— 999 files already formattedmypy messagefoundry— Success: no issues found in 258 source filesAn intermediate
mypyrun reported 21 errors; all of them were unresolved imports and now-unusedtype: ignorecomments in four optional-extra shims (auth/webauthn.py,parsing/fhir/_deps.py,transports/dicom.py,parsing/dicom/_deps.py) because the venv had only[dev,harness]. With thefull extra set installed they are gone. Noting it so the number isn't rediscovered as a regression.
pytestis not run locally for this PR — CI ran the full matrix green on this exact commit(ubuntu-latest, windows-2022, windows-2025, both SQL Server legs, Postgres), which is stronger evidence
than a single local run. The AST-identity proof above is the reason a docs-only diff cannot move any of
these; it would not license skipping them for a code change.
One CI note for the reviewer. The first run failed on
sql server (store + connector) 2022attests/test_stage_dispatcher.py::test_adr0070_9_content_retry_is_not_an_infra_fault[sqlserver]. That isa flake, demonstrated rather than assumed: the 2025 leg passed on the same commit, a re-run of the
identical SHA passed, the diff is AST-identical, and the failing assertion is
_wait_until(
test_stage_dispatcher.py:356) pollingloop.time()against a hardcoded 8.0 s wall-clock budgetwhile the dispatcher under test runs on an injected
ManualClock— every logic assertion in that sameloop passed. Flagged to the session working the CI timeouts, since a fixed real-time bound that has
drifted out of proportion to the work is the same defect class as the 26:00 Windows step cap.
Not included, deliberately. The re-check also produced findings that need code changes, and those
are held back for separate PRs so this one stays a pure documentation correction:
symptoms. They turn on two design questions — whether post-claim writes carry the fencing token, and
whether demotion gets a deadline (and what happens to an inbound that cannot meet it;
FileSource.stop()has no cancel and no timeout) — so they are ADR-gated, not a drive-by fix.
mark_failed, whichspends the attempt the claim already incremented and, under a finite
retry_max_attempts,dead-letters a row that was never sent. It is blocked here only by a parallel session holding
wiring_runner.py; it ships with its own test.No backlog item is claimed by this PR, so the
Implements: BACKLOG #line is omitted per the template.Checklist
auth or graph-model behavior is altered (AST-identical; see Reviewer notes). The behavioral
follow-ups it identifies are held for an ADR, per GOVERNANCE.md.
commit's added lines, and by the repo's own leak-guard pre-commit hook.
ruff check .,ruff format --check .andmypy messagefoundryare clean(258 source files, no issues).
pytestdeferred to CI, which ran the full matrix green on thisexact commit — see Reviewer notes.
GUI/web-framework imports in the engine packages; no Black.
docs/BACKLOG.mditem — N/A, no backlog item is claimed.🤖 Generated with Claude Code