Skip to content

docs(ha): correct split-brain claims the fence does not actually deliver - #129

Merged
wshallwshall merged 4 commits into
mainfrom
claude/ha-construct-recheck-6070ba
Aug 2, 2026
Merged

docs(ha): correct split-brain claims the fence does not actually deliver#129
wshallwshall merged 4 commits into
mainfrom
claude/ha-construct-recheck-6070ba

Conversation

@wshallwshall

@wshallwshall wshallwshall commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

What this changes

An HA re-check of the active-passive leadership-lease construct. The construct held — the lease
algebra, the staged-queue handoffs and the store finalizer are correct as designed, and the classic
node-clock split-brain hole is closed (expiry is evaluated on the DB clock on both backends). What the
re-check found instead was prose across code and docs asserting guarantees the fence does not
provide
. This PR corrects the statements. No behavior changes.

Five classes of defect, each corrected at every site:

  1. The H1 epoch fence guards the CLAIM and nothing else. Post-claim disposition writes
    (mark_done / mark_failed / dead_letter_now / complete_with_response) resolve their row by
    id alone — no epoch, no owner, no status precondition — and the cross-owner stranded-lease reclaim
    inside the FIFO claim transaction is a separate, unguarded statement that commits even when the
    guarded claim matches zero rows. current_epoch's docstring and CLUSTERING.md now scope the token
    explicitly instead of implying a general write fence.

  2. "Fenced" does not imply "stopped processing." _check_fence flips an in-memory boolean and
    cancels nothing. Nothing budgets graph teardown against the fence-to-expiry margin, and that margin
    is itself smaller than ttl - fence: the fence baseline is stamped after the renew round trip
    returns while the expiry is stamped on the DB clock at statement execution, plus up to one
    _fence_tick of detection lag. _fence_ordering validates the ordering only, never a margin.

  3. Two compensating controls rested on a false premise (CLAUDE.md §11).
    recover_inflight_on_promotion and the SQL Server on-promotion reset_stale_inflight were both
    justified by "the prior leader has stopped processing," which (2) shows does not follow. Replaced
    with the argument that actually holds and is directional: re-pending a row whose sender is still
    winding down risks a duplicate, which at-least-once permits and idempotent outbounds absorb;
    not re-pending it risks a strand, which the count-and-log invariant forbids. Erring toward
    duplication is the invariant-safe direction.

  4. The row lease is stamped once at claim and is NEVER renewed. settings.py, postgres.py
    (twice) and the sweep's own no-theft argument all cited a "renew timer" that has no implementation
    anywhere in the store (grep -i renew messagefoundry/store/ returns two docstring lines and no
    code). The sizing rule derived from it was wrong: size lease_ttl_seconds against the longest
    legitimate claim-to-terminal-write hold, not against a renew interval.

  5. The derived leader_node_id is a one-sided cross-node wall-clock comparison. The freshness test
    has an upper bound only, so a row stamped by a fast-clocked node has a negative age, passes, and —
    being the largest last_seen — wins the pick, including after that node hard-crashes. Documented
    that it is observability and the lease is authoritative, and that the web console's "cluster has no
    leader" health check keys off this field being absent and can therefore be masked by it.

Also replaces the exhaustive "leadership is DB-clock, row leases are wall-clock" pairing with an
"at least" formulation (CLAUDE.md §11 — a completeness claim is a liability); nodes.last_seen was
the omitted third clock-sensitive value.

ADR 0008 and ADR 0056 carry inline ⚠️ CORRECTION errata in the established repo style rather than
rewritten reasoning — ADR 0056's VIP release budget was derived from the uncorrected margin, so that
derivation needed flagging where a reader will meet it.

Type of change

  • Documentation
  • Bug fix (a test reproducing the bug is included)
  • New Connection/transport or example Router/Handler
  • Refactor / internal change
  • Architecture change (an ADR under docs/adr/ is included or linked)

Reviewer notes

Verified zero behavior change, not merely asserted. The four .py files are AST-identical to
their parents with docstrings stripped — checked by parsing both revisions, removing every module /
class / function docstring, and comparing ast.dump output. Every changed line is a comment or
docstring.

On the "gates pass locally" checkbox. Now genuinely green, after a correction worth recording. This
worktree had been created without a .venv, so at first submission only ruff check / ruff format --check and the pre-commit suite (ledger gate, customer/PHI leak guard, secret detection,
bandit) had run, and the box was left unticked rather than assumed. Since bootstrapped:

  • ruff check .All checks passed
  • ruff format --check .999 files already formatted
  • mypy messagefoundrySuccess: no issues found in 258 source files

An intermediate mypy run reported 21 errors; all of them were unresolved imports and now-unused
type: ignore comments in four optional-extra shims (auth/webauthn.py, parsing/fhir/_deps.py,
transports/dicom.py, parsing/dicom/_deps.py) because the venv had only [dev,harness]. With the
full extra set installed they are gone. Noting it so the number isn't rediscovered as a regression.

pytest is not run locally for this PR — CI ran the full matrix green on this exact commit
(ubuntu-latest, windows-2022, windows-2025, both SQL Server legs, Postgres), which is stronger evidence
than a single local run. The AST-identity proof above is the reason a docs-only diff cannot move any of
these; it would not license skipping them for a code change.

One CI note for the reviewer. The first run failed on sql server (store + connector) 2022 at
tests/test_stage_dispatcher.py::test_adr0070_9_content_retry_is_not_an_infra_fault[sqlserver]. That is
a flake, demonstrated rather than assumed: the 2025 leg passed on the same commit, a re-run of the
identical SHA passed, the diff is AST-identical, and the failing assertion is _wait_until
(test_stage_dispatcher.py:356) polling loop.time() against a hardcoded 8.0 s wall-clock budget
while the dispatcher under test runs on an injected ManualClock — every logic assertion in that same
loop passed. Flagged to the session working the CI timeouts, since a fixed real-time bound that has
drifted out of proportion to the work is the same defect class as the 26:00 Windows step cap.

Not included, deliberately. The re-check also produced findings that need code changes, and those
are held back for separate PRs so this one stays a pure documentation correction:

  • The unfenced disposition writes and the unbounded demotion stop are one root cause with several
    symptoms. They turn on two design questions — whether post-claim writes carry the fencing token, and
    whether demotion gets a deadline (and what happens to an inbound that cannot meet it; FileSource.stop()
    has no cancel and no timeout) — so they are ADR-gated, not a drive-by fix.
  • A separate small fix: the "leadership lost before send" path re-queues through mark_failed, which
    spends the attempt the claim already incremented and, under a finite retry_max_attempts,
    dead-letters a row that was never sent. It is blocked here only by a parallel session holding
    wiring_runner.py; it ships with its own test.

No backlog item is claimed by this PR, so the Implements: BACKLOG # line is omitted per the template.

Checklist

  • I have read CONTRIBUTING.md and will agree to the CLA (the bot records it).
  • Touches reliability-invariant documentation only — no invariant, store/queue, staged-pipeline,
    auth or graph-model behavior is altered (AST-identical; see Reviewer notes). The behavioral
    follow-ups it identifies are held for an ADR, per GOVERNANCE.md.
  • No real PHI or customer data anywhere in the diff or commit message — verified by scanning the
    commit's added lines, and by the repo's own leak-guard pre-commit hook.
  • Tests added/updated for new behavior — N/A, there is no new behavior.
  • Any new dependency verified — N/A, no dependency change.
  • Gates pass locally: ruff check ., ruff format --check . and mypy messagefoundry are clean
    (258 source files, no issues). pytest deferred to CI, which ran the full matrix green on this
    exact commit — see Reviewer notes.
  • Uses Connection / Router / Handler vocabulary; no new declarative "channel" element; no
    GUI/web-framework imports in the engine packages; no Black.
  • Docs updated if behavior or configuration changed — this PR is that update.
  • Completes a docs/BACKLOG.md item — N/A, no backlog item is claimed.

🤖 Generated with Claude Code

An HA re-check of the leadership-lease construct confirmed the lease algebra,
the staged-queue handoffs and the store finalizer are correct as designed, but
found prose across code and docs asserting guarantees the fence does not
provide. This corrects the statements only -- the four .py files are
AST-identical (docstrings and comments), ruff format/check clean.

- The H1 epoch fence guards the CLAIM and nothing else. Post-claim disposition
  writes (mark_done/mark_failed/dead_letter_now/complete_with_response) resolve
  by id with no epoch, owner or status precondition, and the in-claim
  stranded-lease reclaim is a separate UNGUARDED statement that commits even
  when the guarded claim matches zero rows. Scoped the token accordingly.

- "Fenced" does not imply "stopped processing". _check_fence flips a boolean
  and cancels nothing; graph teardown is not budgeted against the
  fence-to-expiry margin, and that margin is itself short by the renew round
  trip plus up to one fence tick. The validator checks ordering, never margin.

- recover_inflight_on_promotion and the SQL Server reset_stale_inflight were
  justified by "the prior leader has stopped processing" -- a compensating
  control resting on a false premise (CLAUDE.md 11). Replaced with the sound
  directional argument: re-pending risks a DUPLICATE, which at-least-once
  permits and idempotent outbounds absorb; not re-pending risks a STRAND,
  which the count-and-log invariant forbids.

- The row lease is stamped once at claim and is NEVER renewed. settings.py,
  postgres.py (twice) and the sweep's own no-theft argument all cited a renew
  timer with no implementation anywhere in the store, and the sizing rule
  derived from it was wrong: size lease_ttl_seconds against the longest
  legitimate claim-to-terminal-write hold, not a renew interval.

- The derived leader_node_id is a one-sided cross-node wall-clock comparison
  with no lower bound, so a fast-clocked node wins the freshness pick even
  after it dies -- including masking the console's "cluster has no leader"
  health check, which keys off that field being absent.

ADR 0008 and ADR 0056 carry inline CORRECTION errata rather than rewritten
reasoning; ADR 0056's VIP release budget was derived from the uncorrected
margin. Also replaces the exhaustive "leadership is DB-clock, row leases are
wall-clock" pairing with an "at least" formulation (CLAUDE.md 11) -- the
nodes.last_seen heartbeat was the omitted third.
bc9ccd7 corrected two of the three places justifying an on-promotion
recovery write with "the prior leader has stopped processing" --
postgres.py's recover_inflight_on_promotion and engine.py's SQL Server
reset_stale_inflight call. It missed the third, ~100 lines above the second:
the comment on the leader-maintenance branch still read "the old leader
self-fenced before its lease expired, so re-pending its in-flight rows can't
steal from a live processor."

That enumeration being incomplete is itself the defect class the original
commit was fixing (CLAUDE.md 11 -- a completeness claim is a liability), so
the replacement states the reason positively rather than listing sites: the
fence stops the node REPORTING leader, not working, so the reset is correct
because it errs toward a duplicate (permitted) rather than a strand
(forbidden).

Also records what the re-check turned up while verifying this site: SQL
Server has NO periodic in-flight recovery at all. reclaim_expired_leases is
defined on the Postgres store alone (0 definitions in sqlserver.py) and the
sweep is gated on hasattr() at engine.py:1062, so a row left INFLIGHT
outside a promotion -- including by the deliberate teardown path in
stage_dispatcher, which leaves a cancelled prefix INFLIGHT for exactly this
recovery -- has nothing periodic to reclaim it and is stranded until the
next promotion or restart. Postgres bounds the same case at roughly
reclaim_interval + lease_ttl. Tracked in ADR 0157.

Comment-only: AST-identical with docstrings stripped; ruff clean.
@wshallwshall
wshallwshall enabled auto-merge (squash) August 2, 2026 02:05
@wshallwshall
wshallwshall merged commit 884036f into main Aug 2, 2026
33 checks passed
@wshallwshall
wshallwshall deleted the claude/ha-construct-recheck-6070ba branch August 2, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant