feat(water-leak): add durable journal-backed recovery service

[hyperpolymath]

Summary

This adds Firmboot.WaterLeak.Durable, a finite, disk-backed, single-writer version of the water-warning experiment.

• Journaling. Every new valid command is journaled before it is answered, in a 512-byte slot that is chained to the previous slot by SHA-256.
• What survives. Accepted readings, pending warning obligations and the first acknowledgement receipt survive the tested local process crashes.
• Gaps. Readings that arrive after a sequence gap are buffered and the gap is reported. A missing reading is never filled in.
• Retries. A retried operation ID returns its original reply.

The work existed only in an unpushed local commit, 7c29853 ("chore: land estate governance sync", 2026-09-15). That commit had swept it in alongside governance files that must not land. Its workflows do not parse, and its composite actions are not in the template. This PR extracts only the durable-recovery code, its tests and its doc onto a fresh branch from main @ 7c2b90a.

Files:

• lib/firmboot/water_leak/durable.ex and durable/{codec,journal,transition}.ex: the service, the fixed binary schema, the chained journal and deterministic replay.
• test/firmboot/durable_test.exs: 14 tests.
• docs/durable-recovery.md: the API, the commit boundary, the journal format, bounds and the stated limits.
• mix.exs: extra_applications: [:crypto], needed for :crypto.hash/2. There are no new dependencies.

No linked issue.

Type of change

• 🐛 Bug fix: not applicable.
• ✨ New feature: a new module tree. No existing module is modified.
• 💥 Breaking change: none. Firmboot.WaterLeak, RecoveryExperiment and Add checkpoint recovery, CI evidence checks, and proof artifacts #4's checkpoint replay are untouched.
• 🕳️ Soundness fix: not a checker or proof change.
• 📖 Documentation: the new docs/durable-recovery.md.
• 🧹 Refactor / tech debt: not applicable.
• ⚡ Performance: not applicable. No timing claims are made.
• 🔧 Build / CI / tooling: the only build change is the one mix.exs line above. No workflow is changed.

📌 New pins

• PR head SHA: b154e0a736768161ee350c6c660a61666cf1bdd5
• None. There is no uses: change, no actions.lock change, no new Hex dependency (mix.lock is untouched) and no container digest. :crypto is an OTP application that ships with Erlang.

How has this been verified?

Everything was run locally in the PR worktree at the head above, with system Elixir 1.18.3 on OTP 27.

• MIX_ENV=test mix test: 59 tests, 0 failures. That is the existing 45 plus the 14 new durable tests.
  • One test runs all 720 arrival permutations of a six-reading fixture.
  • Another runs 56 operation × crash-point cases: each of 14 operations is killed at before_write, partial_write, after_write and after_sync, and the final state is compared against an independent fixture.
• mix format --check-formatted: exit 0.
• reuse lint: compliant, 89/89 files with copyright and licence information.
• I checked the doc's numeric claims against the test file: there are 14 test blocks, a test named "all 720 arrival permutations…", and for index <- 0..13, point <- [4 points] (56 cases).
• git log -1 --show-signature: good ED25519 signature, as required_signatures needs.
• Not covered by CI: firmboot's CI (verify.yml) runs the Agda and Lean checks, the experiment runners, the lock check and reuse lint, but never mix test. The 14 new tests are therefore verified locally only. That gap is pre-existing and also applies to the existing 45 tests. It is recorded as a finding for the owner to schedule, and this PR deliberately does not change verify.yml (see Notes).

Checklist

• My commits are signed: SSH ED25519, verified locally.
• I ran the project's own checks locally and they pass: mix test, mix format --check-formatted and reuse lint, as above.
• New files carry the correct licence. They use REUSE.toml annotations rather than in-file headers, the same as every existing lib/ and docs/ file: lib/** and test/** are MPL-2.0, docs/** is CC-BY-SA-4.0. No file was relicensed.
• Docs are updated, and no public claim now overstates what the code does. The original doc described a 3-trial timing measurement, a JSON evidence file and an assurance register, none of which are in this PR. Those passages were removed, and the doc now says outright that it makes no latency or throughput claims.
• I have not introduced a soundness hole. No proof, checker or gate is touched. The doc states that no theorem covers this file format or replay implementation, and that the hash chain is not authentication or rollback protection. The test suite includes the complete-prefix rollback counterexample.

Notes for reviewers

• Not carried over from 7c29853. These stay in the local rescue ref refs/rescued/20261008-firmboot-unpushed/chore/bump-standards-pins, with a triage note in the owner's dev-notes inbox:
  • its rewrite of recovery_experiment.ex and water_leak_test.exs. It is a different design from Add checkpoint recovery, CI evidence checks, and proof artifacts #4's checkpoint replay, which merged later, and taking it would revert Add checkpoint recovery, CI evidence checks, and proof artifacts #4's tests;
  • its timing-measurement runner and recorded result (docs/evidence/recovery-local.json);
  • its assurance register (ASSURANCE.md, docs/assurance/*), which is read only by an extended scripts/assurance.py. Python is banned in the estate, so porting the checker is a separate decision;
  • its README.md, CONTINUITY.md, PURPOSE.md, VERIFICATION.md and REUSE.toml edits.
• Why verify.yml is untouched. Adding a mix test step is the natural follow-up. Under the estate's KYAML rule, an edited workflow has to be converted in the same PR, but only once the repo's gates are known to read KYAML. That can't be established for firmboot yet: it has no governance caller, and gh actions-lock --verify-local on flow-style YAML has not been measured. So the step is left for a separate PR rather than half-done here.
• @doc is on every public function in the four new modules. The @impl true GenServer callbacks take their docs from the behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_015bTuGfwCcvjrmNFejydTML

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 387e9f68-3b98-4dd8-8f2e-5850d088c6d1

📥 Commits

Reviewing files that changed from the base of the PR and between 7c2b90a and b154e0a.

📒 Files selected for processing (7)

• docs/durable-recovery.md
• lib/firmboot/water_leak/durable.ex
• lib/firmboot/water_leak/durable/codec.ex
• lib/firmboot/water_leak/durable/journal.ex
• lib/firmboot/water_leak/durable/transition.ex
• mix.exs
• test/firmboot/durable_test.exs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details

⏰ Context from checks skipped due to timeout. (8)

• GitHub Check: GitGuardian Security Checks
• GitHub Check: CodeFactor
• GitHub Check: Agda contract and Elixir agreement
• GitHub Check: Lean continuity and axiom audit
• GitHub Check: Code and documentation licences
• GitHub Check: CodeQL (python)
• GitHub Check: CodeQL (actions)
• GitHub Check: semgrep-cloud-platform/scan

🧰 Additional context used

🪛 LanguageTool

docs/durable-recovery.md

[typographical] ~70-~70: Two consecutive dots
Context: ...3. Current gap ranges are respectively [1..2], [2..2]`, then empty. These ranges ...

(DOUBLE_PUNCTUATION)

---

[typographical] ~70-~70: Two consecutive dots
Context: ...t gap ranges are respectively [1..2], [2..2], then empty. These ranges concern on...

(DOUBLE_PUNCTUATION)

---

[uncategorized] ~113-~113: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...to the retained workload; no worst-case time bound follows. The fingerprint binds the ava...

(EN_COMPOUND_ADJECTIVE_INTERNAL)

🔇 Additional comments (8)

lib/firmboot/water_leak/durable/codec.ex (1)

1-134: LGTM!

mix.exs (1)

13-14: LGTM!

test/firmboot/durable_test.exs (1)

1-392: LGTM!

lib/firmboot/water_leak/durable/journal.ex (1)

1-220: LGTM!

lib/firmboot/water_leak/durable.ex (1)

1-181: LGTM!

docs/durable-recovery.md (1)

1-155: LGTM!

lib/firmboot/water_leak/durable/transition.ex (2)

64-74: 🎯 Functional Correctness

The stale-reading concern is refuted. WaterLeak prepends every processed reading to observations and does not trim the list. Therefore, an accepted reading remains available to the duplicate check, so the described buffering path does not occur.

---

116-124: 🩺 Stability & Availability

A buffered reading can block the journal permanently if WaterLeak.sample/2 can fail.

drain/1 matches {:ok, water} = WaterLeak.sample(...). If a codec-valid buffered reading can return {:error, _}, the writer can terminate when the predecessor drains it. The predecessor is not journaled because Transition.commit/3 applies the transition before it appends the journal record. Retries then repeat the failure.

Validate buffered readings with the same rules before journalling them, or handle the error in drain/1 as a domain result instead of raising.

---

📝 Summary

Summary by CodeRabbit

• New Features
  • Added durable recovery for water-leak state, including journal replay, operation receipts, snapshots and status reporting.
  • Repeated operation IDs with the same command return the original result; conflicting reuse is rejected.
  • Reading gaps are reported, and incomplete journal tails can be repaired when explicitly enabled.
  • Journal integrity checks and capacity limits help prevent unsafe recovery and writes beyond available storage.
• Documentation
  • Added guidance on recovery behaviour, storage assumptions, limitations and test evidence.

Walkthrough

Adds a durable water-leak service that validates commands, records them in a bounded journal, and replays records on startup. The change also adds recovery and integrity checks, service status reporting, tests, and documentation.

Changes

Durable recovery service

Layer / File(s)	Summary
Command contracts and state transitions lib/firmboot/water_leak/durable/codec.ex, lib/firmboot/water_leak/durable/transition.ex, test/firmboot/durable_test.exs	Adds bounded configuration and command validation, tagged record encoding and decoding, replay, retry classification, and water-leak command transitions. Tests cover reading gaps, duplicates, conflicts, and arrival permutations.
Journal persistence and recovery lib/firmboot/water_leak/durable/journal.ex, mix.exs, test/firmboot/durable_test.exs	Adds fixed-slot, hash-chained journal creation and append operations, fingerprint and integrity checks, and optional incomplete-tail repair. Tests cover capacity, crash points, corruption, and truncation. The application configuration adds :crypto.
Service API and command commits lib/firmboot/water_leak/durable.ex, docs/durable-recovery.md, test/firmboot/durable_test.exs	Adds the path-based writer API, startup replay, status and snapshot responses, validation at the server boundary, and append-before-reply commits. Tests and documentation describe recovery and operation behaviour.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant Durable
  participant Transition
  participant Journal
  Caller->>Durable: submit(operation_id, command)
  Durable->>Transition: classify retry or apply command
  Transition-->>Durable: command reply and updated state
  Durable->>Journal: append and sync record
  Journal-->>Durable: updated journal
  Durable-->>Caller: return reply after successful append

Loading

Merge Risk: ⚪ Minimal · up to b154e

This change adds a self-contained durable water-leak service with journaled recovery. It does not modify existing modules, and no blocking defects were found. The tests run only locally, not in CI.

Security Architecture Review

Security architecture risk: 🔵 Low · up to b154e

The service validates bounded commands, synchronizes them before replying, and stops on uncertain write outcomes. Its guarantees depend on trusted callers and a trusted journal file. Production access and filesystem ownership are not established, so this is a limited assessment rather than assurance of deployment safety.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The evidenced state scope is a journal's configured source, readings, operations and warning receipts. A trusted in-VM caller can submit commands, read snapshots, stop the process or arm local faults. Filesystem operations execute with the hosting process's authority. Production caller authorization, OS identity, permissions and any wider service or tenant exposure remain unresolved.

Security Findings and Attack Paths

• inferred — An actor able to modify the trusted journal can replace it with an older valid prefix or recompute the unkeyed hash chain, potentially removing acknowledgement history. This is an explicit new persistence limitation, not an authenticated audit guarantee. The required filesystem access and external attacker reachability are not established, so no verified vulnerability is asserted.

Trust Boundaries and Controls

• observed — Journal opening checks file type and size before a read/write open, then validates framing, hashes, schema, engine identity and replay semantics. These controls constrain persisted input but do not establish filesystem ownership: stat/open assumes a trusted directory, and direct Journal.open callers do not participate in the service's path registration.

Resilience and Maintainability Implications

• observed — Storage is capped at 4096 records of 512 bytes plus a header. Existing-ID retries remain available at capacity, but new acknowledgements can be refused because no receipt capacity is reserved. Heap, mailbox and response-time bounds are not guaranteed; blocked sync or overload also requires an external observer rather than relying on timely status responses.

Hardening Proposals

• proposed — Before extending this experiment to untrusted callers or shared storage, establish caller authorization and journal ownership, including alias and cross-process exclusion. If receipts must resist a malicious storage writer, add an independently protected freshness anchor; hashing or a MAC alone would not prevent rollback to an older valid journal.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 65.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 43 functions across 6 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarises the main change: a durable, journal-backed recovery service for the water-leak feature.
Description check	✅ Passed	The description is directly related to the changeset and provides relevant details about the service, journal, recovery behaviour, tests, documentation, and scope.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 65.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 43 functions across 6 files. (1 skipped: 1 unsupported.)

✅ Autofix completed

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{I’m a rabbit with a journal to keep,
I count each command before I sleep.
If a tail is cut, I check the seam,
Then replay the records in sequence.
Safe in my burrow, I thump my feet.}

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Add Carrot credits or activate Agent usage billing to use Autopilot

[coderabbitai]

Fixes Applied Successfully

Fixed 6 file(s) based on 1 failed pre-merge check.

A follow-up PR containing fixes has been created.

• Follow-up PR: #10
• Files modified:

• lib/firmboot/water_leak/durable.ex
• lib/firmboot/water_leak/durable/codec.ex
• lib/firmboot/water_leak/durable/journal.ex
• lib/firmboot/water_leak/durable/transition.ex
• mix.exs
• test/firmboot/durable_test.exs

Time taken: 3m 20s