From 48cdfe9ec098cdf8f1180a0a0f7d00a060c5bfe6 Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 03:51:39 +0000 Subject: [PATCH 1/8] feat(threshold): durable share store with candidate persistence and positive-proof retirement; Details: TC-2221 --- tinycloud-threshold/README.md | 5 +- tinycloud-threshold/SPEC.md | 18 +- tinycloud-threshold/src/cert.rs | 49 ++ tinycloud-threshold/src/lib.rs | 1 + tinycloud-threshold/src/store.rs | 1397 ++++++++++++++++++++++++++++++ 5 files changed, 1466 insertions(+), 4 deletions(-) create mode 100644 tinycloud-threshold/src/store.rs diff --git a/tinycloud-threshold/README.md b/tinycloud-threshold/README.md index 9d7aa58e..5642296d 100644 --- a/tinycloud-threshold/README.md +++ b/tinycloud-threshold/README.md @@ -4,7 +4,7 @@ BLS12-381 identity-based wrapping of 32-byte data keys and context-bound encrypt The crate is a workspace member with `publish = false`. No other workspace crate depends on it; the node server remains the default member. Release-plz neither releases nor publishes it. -## Implemented surface through S0.5 +## Implemented surface through S0.6 - Bounded, canonical scalar, point, identity, ciphertext, request and share encodings. - Boneh–Franklin FullIdent wrapping and unwrapping with the TinyCloud domains in [SPEC.md](SPEC.md). @@ -18,8 +18,9 @@ The crate is a workspace member with `publish = false`. No other workspace crate - Genesis and re-key DKG (GJKR New-DKG) over owner-closed rounds, the re-key under an owner-authorized `re-key` ceremony pinned to the current certificate with the genesis fault budget: consume-self member states for dealing, complaints, answers, Feldman extraction, the common R5 round and R6 reconstruction, and `dkg_verify_transcript`, which recomputes the outcome from the raw signed round records. - Redistribution for reshare and refresh over owner-closed rounds R1–R3: Feldman dealings from the current certificate's members to an authorized cohort with a new threshold, complaints and answers, deterministic selection of the first t eligible dealers, and `reshare_verify_transcript`. The master public key is unchanged; the output is a candidate for the successor certificate. - A real-ceremony chain test: genesis 2-of-3, reshare to 3-of-4 and back to 2-of-3, refresh, re-key, and further reshares and re-keys, with a certificate, readiness receipts and activation at every generation and each retired generation's shares refused in the next context. +- A durable share store: `CandidateShare::to_bytes`/`from_bytes` (canonical, secret-guarded bytes), one file per generation under a per-network directory, install before any readiness receipt, a pin recomputed from verified activated proofs on every open, and positive-proof retirement. -`VerifiedCertificate` supplies `KeyContext` and `PublicKeySet`; `KeyShare::bind` validates a candidate against its certificate. Raw candidate-scalar construction remains crate-private, and the dealer exists only under `#[cfg(test)]`. Redistribution does not erase earlier shares; old and new shares carry different key contexts. Durable storage, retirement and node-runtime paths remain separate slices. Their contracts and the implemented canonical byte layouts are recorded in [SPEC.md](SPEC.md). +`VerifiedCertificate` supplies `KeyContext` and `PublicKeySet`; `KeyShare::bind` validates a candidate against its certificate. Raw candidate-scalar construction remains crate-private, and the dealer exists only under `#[cfg(test)]`. Redistribution does not erase earlier shares; old and new shares carry different key contexts. The `store` module implements the file-volume durable share store and retirement; node-runtime wiring remains a separate slice. Their contracts and the implemented canonical byte layouts are recorded in [SPEC.md](SPEC.md). The §3.6 request-digest amendment is explicit in SPEC.md. It makes an identity or transport-key transplant a context mismatch before share equations are evaluated. diff --git a/tinycloud-threshold/SPEC.md b/tinycloud-threshold/SPEC.md index 2c8f189f..510f6ab1 100644 --- a/tinycloud-threshold/SPEC.md +++ b/tinycloud-threshold/SPEC.md @@ -231,7 +231,7 @@ All integer fields below are big-endian. `lp` is u32 length followed by bytes. - No S0.1 public decoder consumes a member list; lists introduced in later slices use the §3.2 sorted/count rules. - An availability budget counts all unavailable ready signers, including dishonest claims: `q - b >= t′`. All four ready holders in 3-of-4 support one unavailable holder, not two. `WithOutageBudget(0)` is not accepted. -## Implemented surface through S0.5 +## Implemented surface through S0.6 S0.1 implements §§3.1–3.6 and the typed parameters in §3.0. S0.2 adds the authenticated-context boundaries in §3.0, the certificate and fault-proof APIs in §3.9, and the receipt/activation verification in §3.12: @@ -248,7 +248,9 @@ S0.4 adds the §3.8 redistribution for reshare and refresh: dealer and recipient S0.5 runs the §3.7 DKG under a `re-key` ceremony (`dkg_start` and `dkg_verify_transcript` accept kinds `genesis` and `re-key` and refuse reshare and refresh with `InvalidCeremony`), executes the full genesis → reshare → reshare → refresh → re-key chain with real ceremonies, certificates, readiness receipts and activation, and freezes the chain and re-key vectors described in the S0.5 appendix. -The dealer remains test-only. Durable storage/retirement and node-runtime integration are separate slices. The crate remains an unwired workspace member. +S0.6 adds the `store` module: the durable file-volume share store of §3.11–3.12 — `CandidateShare` canonical byte persistence, one proof and one share file per certificate, receipt-only-after-install, the activated-chain pin recomputed from verified proofs at open, and positive-proof retirement (superseded generations, losing candidates, unnamed files and replayed ceremony ids). Its layouts are in the S0.6 appendix below. + +The dealer remains test-only. Node-runtime integration (volume mount, peer/DB pin sources, boot freshness, audit rows) is a separate slice. The crate remains an unwired workspace member. ## Verifier conformance @@ -545,3 +547,15 @@ A chain of generations is: genesis; then any sequence of reshare, refresh and re `vectors/v1/dkg-rekey.json` has the shape of `dkg-genesis.json` with `"spec_sections"` naming `S0.5 re-key appendix`, and each case adds `inputs.current_certificate`, the signed pinned certificate (`body` and `owner_signature`, as in `reshare.json`). The cases mirror the genesis cases under a re-key ceremony; names begin `rekey_` (`insufficient_*` keep their names). `vectors/v1/chain.json` has one case per generation, in chain order, with operation `chain_step`. A case's inputs are the signed ceremony statement and its digest, the initial transcript link, every owner closure and member message, `previous_certificate` (absent for genesis, otherwise the previous case's `certificate`), the candidate `certificate` (`body` and `owner_signature`), one readiness receipt (`member`, `signature`) per holder, `quorum` `all_holders` and the owner's `activation` statement (`body` is the exact 32-byte activation preimage). The outputs are the recomputed kind, epoch, re-key counter, genesis t, threshold, members, mpk, commitments, public shares, transcript hash, certificate digest, activation preimage and the ceremony outcome (`qual` and `reconstructed` for genesis and re-key, `dealers` for reshare and refresh). A verifier recomputes the ceremony outcome from the transcript (S0.3 or S0.4 rules), rebuilds the certificate body from that outcome and the pinned predecessor, requires it to equal the supplied body byte for byte, verifies the owner signature and lineage (S0.2 appendix), verifies every receipt and the activation, and compares its own results with the outputs; it never takes an output field as input to its decision. + +## S0.6 durable share-store appendix + +The `store` module keeps one member's share material for one network under `//` (directory mode 0700, files 0600). Every write is write-temp, fsync, rename, fsync-directory; erasure is zero-overwrite, fsync, unlink, fsync-directory. Shares exist only as files named `share-`; `proofs/` holds certificate and activation evidence; `dealt/` is an empty marker; `state` is the single mutable record. + +**Candidate share file.** `version:u8=1 || transcript_hash:32 || index:u16 || scalar:32` (67 bytes). Decode is canonical only: exact length, version 1, index 1..=16, scalar < r and non-zero. Secret bytes travel inside `Zeroizing` and `bind` re-verifies them against the certificate before use. + +**Proof file.** `version:u8=1 || lp(signed_certificate) || activation_flag:u8 || [lp(activation_bytes) || lp(owner_signature) || receipt_count:u16 || (index:u16 || lp(signature))…]`. The signed certificate is `version:u8=1 || lp(certificate_body) || lp(owner_signature)`. + +**State file.** `version:u8=1 || option(pin:32) || option(candidate: digest:32 || ceremony_id:32 || option(predecessor:32) || receipt_sent:u8) || seen_count:u32 || seen_ceremony_id:32… || retired_count:u32 || (digest:32 || erased_at:u64)… || seq:u64`. + +**Rules.** At most one candidate record exists at a time and only for the pinned predecessor (or for genesis before any pin); a `receipt_sent` flag may be set only after proof and share files are durable. Open recomputes the activated chain from `proofs/` (genesis through verified `AllHolders` activations), refuses a recorded pin the chain cannot prove, and freezes at any activated equivocation. A share file is erased when the pin's chain passes its certificate (R1), when a candidate's predecessor is covered or superseded (C2), when a fresh ceremony id arrives for its predecessor (C1, applied in `accept_ceremony` before any dealing), or when no record names it (S1). A `dealt/` marker consumes its ceremony id even when the state write never happened; replayed ids change nothing. `record_activation` is idempotent per certificate digest. diff --git a/tinycloud-threshold/src/cert.rs b/tinycloud-threshold/src/cert.rs index 9ee678c3..f6cc6986 100644 --- a/tinycloud-threshold/src/cert.rs +++ b/tinycloud-threshold/src/cert.rs @@ -104,7 +104,44 @@ impl CandidateShare { pub fn index(&self) -> MemberIndex { self.index } + /// Secret bytes: version(1) ‖ transcript hash(32) ‖ index u16be(2) ‖ scalar(32). Details: TC-2221 + /// + /// The caller owns the returned buffer's erasure; it is wrapped in + /// [`zeroize::Zeroizing`] so a dropped copy clears itself. [`CandidateShare::from_bytes`] + /// restores the candidate for [`KeyShare::bind`] against the pinned certificate. + pub fn to_bytes(&self) -> zeroize::Zeroizing<[u8; 67]> { + let mut e = crate::encoding::Encoder::new(); + e.u8(1); + e.fixed(self.transcript.as_bytes()); + e.u16(self.index.get()); + e.fixed(&self.scalar.to_bytes()); + let bytes = e.finish(); + let mut out = zeroize::Zeroizing::new([0u8; 67]); + out.copy_from_slice(&bytes); + out + } + + /// Canonical decode only: version 1, index 1..=16, scalar < r and != 0, exact length. Details: TC-2221 + pub fn from_bytes(bytes: &[u8]) -> Result { + let mut d = crate::encoding::Decoder::new(bytes)?; + if d.u8()? != 1 { + return Err(Error::InvalidEncoding); + } + let transcript = TranscriptHash::new(d.fixed()?); + let index = MemberIndex::new(d.u16()?)?; + let scalar = Scalar::from_bytes(&d.fixed::<32>()?)?; + if scalar.is_zero() { + return Err(Error::InvalidScalar); + } + d.finish()?; + Ok(Self { + transcript, + index, + scalar, + }) + } } + impl fmt::Debug for CandidateShare { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.write_str("CandidateShare([REDACTED])") @@ -417,6 +454,18 @@ impl CertificateBody { pub fn transcript_hash(&self) -> TranscriptHash { self.transcript } + /// Pinned predecessor digest, absent only at genesis. Details: TC-2221 + pub fn previous(&self) -> Option<&CertDigest> { + self.previous.as_ref() + } + /// Certificate epoch. Details: TC-2221 + pub fn epoch(&self) -> Epoch { + self.epoch + } + /// Re-key counter. Details: TC-2221 + pub fn rekey(&self) -> RekeyCounter { + self.rekey + } } /// Raw owner-signed certificate. Details: TC-2222 diff --git a/tinycloud-threshold/src/lib.rs b/tinycloud-threshold/src/lib.rs index 01b3b368..5d61f8d9 100644 --- a/tinycloud-threshold/src/lib.rs +++ b/tinycloud-threshold/src/lib.rs @@ -32,6 +32,7 @@ pub mod ids; mod poly; pub mod reshare; pub mod seal; +pub mod store; pub use activation::{ activation_preimage, readiness_preimage, verify_activation, ReadinessReceipt, ReadyQuorum, diff --git a/tinycloud-threshold/src/store.rs b/tinycloud-threshold/src/store.rs new file mode 100644 index 00000000..d9295b5f --- /dev/null +++ b/tinycloud-threshold/src/store.rs @@ -0,0 +1,1397 @@ +//! Durable share store: candidate persistence, activation pin and retirement. +//! Details: TC-2221 +//! +//! A member keeps at most one generation current, at most one candidate +//! awaiting activation, and positive-proof records for everything it erased. +//! Nothing in the workspace calls this module yet; the node runtime wires it +//! in a later slice. +//! +//! Layout under `//`: +//! +//! * `state`: versioned record of the pinned certificate, the pending +//! candidate, accepted ceremony ids and retired generations. +//! * `proofs/`: the signed certificate bytes plus, once +//! activated, the owner activation statement and readiness receipts. +//! * `share-`: `CandidateShare::to_bytes()` for this member's +//! share under that certificate. +//! * `dealt/`: marker written before the first secret-bearing +//! message of that ceremony leaves; the id is consumed forever. +//! +//! Every write is write-temp, fsync, rename, fsync-directory. The pin is the +//! highest-epoch certificate whose activation chain verifies from genesis; a +//! restart replays `proofs/` to recompute it and refuses a recorded `state` +//! pin the proofs cannot support. Shares are erased only on positive proof: +//! a verified activation that moves past them, a fresh ceremony for their +//! predecessor, or no record at all. + +use std::{ + collections::BTreeMap, + fs, + path::{Path, PathBuf}, + sync::Mutex, +}; + +use sha2::{Digest, Sha256}; +use zeroize::Zeroizing; + +use crate::{ + activation::{verify_activation, ReadinessReceipt, ReadyQuorum}, + ceremony::{AuthorizedCeremony, DidSignatureVerifier, SignedStatement, TrustAnchor}, + cert::{ + verify_certificate, verify_successor, CandidateShare, CertificateBody, SignedCertificate, + VerifiedCertificate, + }, + encoding::{Decoder, Encoder}, + error::{CertError, Error}, + ids::{CeremonyId, CertDigest, MemberIndex, NetworkUrn}, + KeyShare, +}; + +/// One retired generation. Details: TC-2221 +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct RetiredGeneration { + /// Retired certificate digest. + pub digest: CertDigest, + /// Retirement sequence counter for this member. + pub erased_at: u64, +} + +/// The pending candidate record. +#[derive(Clone, Debug)] +struct CandidateRecord { + digest: CertDigest, + ceremony: CeremonyId, + predecessor: Option, + receipt_sent: bool, +} + +#[derive(Clone, Debug, Default)] +struct State { + pin: Option, + candidate: Option, + seen_ceremonies: Vec, + retired: Vec, + seq: u64, +} + +fn encode_state(state: &State) -> Vec { + let mut e = Encoder::new(); + e.u8(1); + e.u8(u8::from(state.pin.is_some())); + if let Some(pin) = state.pin { + e.fixed(pin.as_bytes()); + } + e.u8(u8::from(state.candidate.is_some())); + if let Some(candidate) = &state.candidate { + e.fixed(candidate.digest.as_bytes()); + e.fixed(candidate.ceremony.as_bytes()); + e.u8(u8::from(candidate.predecessor.is_some())); + if let Some(predecessor) = candidate.predecessor { + e.fixed(predecessor.as_bytes()); + } + e.u8(u8::from(candidate.receipt_sent)); + } + // Ceremony ids and retired entries are bounded by the on-disk format cap + // enforced in decode_state; u32 cannot overflow below memory limits. + e.u32(state.seen_ceremonies.len() as u32); + for id in &state.seen_ceremonies { + e.fixed(id.as_bytes()); + } + e.u32(state.retired.len() as u32); + for retired in &state.retired { + e.fixed(retired.digest.as_bytes()); + e.u64(retired.erased_at); + } + e.u64(state.seq); + e.finish() +} + +fn decode_state(bytes: &[u8]) -> Result { + let mut d = Decoder::new(bytes)?; + if d.u8()? != 1 { + return Err(Error::InvalidEncoding.into()); + } + let pin = if d.u8()? == 1 { + Some(CertDigest::new(d.fixed()?)) + } else { + None + }; + let candidate = if d.u8()? == 1 { + Some(CandidateRecord { + digest: CertDigest::new(d.fixed()?), + ceremony: CeremonyId::new(d.fixed()?), + predecessor: if d.u8()? == 1 { + Some(CertDigest::new(d.fixed()?)) + } else { + None + }, + receipt_sent: d.u8()? == 1, + }) + } else { + None + }; + let seen_len = usize::try_from(d.u32()?).map_err(|_| Error::InvalidEncoding)?; + if seen_len > 1_000_000 { + return Err(Error::InvalidEncoding.into()); + } + let mut seen_ceremonies = Vec::with_capacity(seen_len.min(4096)); + for _ in 0..seen_len { + seen_ceremonies.push(CeremonyId::new(d.fixed()?)); + } + let retired_len = usize::try_from(d.u32()?).map_err(|_| Error::InvalidEncoding)?; + if retired_len > 1_000_000 { + return Err(Error::InvalidEncoding.into()); + } + let mut retired = Vec::with_capacity(retired_len.min(4096)); + for _ in 0..retired_len { + retired.push(RetiredGeneration { + digest: CertDigest::new(d.fixed()?), + erased_at: d.u64()?, + }); + } + let seq = d.u64()?; + d.finish()?; + Ok(State { + pin, + candidate, + seen_ceremonies, + retired, + seq, + }) +} + +fn encode_signed_certificate(signed: &SignedCertificate) -> Vec { + let mut e = Encoder::new(); + e.u8(1); + e.lp(&signed.body.encode()); + e.lp(&signed.owner_signature); + e.finish() +} + +fn decode_signed_certificate(bytes: &[u8]) -> Result { + let mut d = Decoder::new(bytes)?; + if d.u8()? != 1 { + return Err(Error::InvalidEncoding.into()); + } + let body = CertificateBody::decode(d.lp(64 * 1024)?)?; + let owner_signature = d.lp(512)?.to_vec(); + d.finish()?; + Ok(SignedCertificate { + body, + owner_signature, + }) +} + +fn encode_proof( + signed: &SignedCertificate, + activation: Option<&SignedStatement>, + receipts: &[ReadinessReceipt], +) -> Vec { + let mut e = Encoder::new(); + e.u8(1); + e.lp(&encode_signed_certificate(signed)); + e.u8(u8::from(activation.is_some())); + if let Some(activation) = activation { + e.lp(&activation.bytes); + e.lp(&activation.signature); + // Receipts are bounded by n <= 16 members. + e.u16(receipts.len() as u16); + for receipt in receipts { + e.u16(receipt.index.get()); + e.lp(&receipt.signature); + } + } + e.finish() +} + +/// An activated proof's owner statement and member receipts. Details: TC-2221 +type ProofActivation = (SignedStatement, Vec); + +fn decode_proof(bytes: &[u8]) -> Result<(SignedCertificate, Option), StoreError> { + let mut d = Decoder::new(bytes)?; + if d.u8()? != 1 { + return Err(Error::InvalidEncoding.into()); + } + let signed = decode_signed_certificate(d.lp(128 * 1024)?)?; + let activation = if d.u8()? == 1 { + let statement = SignedStatement { + bytes: d.lp(64 * 1024)?.to_vec(), + signature: d.lp(512)?.to_vec(), + }; + let count = usize::from(d.u16()?); + if count > 16 { + return Err(Error::InvalidEncoding.into()); + } + let mut receipts = Vec::with_capacity(count); + for _ in 0..count { + receipts.push(ReadinessReceipt { + index: MemberIndex::new(d.u16()?)?, + signature: d.lp(512)?.to_vec(), + }); + } + Some((statement, receipts)) + } else { + None + }; + d.finish()?; + Ok((signed, activation)) +} + +/// Store errors. Details: TC-2221 +#[derive(Debug, thiserror::Error)] +pub enum StoreError { + /// Filesystem failure. + #[error("share store I/O: {0}")] + Io(#[from] std::io::Error), + /// Threshold encoding/verification failure. + #[error("{0}")] + Crypto(#[from] Error), + /// Certificate verification failure. + #[error("{0}")] + Cert(#[from] CertError), + /// On-disk content failed to decode or a recorded promise lacks proof. + #[error("share store state is inconsistent")] + Corrupt, + /// A second candidate exists before the first was erased. + #[error("candidate already installed")] + CandidateExists, + /// The certificate does not extend the pin, or a genesis arrives after one. + #[error("activation does not extend the pinned certificate")] + InvalidActivation, + /// Activated equivocation froze the pin; further activations are refused. + #[error("activated equivocation froze the pin")] + Equivocation, +} + +/// Result of presenting an authorized ceremony to the store. Details: TC-2221 +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum AcceptOutcome { + /// First sight of this ceremony id: dealt marker written, id consumed, + /// and any unactivated candidate for its predecessor erased. + Fresh, + /// Ceremony id already consumed; nothing changed. + Replay, +} + +/// One member's durable share store for one network. Details: TC-2221 +pub struct ShareStore { + network_dir: PathBuf, + anchor: TrustAnchor, + inner: Mutex, +} + +struct Inner { + state: State, + pin: Option, + equivocation: bool, +} + +fn create_file(path: &Path) -> Result { + let mut options = fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + Ok(options.open(path)?) +} + +fn write_atomic(path: &Path, bytes: &[u8]) -> Result<(), StoreError> { + let tmp = path.with_extension("tmp"); + { + use std::io::Write; + let mut file = create_file(&tmp)?; + file.write_all(bytes)?; + file.sync_all()?; + } + fs::rename(&tmp, path)?; + if let Some(parent) = path.parent() { + fs::File::open(parent)?.sync_all()?; + } + Ok(()) +} + +fn erase_share_file(path: &Path) -> Result<(), StoreError> { + if !path.exists() { + return Ok(()); + } + let length = usize::try_from(fs::metadata(path)?.len()).unwrap_or(usize::MAX); + { + use std::io::Write; + let mut file = fs::OpenOptions::new().write(true).open(path)?; + file.write_all(&Zeroizing::new(vec![0u8; length]))?; + file.sync_all()?; + } + fs::remove_file(path)?; + if let Some(parent) = path.parent() { + fs::File::open(parent)?.sync_all()?; + } + Ok(()) +} + +fn share_path(dir: &Path, digest: &CertDigest) -> PathBuf { + dir.join(format!("share-{}", hex(digest.as_bytes()))) +} + +fn proof_path(dir: &Path, digest: &CertDigest) -> PathBuf { + dir.join("proofs").join(hex(digest.as_bytes())) +} + +fn record_retired(state: &mut State, digest: CertDigest) { + state.seq = state.seq.saturating_add(1); + state.retired.push(RetiredGeneration { + digest, + erased_at: state.seq, + }); +} + +fn hex(bytes: &[u8]) -> String { + const HEX: &[u8; 16] = b"0123456789abcdef"; + let mut out = String::with_capacity(bytes.len().saturating_mul(2)); + for byte in bytes { + if let Some(hi) = HEX.get(usize::from(byte >> 4)) { + out.push(*hi as char); + } + if let Some(lo) = HEX.get(usize::from(byte & 0x0f)) { + out.push(*lo as char); + } + } + out +} + +fn hex_val(byte: u8) -> Option { + match byte { + b'0'..=b'9' => byte.checked_sub(b'0'), + b'a'..=b'f' => byte.checked_sub(b'a').and_then(|v| v.checked_add(10)), + _ => None, + } +} + +fn decode_hex(text: &str) -> Option<[u8; 32]> { + if text.len() != 64 { + return None; + } + let mut out = [0u8; 32]; + let bytes = text.as_bytes(); + for (i, slot) in out.iter_mut().enumerate() { + let hi = hex_val(*bytes.get(i.saturating_mul(2))?)?; + let lo = hex_val(*bytes.get(i.saturating_mul(2).saturating_add(1))?)?; + *slot = (hi << 4) | lo; + } + Some(out) +} + +fn store_dir(dir: &Path, network: &NetworkUrn) -> PathBuf { + let digest = Sha256::digest(network.as_bytes()); + let name = hex(digest.get(..16).unwrap_or_default()); + dir.join(name) +} + +impl ShareStore { + /// Open or create the member's share directory and converge it. Details: TC-2221 + /// + /// Convergence order: rebuild the verified activated chain from `proofs/`, + /// refuse a recorded pin the chain cannot prove, then apply the erase + /// rules — R1 (superseded generations), C2 (losing candidates), S1 + /// (files named in no record) and orphan dealt markers. + pub fn open( + dir: &Path, + network: &NetworkUrn, + anchor: TrustAnchor, + v: &impl DidSignatureVerifier, + ) -> Result { + let network_dir = store_dir(dir, network); + fs::create_dir_all(network_dir.join("proofs"))?; + fs::create_dir_all(network_dir.join("dealt"))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + fs::set_permissions(&network_dir, fs::Permissions::from_mode(0o700))?; + } + + let mut state = match fs::read(network_dir.join("state")) { + Ok(bytes) => decode_state(&bytes)?, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => State::default(), + Err(e) => return Err(e.into()), + }; + let mut state_dirty = false; + + // Orphan dealt markers: a marker whose id was never recorded consumed + // the id anyway, so a retried authorization is never "new". + for entry in fs::read_dir(network_dir.join("dealt"))? { + let entry = entry?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + if name == "state" || name.len() != 64 { + continue; + } + let Some(id_bytes) = decode_hex(name) else { + continue; + }; + let id = CeremonyId::new(id_bytes); + if !state.seen_ceremonies.contains(&id) { + state.seen_ceremonies.push(id); + state_dirty = true; + } + } + + // Load every proof; a malformed file is corruption, never skipped. + let mut proofs: BTreeMap = BTreeMap::new(); + let mut activations: BTreeMap)> = + BTreeMap::new(); + for entry in fs::read_dir(network_dir.join("proofs"))? { + let entry = entry?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + if name.ends_with(".tmp") { + let _ = fs::remove_file(entry.path()); + continue; + } + let (signed, activation) = decode_proof(&fs::read(entry.path())?)?; + let digest = signed.body.digest(); + if hex(digest.as_bytes()) != name { + return Err(StoreError::Corrupt); + } + proofs.insert(digest, signed); + if let Some((statement, receipts)) = activation { + activations.insert(digest, (statement, receipts)); + } + } + + // Build the activated chain from genesis forward. Two activated + // successors of one certificate (or two activated geneses) freeze the + // pin at the fork. + let mut verified: BTreeMap = BTreeMap::new(); + let mut chain: Vec = Vec::new(); + let mut equivocation = false; + let mut geneses: Vec = Vec::new(); + for signed in proofs.values() { + if signed.body.previous().is_some() || !activations.contains_key(&signed.body.digest()) + { + continue; + } + geneses.push(verify_certificate(&anchor, signed, v)?); + } + geneses.sort_by(|a, b| a.digest().as_bytes().cmp(b.digest().as_bytes())); + if geneses.len() > 1 { + equivocation = true; + } + if let Some(first) = geneses.into_iter().next() { + let mut current = first; + chain.push(current.digest()); + verified.insert(current.digest(), current.clone()); + loop { + let mut successors: Vec = Vec::new(); + for signed in proofs.values() { + if signed.body.previous() != Some(¤t.digest()) + || !activations.contains_key(&signed.body.digest()) + { + continue; + } + successors.push(verify_successor(¤t, signed, v)?); + } + successors.sort_by(|a, b| a.digest().as_bytes().cmp(b.digest().as_bytes())); + if successors.len() > 1 { + // The pin stops advancing at the fork. + equivocation = true; + break; + } + let Some(next) = successors.into_iter().next() else { + break; + }; + let (statement, receipts) = + activations.get(&next.digest()).ok_or(StoreError::Corrupt)?; + verify_activation(&next, receipts, ReadyQuorum::AllHolders, statement, v)?; + verified.insert(next.digest(), next.clone()); + chain.push(next.digest()); + current = next; + } + } + let computed_pin = chain.last().copied(); + let pin = computed_pin.and_then(|d| verified.get(&d).cloned()); + + // A recorded pin the activated chain cannot prove is corruption. + if let Some(recorded) = state.pin { + if !chain.contains(&recorded) { + return Err(StoreError::Corrupt); + } + } + + // Candidate reconciliation. + if let Some(candidate) = state.candidate.clone() { + let share = share_path(&network_dir, &candidate.digest); + let proof = proof_path(&network_dir, &candidate.digest); + if !share.exists() || !proof.exists() { + if candidate.receipt_sent { + // A receipt was signed over a durable install; a missing + // file means the member promised what it cannot serve. + return Err(StoreError::Corrupt); + } + let _ = erase_share_file(&share); + state.candidate = None; + state_dirty = true; + } else if chain.contains(&candidate.digest) { + // Already activated before the crash; the chain is the record. + state.candidate = None; + state_dirty = true; + } else { + // C2: the candidate lost if its predecessor is covered by the + // pin (any pin for a genesis predecessor) or the chain moved + // past it through another certificate. + let lost = match candidate.predecessor { + None => computed_pin.is_some(), + Some(p) => chain + .iter() + .position(|d| *d == p) + .map_or(computed_pin.is_some(), |i| { + computed_pin != Some(p) || i.saturating_add(1) < chain.len() + }), + }; + if lost { + erase_share_file(&share)?; + record_retired(&mut state, candidate.digest); + state.candidate = None; + state_dirty = true; + } + } + } + + // R1 and S1 over share files. + for entry in fs::read_dir(&network_dir)? { + let entry = entry?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + let Some(hex_digest) = name.strip_prefix("share-") else { + continue; + }; + let Some(digest_bytes) = decode_hex(hex_digest) else { + // An unnamed partial write (including .tmp leftovers) can + // never have been recorded; S1 erases it. + erase_share_file(&entry.path())?; + continue; + }; + let digest = CertDigest::new(digest_bytes); + let is_pin = computed_pin == Some(digest); + let is_candidate = state.candidate.as_ref().is_some_and(|c| c.digest == digest); + if is_pin || is_candidate { + continue; + } + if chain.contains(&digest) { + // Superseded activated generation (R1). + erase_share_file(&entry.path())?; + if !state.retired.iter().any(|r| r.digest == digest) { + record_retired(&mut state, digest); + state_dirty = true; + } + } else { + // Outside the activated chain and named in no record: S1 + // erases unconditionally (the pin's existence changes + // nothing — such a file never had a durable install record). + erase_share_file(&entry.path())?; + } + } + + if state.pin != computed_pin || state_dirty { + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + } + + Ok(Self { + network_dir, + anchor, + inner: Mutex::new(Inner { + state, + pin, + equivocation, + }), + }) + } + + /// The pinned certificate, if any activation chain exists. Details: TC-2221 + pub fn pinned(&self) -> Option { + self.inner.lock().ok()?.pin.clone() + } + + /// Whether activated equivocation froze the pin. Details: TC-2221 + pub fn equivocation(&self) -> bool { + self.inner.lock().map(|i| i.equivocation).unwrap_or(true) + } + + /// Retired generations recorded so far. Details: TC-2221 + pub fn retired(&self) -> Vec { + self.inner + .lock() + .map(|i| i.state.retired.clone()) + .unwrap_or_default() + } + + /// The bound share for the pinned certificate, or none when this member + /// holds no share for it. Details: TC-2221 + pub fn current_share(&self) -> Result, StoreError> { + let pin = self + .inner + .lock() + .map_err(|_| StoreError::Corrupt)? + .pin + .clone(); + let Some(pin) = pin else { return Ok(None) }; + let path = share_path(&self.network_dir, &pin.digest()); + if !path.exists() { + return Ok(None); + } + let bytes = Zeroizing::new(fs::read(path)?); + let candidate = CandidateShare::from_bytes(&bytes)?; + Ok(Some(KeyShare::bind(candidate, &pin)?)) + } + + /// The pending candidate's digest and member index, awaiting its + /// readiness receipt. Details: TC-2221 + pub fn pending_candidate(&self) -> Option<(CertDigest, MemberIndex)> { + let inner = self.inner.lock().ok()?; + let candidate = inner.state.candidate.as_ref()?; + let path = share_path(&self.network_dir, &candidate.digest); + let bytes = fs::read(path).ok()?; + let share = CandidateShare::from_bytes(&bytes).ok()?; + Some((candidate.digest, share.index())) + } + + /// Durably install a candidate share for `signed`. Details: TC-2221 + /// + /// `signed` must verify as the successor of the current pin (or as a + /// genesis before any pin). The candidate's bytes bind against the + /// certificate, then proof and share are written before the state record: + /// a crash before the record leaves orphan files the next open sweeps. + /// Returns the candidate certificate digest. + pub fn install_candidate( + &self, + signed: &SignedCertificate, + candidate: CandidateShare, + ceremony: &AuthorizedCeremony, + v: &impl DidSignatureVerifier, + ) -> Result { + let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + if inner.state.candidate.is_some() { + return Err(StoreError::CandidateExists); + } + let (verified, predecessor) = match &inner.pin { + Some(pin) => (verify_successor(pin, signed, v)?, Some(pin.digest())), + None => { + if signed.body.previous().is_some() { + return Err(StoreError::InvalidActivation); + } + (verify_certificate(&self.anchor, signed, v)?, None) + } + }; + if ceremony.statement().current != predecessor { + return Err(StoreError::InvalidActivation); + } + let bytes = candidate.to_bytes(); + let reloaded = CandidateShare::from_bytes(bytes.as_slice())?; + // The written form must bind before it is persisted. + let _bound = KeyShare::bind(reloaded, &verified)?; + let digest = verified.digest(); + write_atomic( + &proof_path(&self.network_dir, &digest), + &encode_proof(signed, None, &[]), + )?; + write_atomic(&share_path(&self.network_dir, &digest), bytes.as_slice())?; + inner.state.candidate = Some(CandidateRecord { + digest, + ceremony: ceremony.statement().id, + predecessor, + receipt_sent: false, + }); + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + Ok(digest) + } + + /// Record that this member's readiness receipt was sent. Details: TC-2221 + /// + /// Called only after `install_candidate` committed; a restart that finds + /// `receipt_sent` with a missing share file refuses to open. + pub fn mark_receipt_sent(&self) -> Result<(), StoreError> { + let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + let Some(candidate) = &mut inner.state.candidate else { + return Err(StoreError::Corrupt); + }; + candidate.receipt_sent = true; + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state)) + } + + /// Record a verified activation; promotes the candidate when it matches + /// and retires every superseded share. Details: TC-2221 + /// + /// Proof and share order keeps the crash window safe: the activated proof + /// lands first, so a restart recomputes the higher pin from `proofs/` and + /// finishes retirement even if `state` was never rewritten. + pub fn record_activation( + &self, + signed: &SignedCertificate, + activation: &SignedStatement, + receipts: &[ReadinessReceipt], + v: &impl DidSignatureVerifier, + ) -> Result<(), StoreError> { + let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + if inner.equivocation { + return Err(StoreError::Equivocation); + } + // Idempotent: a certificate already activated on the pinned chain is + // a no-op, so concurrent or replayed recordings converge. + if inner + .pin + .as_ref() + .is_some_and(|pin| pin.digest() == signed.body.digest()) + { + return Ok(()); + } + let verified = match &inner.pin { + Some(pin) => verify_successor(pin, signed, v)?, + None => verify_certificate(&self.anchor, signed, v)?, + }; + verify_activation(&verified, receipts, ReadyQuorum::AllHolders, activation, v)?; + let digest = verified.digest(); + write_atomic( + &proof_path(&self.network_dir, &digest), + &encode_proof(signed, Some(activation), receipts), + )?; + // C2: a pending candidate that lost to this activation is erased; the + // winning candidate's record is cleared (the chain is its record). + if let Some(candidate) = inner.state.candidate.take() { + if candidate.digest != digest { + erase_share_file(&share_path(&self.network_dir, &candidate.digest))?; + if !inner + .state + .retired + .iter() + .any(|r| r.digest == candidate.digest) + { + record_retired(&mut inner.state, candidate.digest); + } + } + } + // R1: the outgoing pin's share is retired. + if let Some(old_pin) = &inner.pin { + let old = old_pin.digest(); + erase_share_file(&share_path(&self.network_dir, &old))?; + if !inner.state.retired.iter().any(|r| r.digest == old) { + record_retired(&mut inner.state, old); + } + } + inner.pin = Some(verified); + inner.state.pin = Some(digest); + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state)) + } + + /// Present a freshly verified authorized ceremony before any dealing. + /// Details: TC-2221 + /// + /// A new id erases any unactivated candidate for the same predecessor + /// (C1), writes the dealt marker and consumes the id; a replayed id + /// changes nothing. + pub fn accept_ceremony( + &self, + ceremony: &AuthorizedCeremony, + ) -> Result { + let id = ceremony.statement().id; + let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + if inner.state.seen_ceremonies.contains(&id) { + return Ok(AcceptOutcome::Replay); + } + let predecessor = ceremony.statement().current; + if let Some(candidate) = &inner.state.candidate { + if candidate.predecessor == predecessor { + let digest = candidate.digest; + erase_share_file(&share_path(&self.network_dir, &digest))?; + record_retired(&mut inner.state, digest); + inner.state.candidate = None; + } + } + write_atomic( + &self.network_dir.join("dealt").join(hex(id.as_bytes())), + b"", + )?; + inner.state.seen_ceremonies.push(id); + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + Ok(AcceptOutcome::Fresh) + } +} + +#[cfg(test)] +mod tests { + use std::fs; + use std::path::{Path, PathBuf}; + use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::Arc; + + use crate::{ + activation::{activation_preimage, readiness_preimage}, + ceremony::{tests::Signers, CeremonyKind}, + cert::{self, tests as cert_tests}, + curve::Scalar, + ids::*, + KeyShare, + }; + + use super::*; + + static TAG: AtomicU64 = AtomicU64::new(0); + + fn tmpdir() -> PathBuf { + let tag = TAG.fetch_add(1, Ordering::Relaxed); + let path = std::env::temp_dir().join(format!("tc2221-store-{}-{tag}", std::process::id())); + let _ = fs::remove_dir_all(&path); + fs::create_dir_all(&path).unwrap(); + path + } + + fn netdir(dir: &Path, network: &NetworkUrn) -> PathBuf { + store_dir(dir, network) + } + + fn open(dir: &Path, signers: &Signers) -> ShareStore { + ShareStore::open( + dir, + &signers.anchor().network, + signers.anchor().clone(), + signers, + ) + .unwrap() + } + + fn receipts_for(cert: &VerifiedCertificate, signers: &Signers) -> Vec { + cert.body() + .members() + .iter() + .map(|m| ReadinessReceipt { + index: m.index, + signature: signers + .sign(&m.node, &readiness_preimage(cert.body(), m.index).unwrap()), + }) + .collect() + } + + fn activation_for( + cert: &VerifiedCertificate, + receipts: &[ReadinessReceipt], + signers: &Signers, + ) -> SignedStatement { + let bytes = activation_preimage(cert.body(), receipts).unwrap(); + SignedStatement { + signature: signers.sign(&cert.anchor().expected_owner, &bytes), + bytes, + } + } + + fn mk_candidate(mut raw: KeyShare, body: &CertificateBody) -> CandidateShare { + let scalar = core::mem::replace(&mut raw.scalar, Scalar::from_u64(0)); + CandidateShare::new(body.transcript_hash(), raw.index, scalar) + } + + /// Genesis: install member 1's share, send receipt, record activation. + fn activated_genesis( + store: &ShareStore, + signers: &Signers, + ) -> (SignedCertificate, VerifiedCertificate) { + let (cert, mut shares) = cert_tests::genesis(signers); + let share = shares.remove(0); + let signed = cert::tests::sign(cert.body().clone(), signers); + let candidate = mk_candidate(share, cert.body()); + store + .install_candidate(&signed, candidate, &signers.genesis(), signers) + .unwrap(); + store.mark_receipt_sent().unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + store + .record_activation(&signed, &activation, &receipts, signers) + .unwrap(); + (signed, cert) + } + + /// A successor certificate over the same ceremony and shares, distinct + /// only by transcript; used where two candidates compete. + fn sibling_successor( + signers: &Signers, + prev: &VerifiedCertificate, + ceremony: &AuthorizedCeremony, + transcript: u8, + ) -> SignedCertificate { + let (public, _shares) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); + let body = CertificateBody::successor( + prev, + ceremony, + &public, + &TranscriptHash::new([transcript; 32]), + ) + .unwrap(); + cert::tests::sign(body, signers) + } + + #[test] + fn candidate_share_codec_round_trips_and_refuses_noncanonical() { + let signers = Signers::new(); + let (cert, mut shares) = cert_tests::genesis(&signers); + let candidate = mk_candidate(shares.remove(0), cert.body()); + let bytes = candidate.to_bytes(); + let restored = CandidateShare::from_bytes(bytes.as_slice()).unwrap(); + assert!(KeyShare::bind(restored, &cert).is_ok()); + + type Mutate = (&'static str, fn(&mut Vec)); + let cases: [Mutate; 7] = [ + ("empty", |b: &mut Vec| b.clear()), + ("version", |b: &mut Vec| b[0] = 2), + ("index-zero", |b: &mut Vec| { + b[33] = 0; + b[34] = 0; + }), + ("index-17", |b: &mut Vec| { + b[33] = 0; + b[34] = 17; + }), + ("zero-scalar", |b: &mut Vec| { + for x in &mut b[35..] { + *x = 0; + } + }), + ("truncated", |b: &mut Vec| { + b.truncate(66); + }), + ("trailing", |b: &mut Vec| b.push(0)), + ]; + for (tag, mutate) in cases { + let mut bad = bytes.to_vec(); + mutate(&mut bad); + assert!( + CandidateShare::from_bytes(&bad).is_err(), + "{tag} must be refused" + ); + } + } + + #[test] + fn candidate_persists_across_restart_until_activation() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + assert_eq!(store.pinned().unwrap().digest(), cert.digest()); + + let (s_succ, mut raw, ceremony) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let candidate = mk_candidate(raw.remove(0), &s_succ.body); + let digest = store + .install_candidate(&s_succ, candidate, &ceremony, &signers) + .unwrap(); + drop(store); + + // Restart: the persisted candidate is pending, the old pin stands. + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert_eq!(store.pinned().unwrap().digest(), cert.digest()); + assert_eq!(store.pending_candidate().unwrap().0, digest); + assert!(store.current_share().unwrap().is_some()); + drop(store); + + // Activate, then restart again: new pin, old share retired. + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let succ = verify_successor(&cert, &s_succ, &signers).unwrap(); + let receipts = receipts_for(&succ, &signers); + let activation = activation_for(&succ, &receipts, &signers); + store + .record_activation(&s_succ, &activation, &receipts, &signers) + .unwrap(); + drop(store); + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert_eq!(store.pinned().unwrap().digest(), succ.digest()); + assert!(store.pending_candidate().is_none()); + let share = store.current_share().unwrap().unwrap(); + assert_eq!(share.index().get(), 1); + } + + #[test] + fn orphan_share_files_are_swept_without_state_record() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + activated_genesis(&store, &signers); + } + let ndir = netdir(&dir, &network); + let stray = ndir.join(format!("share-{}", hex(&[7; 32]))); + fs::write(&stray, [9u8; 67]).unwrap(); + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert!(!stray.exists()); + assert!(store.current_share().unwrap().is_some()); + } + + #[test] + fn candidate_without_receipt_is_dropped_when_share_missing() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (s_succ, mut raw, ceremony) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let candidate = mk_candidate(raw.remove(0), &s_succ.body); + let digest = store + .install_candidate(&s_succ, candidate, &ceremony, &signers) + .unwrap(); + drop(store); + let ndir = netdir(&dir, &network); + fs::remove_file(ndir.join(format!("share-{}", hex(digest.as_bytes())))).unwrap(); + } + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert!(store.pending_candidate().is_none()); + assert!(store.current_share().unwrap().is_some()); + } + + #[test] + fn sent_receipt_with_missing_share_is_corruption() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (s_succ, mut raw, ceremony) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let candidate = mk_candidate(raw.remove(0), &s_succ.body); + let digest = store + .install_candidate(&s_succ, candidate, &ceremony, &signers) + .unwrap(); + store.mark_receipt_sent().unwrap(); + drop(store); + let ndir = netdir(&dir, &network); + fs::remove_file(ndir.join(format!("share-{}", hex(digest.as_bytes())))).unwrap(); + } + assert!(ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).is_err()); + } + + #[test] + fn retried_ceremony_erases_first_candidate_and_replay_is_noop() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + + // Ceremony X installs a candidate, then aborts. + let (s_x, mut raw_x, ceremony_x) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + assert_eq!( + store.accept_ceremony(&ceremony_x).unwrap(), + AcceptOutcome::Fresh + ); + let cand_x = mk_candidate(raw_x.remove(0), &s_x.body); + let digest_x = store + .install_candidate(&s_x, cand_x, &ceremony_x, &signers) + .unwrap(); + store.mark_receipt_sent().unwrap(); + let ndir = netdir(&dir, &signers.anchor().network); + assert!(ndir + .join(format!("share-{}", hex(digest_x.as_bytes()))) + .exists()); + + // Ceremony Y for the same predecessor: C1 erases X's candidate first. + let mut stmt_y = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_y.id = CeremonyId::new([61; 32]); + stmt_y.current = Some(cert.digest()); + stmt_y.epoch = cert.body().epoch().checked_next().unwrap(); + stmt_y.rekey = cert.body().rekey(); + let ceremony_y = + AuthorizedCeremony::verify_for(&cert, &signers.statement(&stmt_y), &signers).unwrap(); + assert_eq!( + store.accept_ceremony(&ceremony_y).unwrap(), + AcceptOutcome::Fresh + ); + assert!(!ndir + .join(format!("share-{}", hex(digest_x.as_bytes()))) + .exists()); + assert!(store.pending_candidate().is_none()); + + // Replaying X's authorization changes nothing. + assert_eq!( + store.accept_ceremony(&ceremony_x).unwrap(), + AcceptOutcome::Replay + ); + } + + #[test] + fn losing_candidate_is_erased_on_other_successor_activation() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + + let (s_d, mut raw, ceremony_d) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let cand = mk_candidate(raw.remove(0), &s_d.body); + let digest_d = store + .install_candidate(&s_d, cand, &ceremony_d, &signers) + .unwrap(); + let ndir = netdir(&dir, &signers.anchor().network); + assert!(ndir + .join(format!("share-{}", hex(digest_d.as_bytes()))) + .exists()); + + // A different successor of the same predecessor activates (C2). + let s_e = sibling_successor(&signers, &cert, &ceremony_d, 0xe0); + let cert_e = verify_successor(&cert, &s_e, &signers).unwrap(); + let receipts = receipts_for(&cert_e, &signers); + let activation = activation_for(&cert_e, &receipts, &signers); + store + .record_activation(&s_e, &activation, &receipts, &signers) + .unwrap(); + assert!(!ndir + .join(format!("share-{}", hex(digest_d.as_bytes()))) + .exists()); + assert!(store.pending_candidate().is_none()); + assert_eq!(store.pinned().unwrap().digest(), cert_e.digest()); + } + + #[test] + fn genesis_candidate_is_erased_by_later_genesis() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (cert_g1, mut shares) = cert_tests::genesis(&signers); + let share = shares.remove(0); + let signed_g1 = cert::tests::sign(cert_g1.body().clone(), &signers); + let cand = mk_candidate(share, cert_g1.body()); + let digest_g1 = store + .install_candidate(&signed_g1, cand, &signers.genesis(), &signers) + .unwrap(); + + // A second genesis activation erases G1 (C2, predecessor "none"). + let mut stmt_g2 = signers.body(CeremonyKind::Genesis, Threshold::new(2, 3).unwrap()); + stmt_g2.id = CeremonyId::new([0x92; 32]); + let ceremony_g2 = AuthorizedCeremony::verify_genesis( + &signers.anchor(), + &signers.statement(&stmt_g2), + &signers, + ) + .unwrap(); + let (public_g2, _shares_g2) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); + let body_g2 = CertificateBody::genesis( + &signers.anchor(), + &ceremony_g2, + &public_g2, + &TranscriptHash::new([0x93; 32]), + ) + .unwrap(); + let signed_g2 = cert::tests::sign(body_g2, &signers); + let cert_g2 = verify_certificate(&signers.anchor(), &signed_g2, &signers).unwrap(); + let receipts = receipts_for(&cert_g2, &signers); + let activation = activation_for(&cert_g2, &receipts, &signers); + store + .record_activation(&signed_g2, &activation, &receipts, &signers) + .unwrap(); + let ndir = netdir(&dir, &signers.anchor().network); + assert!(!ndir + .join(format!("share-{}", hex(digest_g1.as_bytes()))) + .exists()); + assert_eq!(store.pinned().unwrap().digest(), cert_g2.digest()); + } + + #[test] + fn activation_proof_advances_pin_past_state_crash() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let digest_old; + let digest_succ; + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + digest_old = cert.digest(); + let (s_succ, _raw, _ceremony) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let succ = verify_successor(&cert, &s_succ, &signers).unwrap(); + digest_succ = succ.digest(); + // Crash simulation: activated proof written, state never updated. + let receipts = receipts_for(&succ, &signers); + let activation = activation_for(&succ, &receipts, &signers); + let ndir = netdir(&dir, &network); + fs::write( + ndir.join(format!("proofs/{}", hex(digest_succ.as_bytes()))), + encode_proof(&s_succ, Some(&activation), &receipts), + ) + .unwrap(); + } + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert_eq!(store.pinned().unwrap().digest(), digest_succ); + let ndir = netdir(&dir, &network); + assert!(!ndir + .join(format!("share-{}", hex(digest_old.as_bytes()))) + .exists()); + } + + #[test] + fn state_pin_outside_activated_chain_is_corruption() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + activated_genesis(&store, &signers); + } + let ndir = netdir(&dir, &network); + let state_path = ndir.join("state"); + let mut state = decode_state(&fs::read(&state_path).unwrap()).unwrap(); + state.pin = Some(CertDigest::new([0xee; 32])); + fs::write(&state_path, encode_state(&state)).unwrap(); + assert!(ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).is_err()); + } + + #[test] + fn equivocated_successors_freeze_the_pin() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let old_digest; + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + old_digest = cert.digest(); + let (s_a, _ra, _ca) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let s_b = sibling_successor(&signers, &cert, &_ca, 0xb0); + let ndir = netdir(&dir, &network); + for s in [&s_a, &s_b] { + let cert = verify_successor(&cert, s, &signers).unwrap(); + let receipts = receipts_for(&cert, &signers); + let activation = activation_for(&cert, &receipts, &signers); + fs::write( + ndir.join(format!("proofs/{}", hex(cert.digest().as_bytes()))), + encode_proof(s, Some(&activation), &receipts), + ) + .unwrap(); + } + } + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert!(store.equivocation()); + assert_eq!(store.pinned().unwrap().digest(), old_digest); + } + + #[test] + fn concurrent_retirement_converges_once() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let s_succ; + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (s, mut raw, c) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let cand = mk_candidate(raw.remove(0), &s.body); + store.install_candidate(&s, cand, &c, &signers).unwrap(); + store.mark_receipt_sent().unwrap(); + s_succ = s; + } + let store = Arc::new(open(&dir, &signers)); + let pinned = store.pinned().unwrap(); + let succ = verify_successor(&pinned, &s_succ, &signers).unwrap(); + let receipts = receipts_for(&succ, &signers); + let activation = activation_for(&succ, &receipts, &signers); + let old_digest = pinned.digest(); + + let mut joins = Vec::new(); + for _ in 0..4 { + let store = Arc::clone(&store); + let s = s_succ.clone(); + let a = activation.clone(); + let r = receipts.clone(); + let signers = Signers::new(); + joins.push(std::thread::spawn(move || { + store.record_activation(&s, &a, &r, &signers).unwrap(); + })); + } + for j in joins { + j.join().unwrap(); + } + drop(store); + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert_eq!(store.pinned().unwrap().digest(), succ.digest()); + let ndir = netdir(&dir, &network); + assert!(!ndir + .join(format!("share-{}", hex(old_digest.as_bytes()))) + .exists()); + assert_eq!(store.retired().len(), 1); + } + + #[test] + fn dealt_marker_consumes_ceremony_id_across_restart() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let ceremony; + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (_s, _r, c) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + ceremony = c; + } + // Crash between dealt marker and state write: marker exists, id unseen. + let ndir = netdir(&dir, &network); + fs::write( + ndir.join(format!("dealt/{}", hex(ceremony.statement().id.as_bytes()))), + b"", + ) + .unwrap(); + let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + assert_eq!( + store.accept_ceremony(&ceremony).unwrap(), + AcceptOutcome::Replay + ); + } +} From 5e0467bd250b168f51fffd2605e4232bb1481960 Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 04:17:43 +0000 Subject: [PATCH 2/8] fix(threshold): verified-only activations, exclusive open lock, ceremony binding, state-before-unlink retirement; Details: TC-2221 --- tinycloud-threshold/README.md | 2 +- tinycloud-threshold/SPEC.md | 4 +- tinycloud-threshold/src/cert.rs | 37 +- tinycloud-threshold/src/store.rs | 810 ++++++++++++++++++++++++++----- 4 files changed, 717 insertions(+), 136 deletions(-) diff --git a/tinycloud-threshold/README.md b/tinycloud-threshold/README.md index 5642296d..46fbd38b 100644 --- a/tinycloud-threshold/README.md +++ b/tinycloud-threshold/README.md @@ -18,7 +18,7 @@ The crate is a workspace member with `publish = false`. No other workspace crate - Genesis and re-key DKG (GJKR New-DKG) over owner-closed rounds, the re-key under an owner-authorized `re-key` ceremony pinned to the current certificate with the genesis fault budget: consume-self member states for dealing, complaints, answers, Feldman extraction, the common R5 round and R6 reconstruction, and `dkg_verify_transcript`, which recomputes the outcome from the raw signed round records. - Redistribution for reshare and refresh over owner-closed rounds R1–R3: Feldman dealings from the current certificate's members to an authorized cohort with a new threshold, complaints and answers, deterministic selection of the first t eligible dealers, and `reshare_verify_transcript`. The master public key is unchanged; the output is a candidate for the successor certificate. - A real-ceremony chain test: genesis 2-of-3, reshare to 3-of-4 and back to 2-of-3, refresh, re-key, and further reshares and re-keys, with a certificate, readiness receipts and activation at every generation and each retired generation's shares refused in the next context. -- A durable share store: `CandidateShare::to_bytes`/`from_bytes` (canonical, secret-guarded bytes), one file per generation under a per-network directory, install before any readiness receipt, a pin recomputed from verified activated proofs on every open, and positive-proof retirement. +- A durable share store: `CandidateShare::to_bytes`/`from_bytes` (canonical bytes; the codec guards its own secret intermediates), one file per generation under a per-network directory, an exclusive per-network open lock, install before any readiness receipt, a pin recomputed from activations verified on every open, and retirement whose `state` commit always precedes the unlink. `VerifiedCertificate` supplies `KeyContext` and `PublicKeySet`; `KeyShare::bind` validates a candidate against its certificate. Raw candidate-scalar construction remains crate-private, and the dealer exists only under `#[cfg(test)]`. Redistribution does not erase earlier shares; old and new shares carry different key contexts. The `store` module implements the file-volume durable share store and retirement; node-runtime wiring remains a separate slice. Their contracts and the implemented canonical byte layouts are recorded in [SPEC.md](SPEC.md). diff --git a/tinycloud-threshold/SPEC.md b/tinycloud-threshold/SPEC.md index 510f6ab1..bc1dee5b 100644 --- a/tinycloud-threshold/SPEC.md +++ b/tinycloud-threshold/SPEC.md @@ -248,7 +248,7 @@ S0.4 adds the §3.8 redistribution for reshare and refresh: dealer and recipient S0.5 runs the §3.7 DKG under a `re-key` ceremony (`dkg_start` and `dkg_verify_transcript` accept kinds `genesis` and `re-key` and refuse reshare and refresh with `InvalidCeremony`), executes the full genesis → reshare → reshare → refresh → re-key chain with real ceremonies, certificates, readiness receipts and activation, and freezes the chain and re-key vectors described in the S0.5 appendix. -S0.6 adds the `store` module: the durable file-volume share store of §3.11–3.12 — `CandidateShare` canonical byte persistence, one proof and one share file per certificate, receipt-only-after-install, the activated-chain pin recomputed from verified proofs at open, and positive-proof retirement (superseded generations, losing candidates, unnamed files and replayed ceremony ids). Its layouts are in the S0.6 appendix below. +S0.6 adds the `store` module: the durable file-volume share store of §3.11–3.12 — `CandidateShare` canonical byte persistence, one proof and one share file per certificate, an exclusive per-network open lock, receipt-only-after-install (with the persisted file read back and re-bound), the activated-chain pin recomputed from fully verified proofs at open, member/anchor/pin checks on accepted ceremonies, and positive-proof retirement whose `state` commit precedes every unlink (superseded generations, losing candidates, unnamed files and replayed ceremony ids). Its layouts and rules are in the S0.6 appendix below. The dealer remains test-only. Node-runtime integration (volume mount, peer/DB pin sources, boot freshness, audit rows) is a separate slice. The crate remains an unwired workspace member. @@ -558,4 +558,4 @@ The `store` module keeps one member's share material for one network under ` zeroize::Zeroizing<[u8; 67]> { - let mut e = crate::encoding::Encoder::new(); - e.u8(1); - e.fixed(self.transcript.as_bytes()); - e.u16(self.index.get()); - e.fixed(&self.scalar.to_bytes()); - let bytes = e.finish(); let mut out = zeroize::Zeroizing::new([0u8; 67]); - out.copy_from_slice(&bytes); + let scalar = zeroize::Zeroizing::new(self.scalar.to_bytes()); + let (version, rest) = out.split_at_mut(1); + version.copy_from_slice(&[1u8]); + let (transcript, rest) = rest.split_at_mut(32); + transcript.copy_from_slice(self.transcript.as_bytes()); + let (index, scalar_slot) = rest.split_at_mut(2); + index.copy_from_slice(&self.index.get().to_be_bytes()); + scalar_slot.copy_from_slice(&*scalar); out } /// Canonical decode only: version 1, index 1..=16, scalar < r and != 0, exact length. Details: TC-2221 pub fn from_bytes(bytes: &[u8]) -> Result { - let mut d = crate::encoding::Decoder::new(bytes)?; - if d.u8()? != 1 { + if bytes.len() != 67 || bytes.first() != Some(&1) { return Err(Error::InvalidEncoding); } - let transcript = TranscriptHash::new(d.fixed()?); - let index = MemberIndex::new(d.u16()?)?; - let scalar = Scalar::from_bytes(&d.fixed::<32>()?)?; + let mut transcript = [0u8; 32]; + transcript.copy_from_slice(bytes.get(1..33).ok_or(Error::InvalidEncoding)?); + let index = MemberIndex::new(u16::from_be_bytes([ + *bytes.get(33).ok_or(Error::InvalidEncoding)?, + *bytes.get(34).ok_or(Error::InvalidEncoding)?, + ]))?; + let raw = zeroize::Zeroizing::new( + <[u8; 32]>::try_from(bytes.get(35..67).ok_or(Error::InvalidEncoding)?) + .map_err(|_| Error::InvalidEncoding)?, + ); + let scalar = Scalar::from_bytes(&*raw)?; if scalar.is_zero() { return Err(Error::InvalidScalar); } - d.finish()?; Ok(Self { - transcript, + transcript: TranscriptHash::new(transcript), index, scalar, }) diff --git a/tinycloud-threshold/src/store.rs b/tinycloud-threshold/src/store.rs index d9295b5f..71769d53 100644 --- a/tinycloud-threshold/src/store.rs +++ b/tinycloud-threshold/src/store.rs @@ -8,21 +8,27 @@ //! //! Layout under `//`: //! +//! * `lock`: advisory exclusive lock held for the store's lifetime; exactly +//! one `ShareStore` may be open for a network at a time. //! * `state`: versioned record of the pinned certificate, the pending //! candidate, accepted ceremony ids and retired generations. //! * `proofs/`: the signed certificate bytes plus, once //! activated, the owner activation statement and readiness receipts. //! * `share-`: `CandidateShare::to_bytes()` for this member's //! share under that certificate. -//! * `dealt/`: marker written before the first secret-bearing -//! message of that ceremony leaves; the id is consumed forever. +//! * `dealt/`: marker written once a ceremony id is consumed; +//! the id is never fresh again. //! -//! Every write is write-temp, fsync, rename, fsync-directory. The pin is the -//! highest-epoch certificate whose activation chain verifies from genesis; a -//! restart replays `proofs/` to recompute it and refuses a recorded `state` -//! pin the proofs cannot support. Shares are erased only on positive proof: -//! a verified activation that moves past them, a fresh ceremony for their -//! predecessor, or no record at all. +//! Every write is write-temp, fsync, rename, fsync-directory, and temporary +//! files are uniquely named, created with `create_new`, and zeroed plus +//! unlinked on any failure; the next open sweeps leftovers. The pin is the +//! highest certificate on the single chain verified from genesis through +//! `verify_activation` for every activated proof; a restart refuses a +//! recorded `state` pin the proofs cannot support. Shares are erased only +//! on positive proof — a verified activation that moves past them, a fresh +//! ceremony for their predecessor, or no record at all — and the `state` +//! record commits every removal before its file unlink, so a crash is +//! always recovered as "erasure owed", never as a missing promised share. use std::{ collections::BTreeMap, @@ -43,7 +49,7 @@ use crate::{ }, encoding::{Decoder, Encoder}, error::{CertError, Error}, - ids::{CeremonyId, CertDigest, MemberIndex, NetworkUrn}, + ids::{CeremonyId, CertDigest, Did, MemberIndex, NetworkUrn}, KeyShare, }; @@ -261,6 +267,9 @@ pub enum StoreError { /// Activated equivocation froze the pin; further activations are refused. #[error("activated equivocation froze the pin")] Equivocation, + /// Another `ShareStore` already holds this network's lock. + #[error("share store is already open for this network")] + Busy, } /// Result of presenting an authorized ceremony to the store. Details: TC-2221 @@ -274,42 +283,60 @@ pub enum AcceptOutcome { } /// One member's durable share store for one network. Details: TC-2221 +/// +/// `member` is this node's DID; a ceremony is accepted only when it names +/// this member among its recipients. `open` holds an exclusive advisory +/// lock on the network directory for the store's lifetime, so two openers +/// — same process or sibling processes — can never cache divergent state. pub struct ShareStore { network_dir: PathBuf, anchor: TrustAnchor, + member: Did, + _lock: fs::File, inner: Mutex, } struct Inner { state: State, pin: Option, + /// Every certificate on the verified activated chain, by digest. + verified: BTreeMap, equivocation: bool, } -fn create_file(path: &Path) -> Result { - let mut options = fs::OpenOptions::new(); - options.write(true).create(true).truncate(true); - #[cfg(unix)] - { - use std::os::unix::fs::OpenOptionsExt; - options.mode(0o600); - } - Ok(options.open(path)?) -} - fn write_atomic(path: &Path, bytes: &[u8]) -> Result<(), StoreError> { - let tmp = path.with_extension("tmp"); - { - use std::io::Write; - let mut file = create_file(&tmp)?; + use std::io::Write; + static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + let tag = NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let name = format!( + "{}.tmp-{}-{tag}", + path.file_name().and_then(|n| n.to_str()).unwrap_or("part"), + std::process::id() + ); + let tmp = path.with_file_name(name); + let result = (|| { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + let mut file = options.open(&tmp)?; file.write_all(bytes)?; file.sync_all()?; + drop(file); + fs::rename(&tmp, path)?; + if let Some(parent) = path.parent() { + fs::File::open(parent)?.sync_all()?; + } + Ok(()) + })(); + if result.is_err() { + // The temp may hold plaintext secret bytes: zero before unlinking. + let _ = erase_share_file(&tmp); } - fs::rename(&tmp, path)?; - if let Some(parent) = path.parent() { - fs::File::open(parent)?.sync_all()?; - } - Ok(()) + result } fn erase_share_file(path: &Path) -> Result<(), StoreError> { @@ -388,6 +415,26 @@ fn store_dir(dir: &Path, network: &NetworkUrn) -> PathBuf { dir.join(name) } +/// Zero and unlink any leftover `*.tmp-*` or `*.tmp` write-temp file under +/// the network directory. Runs inside `open` after the lock is taken. +fn sweep_temps(network_dir: &Path) -> Result<(), StoreError> { + for sub in [ + network_dir.to_path_buf(), + network_dir.join("proofs"), + network_dir.join("dealt"), + ] { + for entry in fs::read_dir(&sub)? { + let entry = entry?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + if name.ends_with(".tmp") || name.contains(".tmp-") { + let _ = erase_share_file(&entry.path()); + } + } + } + Ok(()) +} + impl ShareStore { /// Open or create the member's share directory and converge it. Details: TC-2221 /// @@ -399,6 +446,7 @@ impl ShareStore { dir: &Path, network: &NetworkUrn, anchor: TrustAnchor, + member: Did, v: &impl DidSignatureVerifier, ) -> Result { let network_dir = store_dir(dir, network); @@ -409,6 +457,26 @@ impl ShareStore { use std::os::unix::fs::PermissionsExt; fs::set_permissions(&network_dir, fs::Permissions::from_mode(0o700))?; } + // Fsync every parent so the directory hierarchy itself is durable + // even if power is lost before the first write completes. + fs::File::open(&network_dir)?.sync_all()?; + fs::File::open(dir)?.sync_all()?; + + // One opener per network directory, held for the store's lifetime. + let lock_path = network_dir.join("lock"); + let lock_file = { + let mut options = fs::OpenOptions::new(); + options.write(true).create(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + options.open(&lock_path)? + }; + lock_file.try_lock().map_err(|_| StoreError::Busy)?; + + sweep_temps(&network_dir)?; let mut state = match fs::read(network_dir.join("state")) { Ok(bytes) => decode_state(&bytes)?, @@ -444,8 +512,7 @@ impl ShareStore { let entry = entry?; let name = entry.file_name(); let Some(name) = name.to_str() else { continue }; - if name.ends_with(".tmp") { - let _ = fs::remove_file(entry.path()); + if name.ends_with(".tmp") || name.contains(".tmp-") { continue; } let (signed, activation) = decode_proof(&fs::read(entry.path())?)?; @@ -465,18 +532,33 @@ impl ShareStore { let mut verified: BTreeMap = BTreeMap::new(); let mut chain: Vec = Vec::new(); let mut equivocation = false; + // Every flagged activation is verified before it can extend the pin + // or count toward a fork; an activation flag alone proves nothing. let mut geneses: Vec = Vec::new(); for signed in proofs.values() { - if signed.body.previous().is_some() || !activations.contains_key(&signed.body.digest()) - { + let Some((statement, receipts)) = activations.get(&signed.body.digest()) else { + continue; + }; + if signed.body.previous().is_some() { continue; } - geneses.push(verify_certificate(&anchor, signed, v)?); + let verified_cert = verify_certificate(&anchor, signed, v)?; + verify_activation( + &verified_cert, + receipts, + ReadyQuorum::AllHolders, + statement, + v, + )?; + geneses.push(verified_cert); } geneses.sort_by(|a, b| a.digest().as_bytes().cmp(b.digest().as_bytes())); if geneses.len() > 1 { equivocation = true; } + for genesis in &geneses { + verified.insert(genesis.digest(), genesis.clone()); + } if let Some(first) = geneses.into_iter().next() { let mut current = first; chain.push(current.digest()); @@ -484,25 +566,30 @@ impl ShareStore { loop { let mut successors: Vec = Vec::new(); for signed in proofs.values() { - if signed.body.previous() != Some(¤t.digest()) - || !activations.contains_key(&signed.body.digest()) - { + if signed.body.previous() != Some(¤t.digest()) { continue; } - successors.push(verify_successor(¤t, signed, v)?); + let Some((statement, receipts)) = activations.get(&signed.body.digest()) else { + continue; + }; + let next = verify_successor(¤t, signed, v)?; + verify_activation(&next, receipts, ReadyQuorum::AllHolders, statement, v)?; + successors.push(next); } successors.sort_by(|a, b| a.digest().as_bytes().cmp(b.digest().as_bytes())); if successors.len() > 1 { - // The pin stops advancing at the fork. + // The pin stops advancing at the fork; every verified fork + // child is retained so a recorded pin among them is + // proven (not corruption) and their shares retire. equivocation = true; + for next in successors { + verified.insert(next.digest(), next); + } break; } let Some(next) = successors.into_iter().next() else { break; }; - let (statement, receipts) = - activations.get(&next.digest()).ok_or(StoreError::Corrupt)?; - verify_activation(&next, receipts, ReadyQuorum::AllHolders, statement, v)?; verified.insert(next.digest(), next.clone()); chain.push(next.digest()); current = next; @@ -511,27 +598,25 @@ impl ShareStore { let computed_pin = chain.last().copied(); let pin = computed_pin.and_then(|d| verified.get(&d).cloned()); - // A recorded pin the activated chain cannot prove is corruption. + // A recorded pin must be a verified activated certificate: either on + // the surviving chain or an equivocated fork child. Anything else is + // corruption. if let Some(recorded) = state.pin { - if !chain.contains(&recorded) { + if !verified.contains_key(&recorded) { return Err(StoreError::Corrupt); } } - // Candidate reconciliation. + // Candidate reconciliation. Chain membership and C2 losses are + // judged BEFORE the missing-file check: a candidate already retired + // by durable proof is settled even when its share is gone, while a + // still-live receipt-bearing candidate that lost its share is the + // only case that is truly corrupt. + let mut owed_erasures: Vec = Vec::new(); if let Some(candidate) = state.candidate.clone() { let share = share_path(&network_dir, &candidate.digest); let proof = proof_path(&network_dir, &candidate.digest); - if !share.exists() || !proof.exists() { - if candidate.receipt_sent { - // A receipt was signed over a durable install; a missing - // file means the member promised what it cannot serve. - return Err(StoreError::Corrupt); - } - let _ = erase_share_file(&share); - state.candidate = None; - state_dirty = true; - } else if chain.contains(&candidate.digest) { + if chain.contains(&candidate.digest) { // Already activated before the crash; the chain is the record. state.candidate = None; state_dirty = true; @@ -549,15 +634,31 @@ impl ShareStore { }), }; if lost { - erase_share_file(&share)?; + // Commit the retirement first; the unlink is owed work a + // crash simply re-discovers as S1. record_retired(&mut state, candidate.digest); state.candidate = None; - state_dirty = true; + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + owed_erasures.push(share); + } else if !share.exists() || !proof.exists() { + if candidate.receipt_sent { + // A receipt was signed over a durable install; a + // missing file means the member promised what it + // cannot serve. + return Err(StoreError::Corrupt); + } + state.candidate = None; + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + owed_erasures.push(share); } } } - // R1 and S1 over share files. + // R1 and S1 over share files. Every retirement is committed to + // `state` before its unlink; files left by a crash re-enter here as + // owed work on the next open. for entry in fs::read_dir(&network_dir)? { let entry = entry?; let name = entry.file_name(); @@ -577,13 +678,15 @@ impl ShareStore { if is_pin || is_candidate { continue; } - if chain.contains(&digest) { + if verified.contains_key(&digest) { // Superseded activated generation (R1). - erase_share_file(&entry.path())?; if !state.retired.iter().any(|r| r.digest == digest) { record_retired(&mut state, digest); - state_dirty = true; + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + state_dirty = false; } + owed_erasures.push(entry.path()); } else { // Outside the activated chain and named in no record: S1 // erases unconditionally (the pin's existence changes @@ -597,12 +700,25 @@ impl ShareStore { write_atomic(&network_dir.join("state"), &encode_state(&state))?; } + // Owed unlinks run only after every corresponding `state` commit. + for path in &owed_erasures { + erase_share_file(path)?; + } + + if state.pin != computed_pin || state_dirty { + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + } + Ok(Self { network_dir, anchor, + member, + _lock: lock_file, inner: Mutex::new(Inner { state, pin, + verified, equivocation, }), }) @@ -628,41 +744,49 @@ impl ShareStore { /// The bound share for the pinned certificate, or none when this member /// holds no share for it. Details: TC-2221 + /// + /// Returns `Equivocation` when a disk-discovered fork froze the pin: a + /// frozen store never exports share material. pub fn current_share(&self) -> Result, StoreError> { - let pin = self - .inner - .lock() - .map_err(|_| StoreError::Corrupt)? - .pin - .clone(); - let Some(pin) = pin else { return Ok(None) }; + use std::io::Read; + let inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + if inner.equivocation { + return Err(StoreError::Equivocation); + } + let Some(pin) = inner.pin.clone() else { + return Ok(None); + }; let path = share_path(&self.network_dir, &pin.digest()); if !path.exists() { return Ok(None); } - let bytes = Zeroizing::new(fs::read(path)?); - let candidate = CandidateShare::from_bytes(&bytes)?; + let mut bytes = Zeroizing::new([0u8; 67]); + fs::File::open(&path)?.read_exact(&mut *bytes)?; + let candidate = CandidateShare::from_bytes(&*bytes)?; Ok(Some(KeyShare::bind(candidate, &pin)?)) } /// The pending candidate's digest and member index, awaiting its /// readiness receipt. Details: TC-2221 pub fn pending_candidate(&self) -> Option<(CertDigest, MemberIndex)> { + use std::io::Read; let inner = self.inner.lock().ok()?; let candidate = inner.state.candidate.as_ref()?; let path = share_path(&self.network_dir, &candidate.digest); - let bytes = fs::read(path).ok()?; - let share = CandidateShare::from_bytes(&bytes).ok()?; + let mut bytes = Zeroizing::new([0u8; 67]); + fs::File::open(path).ok()?.read_exact(&mut *bytes).ok()?; + let share = CandidateShare::from_bytes(&*bytes).ok()?; Some((candidate.digest, share.index())) } /// Durably install a candidate share for `signed`. Details: TC-2221 /// /// `signed` must verify as the successor of the current pin (or as a - /// genesis before any pin). The candidate's bytes bind against the - /// certificate, then proof and share are written before the state record: - /// a crash before the record leaves orphan files the next open sweeps. - /// Returns the candidate certificate digest. + /// genesis before any pin), `ceremony` must be one this store already + /// accepted for that predecessor, and the persisted file is read back + /// and bound before the candidate is recorded — a receipt can never be + /// signed over bytes that were not verified durable. Returns the + /// candidate certificate digest. pub fn install_candidate( &self, signed: &SignedCertificate, @@ -670,10 +794,24 @@ impl ShareStore { ceremony: &AuthorizedCeremony, v: &impl DidSignatureVerifier, ) -> Result { + use std::io::Read; let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + if inner.equivocation { + return Err(StoreError::Equivocation); + } if inner.state.candidate.is_some() { return Err(StoreError::CandidateExists); } + // The candidate may only be installed under a ceremony this store + // accepted for the same predecessor. + if ceremony.anchor() != &self.anchor + || !inner + .state + .seen_ceremonies + .contains(&ceremony.statement().id) + { + return Err(Error::InvalidCeremony.into()); + } let (verified, predecessor) = match &inner.pin { Some(pin) => (verify_successor(pin, signed, v)?, Some(pin.digest())), None => { @@ -686,16 +824,24 @@ impl ShareStore { if ceremony.statement().current != predecessor { return Err(StoreError::InvalidActivation); } - let bytes = candidate.to_bytes(); - let reloaded = CandidateShare::from_bytes(bytes.as_slice())?; - // The written form must bind before it is persisted. - let _bound = KeyShare::bind(reloaded, &verified)?; let digest = verified.digest(); + let index = candidate.index(); write_atomic( &proof_path(&self.network_dir, &digest), &encode_proof(signed, None, &[]), )?; - write_atomic(&share_path(&self.network_dir, &digest), bytes.as_slice())?; + let bytes = candidate.to_bytes(); + write_atomic(&share_path(&self.network_dir, &digest), &*bytes)?; + drop(bytes); + // §5.3: read the persisted file back and bind it before the + // candidate is permitted to receive a receipt. + let mut persisted = Zeroizing::new([0u8; 67]); + fs::File::open(share_path(&self.network_dir, &digest))?.read_exact(&mut *persisted)?; + let reloaded = CandidateShare::from_bytes(&*persisted)?; + if reloaded.index() != index { + return Err(StoreError::Corrupt); + } + let _bound = KeyShare::bind(reloaded, &verified)?; inner.state.candidate = Some(CandidateRecord { digest, ceremony: ceremony.statement().id, @@ -736,15 +882,30 @@ impl ShareStore { if inner.equivocation { return Err(StoreError::Equivocation); } - // Idempotent: a certificate already activated on the pinned chain is - // a no-op, so concurrent or replayed recordings converge. - if inner - .pin - .as_ref() - .is_some_and(|pin| pin.digest() == signed.body.digest()) - { + // Idempotent: a certificate already on the verified activated chain + // is a no-op, so concurrent or replayed recordings converge. + if inner.verified.contains_key(&signed.body.digest()) { return Ok(()); } + // A certificate naming any verified chain certificate other than the + // pin as its predecessor is activated fork evidence: persist the + // activated proof durably first, then freeze the store. + if let Some(parent) = signed.body.previous().and_then(|p| inner.verified.get(p)) { + if inner + .pin + .as_ref() + .is_none_or(|pin| pin.digest() != parent.digest()) + { + let fork = verify_successor(parent, signed, v)?; + verify_activation(&fork, receipts, ReadyQuorum::AllHolders, activation, v)?; + write_atomic( + &proof_path(&self.network_dir, &fork.digest()), + &encode_proof(signed, Some(activation), receipts), + )?; + inner.equivocation = true; + return Err(StoreError::Equivocation); + } + } let verified = match &inner.pin { Some(pin) => verify_successor(pin, signed, v)?, None => verify_certificate(&self.anchor, signed, v)?, @@ -755,64 +916,92 @@ impl ShareStore { &proof_path(&self.network_dir, &digest), &encode_proof(signed, Some(activation), receipts), )?; + // `state` commits every removal BEFORE the unlink, so a crash is + // recovered as "erasure owed", never as a missing promised share. + let mut erasures: Vec = Vec::new(); // C2: a pending candidate that lost to this activation is erased; the // winning candidate's record is cleared (the chain is its record). if let Some(candidate) = inner.state.candidate.take() { - if candidate.digest != digest { - erase_share_file(&share_path(&self.network_dir, &candidate.digest))?; - if !inner + if candidate.digest != digest + && !inner .state .retired .iter() .any(|r| r.digest == candidate.digest) - { - record_retired(&mut inner.state, candidate.digest); - } + { + record_retired(&mut inner.state, candidate.digest); + erasures.push(share_path(&self.network_dir, &candidate.digest)); } } // R1: the outgoing pin's share is retired. if let Some(old_pin) = &inner.pin { let old = old_pin.digest(); - erase_share_file(&share_path(&self.network_dir, &old))?; if !inner.state.retired.iter().any(|r| r.digest == old) { record_retired(&mut inner.state, old); + erasures.push(share_path(&self.network_dir, &old)); } } - inner.pin = Some(verified); + inner.pin = Some(verified.clone()); + inner.verified.insert(digest, verified); inner.state.pin = Some(digest); - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state)) + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + for path in &erasures { + erase_share_file(path)?; + } + Ok(()) } /// Present a freshly verified authorized ceremony before any dealing. /// Details: TC-2221 /// - /// A new id erases any unactivated candidate for the same predecessor - /// (C1), writes the dealt marker and consumes the id; a replayed id - /// changes nothing. + /// The ceremony must be anchored to this store's owner/network, name this + /// member among its recipients, and pin the current certificate as its + /// predecessor. A fresh id then erases any unactivated candidate for that + /// predecessor (C1), writes the dealt marker and consumes the id; a + /// replayed id changes nothing. pub fn accept_ceremony( &self, ceremony: &AuthorizedCeremony, ) -> Result { - let id = ceremony.statement().id; let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; - if inner.state.seen_ceremonies.contains(&id) { + if inner.equivocation { + return Err(StoreError::Equivocation); + } + let statement = ceremony.statement(); + // A ceremony for another owner/network, another member, or a chain + // position this store is not at is not ours to consume. + if ceremony.anchor() != &self.anchor + || !statement.recipients.iter().any(|r| r.node == self.member) + || statement.current != inner.pin.as_ref().map(|p| p.digest()) + { + return Err(Error::InvalidCeremony.into()); + } + if inner.state.seen_ceremonies.contains(&statement.id) { return Ok(AcceptOutcome::Replay); } - let predecessor = ceremony.statement().current; + // `state` commits the removal before the unlink and before the dealt + // marker; a crash re-discovers the unlink as S1. + let mut owed: Option = None; if let Some(candidate) = &inner.state.candidate { - if candidate.predecessor == predecessor { + if candidate.predecessor == statement.current { let digest = candidate.digest; - erase_share_file(&share_path(&self.network_dir, &digest))?; record_retired(&mut inner.state, digest); inner.state.candidate = None; + owed = Some(share_path(&self.network_dir, &digest)); } } + inner.state.seen_ceremonies.push(statement.id); + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; write_atomic( - &self.network_dir.join("dealt").join(hex(id.as_bytes())), + &self + .network_dir + .join("dealt") + .join(hex(statement.id.as_bytes())), b"", )?; - inner.state.seen_ceremonies.push(id); - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + if let Some(path) = owed { + erase_share_file(&path)?; + } Ok(AcceptOutcome::Fresh) } } @@ -854,6 +1043,7 @@ mod tests { dir, &signers.anchor().network, signers.anchor().clone(), + Signers::did(1), signers, ) .unwrap() @@ -897,6 +1087,7 @@ mod tests { let share = shares.remove(0); let signed = cert::tests::sign(cert.body().clone(), signers); let candidate = mk_candidate(share, cert.body()); + store.accept_ceremony(&signers.genesis()).unwrap(); store .install_candidate(&signed, candidate, &signers.genesis(), signers) .unwrap(); @@ -986,20 +1177,38 @@ mod tests { 11, ); let candidate = mk_candidate(raw.remove(0), &s_succ.body); + assert_eq!( + store.accept_ceremony(&ceremony).unwrap(), + AcceptOutcome::Fresh + ); let digest = store .install_candidate(&s_succ, candidate, &ceremony, &signers) .unwrap(); drop(store); // Restart: the persisted candidate is pending, the old pin stands. - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert_eq!(store.pinned().unwrap().digest(), cert.digest()); assert_eq!(store.pending_candidate().unwrap().0, digest); assert!(store.current_share().unwrap().is_some()); drop(store); // Activate, then restart again: new pin, old share retired. - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); let succ = verify_successor(&cert, &s_succ, &signers).unwrap(); let receipts = receipts_for(&succ, &signers); let activation = activation_for(&succ, &receipts, &signers); @@ -1007,7 +1216,14 @@ mod tests { .record_activation(&s_succ, &activation, &receipts, &signers) .unwrap(); drop(store); - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert_eq!(store.pinned().unwrap().digest(), succ.digest()); assert!(store.pending_candidate().is_none()); let share = store.current_share().unwrap().unwrap(); @@ -1026,7 +1242,14 @@ mod tests { let ndir = netdir(&dir, &network); let stray = ndir.join(format!("share-{}", hex(&[7; 32]))); fs::write(&stray, [9u8; 67]).unwrap(); - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert!(!stray.exists()); assert!(store.current_share().unwrap().is_some()); } @@ -1047,6 +1270,7 @@ mod tests { 11, ); let candidate = mk_candidate(raw.remove(0), &s_succ.body); + store.accept_ceremony(&ceremony).unwrap(); let digest = store .install_candidate(&s_succ, candidate, &ceremony, &signers) .unwrap(); @@ -1054,7 +1278,14 @@ mod tests { let ndir = netdir(&dir, &network); fs::remove_file(ndir.join(format!("share-{}", hex(digest.as_bytes())))).unwrap(); } - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert!(store.pending_candidate().is_none()); assert!(store.current_share().unwrap().is_some()); } @@ -1075,6 +1306,7 @@ mod tests { 11, ); let candidate = mk_candidate(raw.remove(0), &s_succ.body); + store.accept_ceremony(&ceremony).unwrap(); let digest = store .install_candidate(&s_succ, candidate, &ceremony, &signers) .unwrap(); @@ -1083,7 +1315,14 @@ mod tests { let ndir = netdir(&dir, &network); fs::remove_file(ndir.join(format!("share-{}", hex(digest.as_bytes())))).unwrap(); } - assert!(ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).is_err()); + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); } #[test] @@ -1154,6 +1393,7 @@ mod tests { 11, ); let cand = mk_candidate(raw.remove(0), &s_d.body); + store.accept_ceremony(&ceremony_d).unwrap(); let digest_d = store .install_candidate(&s_d, cand, &ceremony_d, &signers) .unwrap(); @@ -1186,6 +1426,7 @@ mod tests { let share = shares.remove(0); let signed_g1 = cert::tests::sign(cert_g1.body().clone(), &signers); let cand = mk_candidate(share, cert_g1.body()); + store.accept_ceremony(&signers.genesis()).unwrap(); let digest_g1 = store .install_candidate(&signed_g1, cand, &signers.genesis(), &signers) .unwrap(); @@ -1251,7 +1492,14 @@ mod tests { ) .unwrap(); } - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert_eq!(store.pinned().unwrap().digest(), digest_succ); let ndir = netdir(&dir, &network); assert!(!ndir @@ -1273,7 +1521,14 @@ mod tests { let mut state = decode_state(&fs::read(&state_path).unwrap()).unwrap(); state.pin = Some(CertDigest::new([0xee; 32])); fs::write(&state_path, encode_state(&state)).unwrap(); - assert!(ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).is_err()); + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); } #[test] @@ -1306,7 +1561,14 @@ mod tests { .unwrap(); } } - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert!(store.equivocation()); assert_eq!(store.pinned().unwrap().digest(), old_digest); } @@ -1328,6 +1590,7 @@ mod tests { 11, ); let cand = mk_candidate(raw.remove(0), &s.body); + store.accept_ceremony(&c).unwrap(); store.install_candidate(&s, cand, &c, &signers).unwrap(); store.mark_receipt_sent().unwrap(); s_succ = s; @@ -1354,7 +1617,14 @@ mod tests { j.join().unwrap(); } drop(store); - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert_eq!(store.pinned().unwrap().digest(), succ.digest()); let ndir = netdir(&dir, &network); assert!(!ndir @@ -1388,10 +1658,312 @@ mod tests { b"", ) .unwrap(); - let store = ShareStore::open(&dir, &network, signers.anchor().clone(), &signers).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); assert_eq!( store.accept_ceremony(&ceremony).unwrap(), AcceptOutcome::Replay ); } + + #[test] + fn unverified_activation_flag_is_never_trusted() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (s_succ, mut raw, ceremony) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let candidate = mk_candidate(raw.remove(0), &s_succ.body); + store.accept_ceremony(&ceremony).unwrap(); + store + .install_candidate(&s_succ, candidate, &ceremony, &signers) + .unwrap(); + // A flagged activation with an empty statement and zero receipts. + let succ = verify_successor(&cert, &s_succ, &signers).unwrap(); + let activation = SignedStatement { + bytes: Vec::new(), + signature: Vec::new(), + }; + let ndir = netdir(&dir, &network); + fs::write( + ndir.join(format!("proofs/{}", hex(succ.digest().as_bytes()))), + encode_proof(&s_succ, Some(&activation), &[]), + ) + .unwrap(); + } + // The flag alone must not pin the successor. + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); + } + + #[test] + fn second_opener_is_refused_while_first_holds_lock() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let network = signers.anchor().network.clone(); + assert!(matches!( + ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ), + Err(StoreError::Busy) + )); + drop(store); + let _reopened = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + } + + #[test] + fn foreign_or_unbound_ceremony_is_rejected() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let ndir = netdir(&dir, &signers.anchor().network); + + // Install a candidate under the genuine successor ceremony. + let (s_x, mut raw, ceremony_x) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&ceremony_x).unwrap(); + let cand = mk_candidate(raw.remove(0), &s_x.body); + let digest_x = store + .install_candidate(&s_x, cand, &ceremony_x, &signers) + .unwrap(); + let share_path = ndir.join(format!("share-{}", hex(digest_x.as_bytes()))); + assert!(share_path.exists()); + + // A ceremony verified under a different anchor is not ours. + let other = TrustAnchor { + expected_owner: Signers::did(9), + network: NetworkUrn::new("tinycloud:test:other").unwrap(), + }; + let mut foreign_stmt = signers.body(CeremonyKind::Genesis, Threshold::new(2, 3).unwrap()); + foreign_stmt.id = CeremonyId::new([0xf1; 32]); + foreign_stmt.owner = Signers::did(9); + foreign_stmt.network = other.network.clone(); + let foreign = + AuthorizedCeremony::verify_genesis(&other, &signers.statement(&foreign_stmt), &signers) + .unwrap(); + assert!(store.accept_ceremony(&foreign).is_err()); + assert!(share_path.exists()); + + // A ceremony that drops this member from its recipients is rejected. + let mut stmt = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt.id = CeremonyId::new([0xf2; 32]); + stmt.current = Some(cert.digest()); + stmt.epoch = cert.body().epoch().checked_next().unwrap(); + stmt.rekey = cert.body().rekey(); + stmt.recipients[0].node = Signers::did(9); + let excluding = + AuthorizedCeremony::verify_for(&cert, &signers.statement(&stmt), &signers).unwrap(); + assert!(store.accept_ceremony(&excluding).is_err()); + assert!(share_path.exists()); + + // A ceremony for a stale chain position is rejected: activate the + // candidate, then present a different authorized ceremony still + // pinned to the old certificate. + let cert_x = verify_successor(&cert, &s_x, &signers).unwrap(); + let receipts = receipts_for(&cert_x, &signers); + let activation = activation_for(&cert_x, &receipts, &signers); + store + .record_activation(&s_x, &activation, &receipts, &signers) + .unwrap(); + let mut stmt = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt.id = CeremonyId::new([0xf3; 32]); + stmt.current = Some(cert.digest()); + stmt.epoch = cert.body().epoch().checked_next().unwrap(); + stmt.rekey = cert.body().rekey(); + let stale = + AuthorizedCeremony::verify_for(&cert, &signers.statement(&stmt), &signers).unwrap(); + assert!(store.accept_ceremony(&stale).is_err()); + + drop(store); + } + + #[test] + fn forked_activation_is_persisted_and_freezes_everything() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let digest_b; + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (s1, mut raw, c1) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c1).unwrap(); + let cand = mk_candidate(raw.remove(0), &s1.body); + store.install_candidate(&s1, cand, &c1, &signers).unwrap(); + store.mark_receipt_sent().unwrap(); + // The candidate activates; then a sibling activation over the + // same predecessor arrives: fork evidence for a verified + // predecessor that is no longer the pin. + let cert_a = verify_successor(&cert, &s1, &signers).unwrap(); + let receipts_a = receipts_for(&cert_a, &signers); + let activation_a = activation_for(&cert_a, &receipts_a, &signers); + store + .record_activation(&s1, &activation_a, &receipts_a, &signers) + .unwrap(); + let s_b = sibling_successor(&signers, &cert, &c1, 0xb1); + let cert_b = verify_successor(&cert, &s_b, &signers).unwrap(); + digest_b = cert_b.digest(); + let receipts_b = receipts_for(&cert_b, &signers); + let activation_b = activation_for(&cert_b, &receipts_b, &signers); + assert!(matches!( + store.record_activation(&s_b, &activation_b, &receipts_b, &signers), + Err(StoreError::Equivocation) + )); + assert!(store.equivocation()); + // The fork proof must be durable. + assert!(netdir(&dir, &network) + .join(format!("proofs/{}", hex(digest_b.as_bytes()))) + .exists()); + // A frozen store exports nothing and accepts nothing new. + assert!(matches!( + store.current_share(), + Err(StoreError::Equivocation) + )); + let mut stmt_c = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_c.id = CeremonyId::new([0xcc; 32]); + stmt_c.current = Some(cert.digest()); + stmt_c.epoch = cert.body().epoch().checked_next().unwrap(); + stmt_c.rekey = cert.body().rekey(); + let c_ceremony = + AuthorizedCeremony::verify_for(&cert, &signers.statement(&stmt_c), &signers) + .unwrap(); + assert!(store.accept_ceremony(&c_ceremony).is_err()); + } + // The freeze persists across restart. + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(store.equivocation()); + let _ = digest_b; + } + + #[test] + fn retired_receipted_candidate_with_missing_share_converges() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let digest_c; + let succ_digest; + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + let (s_c, mut raw, ceremony_c) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&ceremony_c).unwrap(); + let cand = mk_candidate(raw.remove(0), &s_c.body); + digest_c = store + .install_candidate(&s_c, cand, &ceremony_c, &signers) + .unwrap(); + store.mark_receipt_sent().unwrap(); + // A different successor activates; the proof lands but the + // candidate's share file is lost before `state` commits (the + // crash window this ordering now survives). + let s_w = sibling_successor(&signers, &cert, &ceremony_c, 0xd0); + let cert_w = verify_successor(&cert, &s_w, &signers).unwrap(); + succ_digest = cert_w.digest(); + let receipts = receipts_for(&cert_w, &signers); + let activation = activation_for(&cert_w, &receipts, &signers); + let ndir = netdir(&dir, &network); + fs::write( + ndir.join(format!("proofs/{}", hex(succ_digest.as_bytes()))), + encode_proof(&s_w, Some(&activation), &receipts), + ) + .unwrap(); + fs::remove_file(ndir.join(format!("share-{}", hex(digest_c.as_bytes())))).unwrap(); + } + // Restart: the durable retirement proof settles the candidate even + // though its share is gone; the store converges, not Corrupt. + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert_eq!(store.pinned().unwrap().digest(), succ_digest); + assert!(store.pending_candidate().is_none()); + assert!(store.retired().iter().any(|r| r.digest == digest_c)); + } + + #[test] + fn leftover_temp_files_are_swept_on_open() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + activated_genesis(&store, &signers); + } + let ndir = netdir(&dir, &network); + let plain = ndir.join("share-0123456789abcdef.tmp-99-7"); + fs::write(&plain, [0x55u8; 67]).unwrap(); + let old_style = ndir.join("state.tmp"); + fs::write(&old_style, [0x55u8; 4]).unwrap(); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(!plain.exists()); + assert!(!old_style.exists()); + assert!(store.current_share().unwrap().is_some()); + } } From db1be172f1ff046bf3e29dffd092ba1f2100c4f9 Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 04:36:10 +0000 Subject: [PATCH 3/8] fix(threshold): single admissible ceremony, frozen reopen classifies fork descendants, strict temp sweep, full parent fsync; Details: TC-2221 --- tinycloud-threshold/SPEC.md | 4 +- tinycloud-threshold/src/store.rs | 320 ++++++++++++++++++++++++++++--- 2 files changed, 291 insertions(+), 33 deletions(-) diff --git a/tinycloud-threshold/SPEC.md b/tinycloud-threshold/SPEC.md index bc1dee5b..4be4ff6e 100644 --- a/tinycloud-threshold/SPEC.md +++ b/tinycloud-threshold/SPEC.md @@ -556,6 +556,6 @@ The `store` module keeps one member's share material for one network under `` +//! directory is self-contained), and provisions a fresh ceremony and +//! directory for the retained chain. The store itself never selects a +//! branch or resumes service. use std::{ collections::BTreeMap, @@ -71,10 +87,20 @@ struct CandidateRecord { receipt_sent: bool, } +/// The single ceremony admissible for installation: the most recently +/// accepted fresh id and the predecessor it was bound to. A superseded or +/// retired id stays in `seen_ceremonies` (history) but can never install. +#[derive(Clone, Debug)] +struct ActiveCeremony { + id: CeremonyId, + predecessor: Option, +} + #[derive(Clone, Debug, Default)] struct State { pin: Option, candidate: Option, + active_ceremony: Option, seen_ceremonies: Vec, retired: Vec, seq: u64, @@ -82,7 +108,7 @@ struct State { fn encode_state(state: &State) -> Vec { let mut e = Encoder::new(); - e.u8(1); + e.u8(2); e.u8(u8::from(state.pin.is_some())); if let Some(pin) = state.pin { e.fixed(pin.as_bytes()); @@ -97,6 +123,14 @@ fn encode_state(state: &State) -> Vec { } e.u8(u8::from(candidate.receipt_sent)); } + e.u8(u8::from(state.active_ceremony.is_some())); + if let Some(active) = &state.active_ceremony { + e.fixed(active.id.as_bytes()); + e.u8(u8::from(active.predecessor.is_some())); + if let Some(predecessor) = active.predecessor { + e.fixed(predecessor.as_bytes()); + } + } // Ceremony ids and retired entries are bounded by the on-disk format cap // enforced in decode_state; u32 cannot overflow below memory limits. e.u32(state.seen_ceremonies.len() as u32); @@ -114,7 +148,7 @@ fn encode_state(state: &State) -> Vec { fn decode_state(bytes: &[u8]) -> Result { let mut d = Decoder::new(bytes)?; - if d.u8()? != 1 { + if d.u8()? != 2 { return Err(Error::InvalidEncoding.into()); } let pin = if d.u8()? == 1 { @@ -136,6 +170,18 @@ fn decode_state(bytes: &[u8]) -> Result { } else { None }; + let active_ceremony = if d.u8()? == 1 { + Some(ActiveCeremony { + id: CeremonyId::new(d.fixed()?), + predecessor: if d.u8()? == 1 { + Some(CertDigest::new(d.fixed()?)) + } else { + None + }, + }) + } else { + None + }; let seen_len = usize::try_from(d.u32()?).map_err(|_| Error::InvalidEncoding)?; if seen_len > 1_000_000 { return Err(Error::InvalidEncoding.into()); @@ -160,6 +206,7 @@ fn decode_state(bytes: &[u8]) -> Result { Ok(State { pin, candidate, + active_ceremony, seen_ceremonies, retired, seq, @@ -270,6 +317,9 @@ pub enum StoreError { /// Another `ShareStore` already holds this network's lock. #[error("share store is already open for this network")] Busy, + /// The filesystem cannot provide the exclusive open lock. + #[error("share store lock unsupported on this filesystem")] + LockUnsupported, } /// Result of presenting an authorized ceremony to the store. Details: TC-2221 @@ -416,7 +466,9 @@ fn store_dir(dir: &Path, network: &NetworkUrn) -> PathBuf { } /// Zero and unlink any leftover `*.tmp-*` or `*.tmp` write-temp file under -/// the network directory. Runs inside `open` after the lock is taken. +/// the network directory. Runs inside `open` after the lock is taken; a +/// temp that cannot be removed (it may hold plaintext secret bytes) refuses +/// the open rather than surviving silently. fn sweep_temps(network_dir: &Path) -> Result<(), StoreError> { for sub in [ network_dir.to_path_buf(), @@ -428,7 +480,7 @@ fn sweep_temps(network_dir: &Path) -> Result<(), StoreError> { let name = entry.file_name(); let Some(name) = name.to_str() else { continue }; if name.ends_with(".tmp") || name.contains(".tmp-") { - let _ = erase_share_file(&entry.path()); + erase_share_file(&entry.path())?; } } } @@ -457,10 +509,14 @@ impl ShareStore { use std::os::unix::fs::PermissionsExt; fs::set_permissions(&network_dir, fs::Permissions::from_mode(0o700))?; } - // Fsync every parent so the directory hierarchy itself is durable - // even if power is lost before the first write completes. - fs::File::open(&network_dir)?.sync_all()?; - fs::File::open(dir)?.sync_all()?; + // Fsync every directory in the created hierarchy — including the + // parents of the caller's base directory — so the tree itself is + // durable even if power is lost before the first write completes. + let mut ancestor = Some(network_dir.as_path()); + while let Some(dir_path) = ancestor { + fs::File::open(dir_path)?.sync_all()?; + ancestor = dir_path.parent(); + } // One opener per network directory, held for the store's lifetime. let lock_path = network_dir.join("lock"); @@ -474,7 +530,13 @@ impl ShareStore { } options.open(&lock_path)? }; - lock_file.try_lock().map_err(|_| StoreError::Busy)?; + // A held contender is Busy; an implementation that cannot lock at + // all (an unsupported filesystem) is a distinct refusal: the store + // will not run where the single-opener guarantee cannot be made. + lock_file.try_lock().map_err(|e| match e { + std::fs::TryLockError::WouldBlock => StoreError::Busy, + std::fs::TryLockError::Error(_) => StoreError::LockUnsupported, + })?; sweep_temps(&network_dir)?; @@ -595,12 +657,42 @@ impl ShareStore { current = next; } } + // Fixpoint: descendants of ANY verified activated certificate — + // including descendants of a fork's losing branch — are verified + // activated too. They cannot move the pin (the fork already froze + // it) but they keep their recorded pin proven and their shares + // retire by the same rules instead of reading as corruption. + loop { + let mut added = Vec::new(); + for signed in proofs.values() { + let digest = signed.body.digest(); + if verified.contains_key(&digest) { + continue; + } + let Some(parent) = signed.body.previous().and_then(|p| verified.get(p)) else { + continue; + }; + let Some((statement, receipts)) = activations.get(&digest) else { + continue; + }; + let next = verify_successor(parent, signed, v)?; + verify_activation(&next, receipts, ReadyQuorum::AllHolders, statement, v)?; + added.push(next); + } + if added.is_empty() { + break; + } + for next in added { + verified.insert(next.digest(), next); + } + } + let computed_pin = chain.last().copied(); let pin = computed_pin.and_then(|d| verified.get(&d).cloned()); // A recorded pin must be a verified activated certificate: either on - // the surviving chain or an equivocated fork child. Anything else is - // corruption. + // the surviving chain or an equivocated branch descendant. Anything + // else is corruption. if let Some(recorded) = state.pin { if !verified.contains_key(&recorded) { return Err(StoreError::Corrupt); @@ -802,14 +894,16 @@ impl ShareStore { if inner.state.candidate.is_some() { return Err(StoreError::CandidateExists); } - // The candidate may only be installed under a ceremony this store - // accepted for the same predecessor. - if ceremony.anchor() != &self.anchor - || !inner - .state - .seen_ceremonies - .contains(&ceremony.statement().id) - { + // Only the currently admissible ceremony — the most recently + // accepted fresh id for the current pin — may install. Superseded + // and retired ids stay in `seen` as history but can never install, + // so a candidate erased by C1 cannot be resurrected. + let admissible = inner + .state + .active_ceremony + .as_ref() + .is_some_and(|a| a.id == ceremony.statement().id); + if ceremony.anchor() != &self.anchor || !admissible { return Err(Error::InvalidCeremony.into()); } let (verified, predecessor) = match &inner.pin { @@ -944,6 +1038,9 @@ impl ShareStore { inner.pin = Some(verified.clone()); inner.verified.insert(digest, verified); inner.state.pin = Some(digest); + // Ceremonies accepted against the old pin are stale: the next + // admissible ceremony must be accepted afresh for this pin. + inner.state.active_ceremony = None; write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; for path in &erasures { erase_share_file(path)?; @@ -979,8 +1076,9 @@ impl ShareStore { if inner.state.seen_ceremonies.contains(&statement.id) { return Ok(AcceptOutcome::Replay); } - // `state` commits the removal before the unlink and before the dealt - // marker; a crash re-discovers the unlink as S1. + // `state` commits the removal and the new admissible ceremony before + // the unlink and before the dealt marker; a crash re-discovers the + // unlink as S1 and the marker's id as consumed. let mut owed: Option = None; if let Some(candidate) = &inner.state.candidate { if candidate.predecessor == statement.current { @@ -991,17 +1089,25 @@ impl ShareStore { } } inner.state.seen_ceremonies.push(statement.id); + inner.state.active_ceremony = Some(ActiveCeremony { + id: statement.id, + predecessor: statement.current, + }); write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; - write_atomic( + let marker = write_atomic( &self .network_dir .join("dealt") .join(hex(statement.id.as_bytes())), b"", - )?; + ); + // The erasure debt is already durable in `state.retired`; attempt the + // unlink even if the marker write failed so a retry (or the next + // open) never finds the retired share still live. if let Some(path) = owed { erase_share_file(&path)?; } + marker?; Ok(AcceptOutcome::Fresh) } } @@ -1966,4 +2072,156 @@ mod tests { assert!(!old_style.exists()); assert!(store.current_share().unwrap().is_some()); } + + #[test] + fn superseded_ceremony_id_cannot_reinstall() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + + // Ceremony X installs, then ceremony Y supersedes it (C1 erases X). + let (s_x, mut raw_x, ceremony_x) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&ceremony_x).unwrap(); + let cand_x = mk_candidate(raw_x.remove(0), &s_x.body); + let cand_x_again = mk_candidate(raw_x.remove(0), &s_x.body); + store + .install_candidate(&s_x, cand_x, &ceremony_x, &signers) + .unwrap(); + + let mut stmt_y = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_y.id = CeremonyId::new([0x61; 32]); + stmt_y.current = Some(cert.digest()); + stmt_y.epoch = cert.body().epoch().checked_next().unwrap(); + stmt_y.rekey = cert.body().rekey(); + let ceremony_y = + AuthorizedCeremony::verify_for(&cert, &signers.statement(&stmt_y), &signers).unwrap(); + assert_eq!( + store.accept_ceremony(&ceremony_y).unwrap(), + AcceptOutcome::Fresh + ); + + // X's id is still in `seen` (history) but is no longer admissible: + // the retired candidate cannot be resurrected. + assert!(matches!( + store.install_candidate(&s_x, cand_x_again, &ceremony_x, &signers), + Err(StoreError::Crypto(Error::InvalidCeremony)) + )); + + // A candidate under Y — the admissible ceremony — still installs. + let s_y = sibling_successor(&signers, &cert, &ceremony_y, 0x70); + let (_pub_y, mut shares_y) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); + let cand_y = mk_candidate(shares_y.remove(0), &s_y.body); + store + .install_candidate(&s_y, cand_y, &ceremony_y, &signers) + .unwrap(); + } + + #[test] + fn frozen_store_reopens_classifying_fork_descendants() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + let (_signed, cert) = activated_genesis(&store, &signers); + // G -> A activates and pins; a sibling A' freezes the store. + let (s_a, mut raw, c_a) = cert_tests::successor( + &signers, + &cert, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_a).unwrap(); + let cand = mk_candidate(raw.remove(0), &s_a.body); + store.install_candidate(&s_a, cand, &c_a, &signers).unwrap(); + store.mark_receipt_sent().unwrap(); + let cert_a = verify_successor(&cert, &s_a, &signers).unwrap(); + let receipts_a = receipts_for(&cert_a, &signers); + let activation_a = activation_for(&cert_a, &receipts_a, &signers); + store + .record_activation(&s_a, &activation_a, &receipts_a, &signers) + .unwrap(); + let s_fork = sibling_successor(&signers, &cert, &c_a, 0xf0); + let cert_fork = verify_successor(&cert, &s_fork, &signers).unwrap(); + let receipts_f = receipts_for(&cert_fork, &signers); + let activation_f = activation_for(&cert_fork, &receipts_f, &signers); + assert!(matches!( + store.record_activation(&s_fork, &activation_f, &receipts_f, &signers), + Err(StoreError::Equivocation) + )); + // B: a descendant of the pinned branch, activated durably. + let (s_b, _r, _c) = cert_tests::successor( + &signers, + &cert_a, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + let cert_b = verify_successor(&cert_a, &s_b, &signers).unwrap(); + let receipts_b = receipts_for(&cert_b, &signers); + let activation_b = activation_for(&cert_b, &receipts_b, &signers); + let ndir = netdir(&dir, &network); + fs::write( + ndir.join(format!("proofs/{}", hex(cert_b.digest().as_bytes()))), + encode_proof(&s_b, Some(&activation_b), &receipts_b), + ) + .unwrap(); + } + // Restart: the frozen store reopens as frozen — the descendant of + // the losing branch is classified, not corruption. + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(store.equivocation()); + assert!(matches!( + store.current_share(), + Err(StoreError::Equivocation) + )); + } + + #[test] + fn unremovable_temp_refuses_open() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + activated_genesis(&store, &signers); + } + // A directory where a temp file is expected cannot be overwritten or + // unlinked as a file — the sweep must refuse the open. + let ndir = netdir(&dir, &network); + let stuck = ndir.join("state.tmp-1-1"); + fs::create_dir(&stuck).unwrap(); + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); + fs::remove_dir(&stuck).unwrap(); + let _reopened = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + } } From 886d7ce73a80749a5eb62e6099fb03f5d89e54b1 Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 05:19:39 +0000 Subject: [PATCH 4/8] test(threshold): seeded model-based invariant suite + bind admissible ceremony to its pin; Details: TC-2221 --- tinycloud-threshold/src/store.rs | 719 ++++++++++++++++++++++++++++++- 1 file changed, 712 insertions(+), 7 deletions(-) diff --git a/tinycloud-threshold/src/store.rs b/tinycloud-threshold/src/store.rs index 28159952..fea5c021 100644 --- a/tinycloud-threshold/src/store.rs +++ b/tinycloud-threshold/src/store.rs @@ -895,14 +895,16 @@ impl ShareStore { return Err(StoreError::CandidateExists); } // Only the currently admissible ceremony — the most recently - // accepted fresh id for the current pin — may install. Superseded + // accepted fresh id FOR the current pin — may install. Superseded // and retired ids stay in `seen` as history but can never install, - // so a candidate erased by C1 cannot be resurrected. - let admissible = inner - .state - .active_ceremony - .as_ref() - .is_some_and(|a| a.id == ceremony.statement().id); + // so a candidate erased by C1 cannot be resurrected. The recorded + // predecessor must equal the actual pin: a crash can restore an + // `active_ceremony` whose accepted-for pin the proofs then move + // past, and that id is contextually stale, never admissible. + let admissible = inner.state.active_ceremony.as_ref().is_some_and(|a| { + a.id == ceremony.statement().id + && a.predecessor == inner.pin.as_ref().map(|p| p.digest()) + }); if ceremony.anchor() != &self.anchor || !admissible { return Err(Error::InvalidCeremony.into()); } @@ -2224,4 +2226,707 @@ mod tests { ) .unwrap(); } + + /// Seeded model-based test: a scripted scheduler drives random sequences + /// of accept / install / receipt / activate / fork / reopen / + /// crash-at-every-step / second-opener over a real `ShareStore`, and a + /// reference model asserts the store's invariants after every step. + /// A failing run prints its seed so it can be replayed. Details: TC-2221 + mod model { + use super::super::tests::*; + use std::collections::{BTreeMap, BTreeSet, VecDeque}; + + /// splitmix64; cheap and stable across platforms for reproducing runs. + struct Rng(u64); + + impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E3779B97F4A7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58476D1CE4E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D049BB133111EB); + z ^ (z >> 31) + } + + fn pick(&mut self, n: u64) -> u64 { + self.next() % n + } + } + + /// The reference model: what a correct store may expose after each + /// step, derived only from successful operations (never from the + /// store's private state). + #[derive(Default)] + struct Model { + /// The activated chain, genesis first; the last entry is the pin. + pins: Vec, + /// Signed certificates ever produced, by digest. + signed: BTreeMap<[u8; 32], SignedCertificate>, + /// Ceremonies accepted for each pinned predecessor, by the + /// predecessor digest; used to build fork siblings. + ceremonies_by_pred: BTreeMap<[u8; 32], AuthorizedCeremony>, + /// Every ceremony ever accepted (for stale/replay probes). + ceremonies: Vec, + seen: Vec, + /// The admissible ceremony: (id, predecessor it was accepted + /// for). A restored record whose predecessor is no longer the + /// pin is contextually stale. + active: Option<(CeremonyId, Option)>, + /// (candidate digest, receipt_sent) — at most one pending + /// candidate, always a successor of the pin. + pending: Option<(CertDigest, bool)>, + /// Digests retired by a committed operation or discovered + /// erasure: their share files may exist only as owed debt. + retired: BTreeSet<[u8; 32]>, + /// Shares whose erasure is owed; empty at every operation + /// boundary in this harness (it never injects I/O failures). + debt: BTreeSet<[u8; 32]>, + frozen: bool, + } + + struct Runner<'a> { + signers: &'a Signers, + dir: &'a Path, + network: NetworkUrn, + store: Option, + model: Model, + rng: Rng, + /// Recent `state` file contents for crash replays. + snapshots: VecDeque>, + /// Set when the disk promises a receipted share it cannot serve: + /// Corrupt is the store's only honest answer and the run ends. + dead: bool, + transcript: u8, + } + + impl<'a> Runner<'a> { + fn ndir(&self) -> PathBuf { + netdir(self.dir, &self.network) + } + + fn store(&mut self) -> &mut ShareStore { + self.store.as_mut().unwrap() + } + + fn share_files(&self) -> BTreeSet<[u8; 32]> { + let mut out = BTreeSet::new(); + if let Ok(entries) = fs::read_dir(self.ndir()) { + for entry in entries.flatten() { + let Some(name) = entry.file_name().to_str().map(str::to_owned) else { + continue; + }; + if let Some(hex) = name.strip_prefix("share-") { + if let Some(digest) = decode_hex(hex) { + out.insert(digest); + } + } + } + } + out + } + + fn snapshot(&mut self) { + let state = self.ndir().join("state"); + if let Ok(bytes) = fs::read(&state) { + self.snapshots.push_back(bytes); + while self.snapshots.len() > 4 { + self.snapshots.pop_front(); + } + } + } + + /// Every rule the store must satisfy at an operation boundary — + /// called after EVERY scheduler step. Details: TC-2221 + fn assert_boundary(&self, seed: u64, step: usize) { + let m = &self.model; + let Some(store) = self.store.as_ref() else { + assert!( + self.dead, + "seed {seed} step {step}: closed store without Corrupt" + ); + return; + }; + let files = self.share_files(); + // Rule 4: once frozen, frozen forever — the recorded pin and + // candidate stay visible for the operator's audit, but no + // share material may leave the store. + if m.frozen { + assert!( + store.equivocation(), + "seed {seed} step {step}: frozen store thawed" + ); + assert!( + matches!(store.current_share(), Err(StoreError::Equivocation)), + "seed {seed} step {step}: frozen store exports a share" + ); + for digest in &files { + assert!( + m.signed.contains_key(digest), + "seed {seed} step {step}: foreign share file on frozen store" + ); + } + return; + } + // Rule 3 / rule 1: exactly one pin, and only the pin's share + // is ever usable. + let pin = m.pins.last().map(|p| p.digest()); + assert_eq!( + store.pinned().map(|p| p.digest()), + pin, + "seed {seed} step {step}: pin diverged from model" + ); + // At most one pending candidate, always the pin's successor. + assert_eq!( + store.pending_candidate().map(|(d, _)| d), + m.pending.map(|(d, _)| d), + "seed {seed} step {step}: pending candidate diverged" + ); + // Rule 2: no live share is lost; rule 7: nothing retired is + // live except as recorded debt — the on-disk share set is + // exactly {pin?} ∪ {pending?} ∪ debt. + let mut live: BTreeSet<[u8; 32]> = BTreeSet::new(); + if let Some(d) = pin { + live.insert(*d.as_bytes()); + } + if let Some((d, _)) = m.pending { + live.insert(*d.as_bytes()); + } + for digest in &live { + assert!( + files.contains(digest), + "seed {seed} step {step}: live share missing" + ); + } + for digest in &files { + assert!( + live.contains(digest) || m.debt.contains(digest), + "seed {seed} step {step}: share file outside live set and debt" + ); + } + assert!( + m.debt.is_subset(&files), + "seed {seed} step {step}: erasure debt dropped" + ); + // Rule 7 half: a retired share may exist only as debt. + for digest in &files { + if m.retired.contains(digest) { + assert!( + m.debt.contains(digest), + "seed {seed} step {step}: retired share live without debt" + ); + } + } + // Rule 6 + usable share: with a pin file on disk the bound + // share is exported. + if let Some(d) = pin { + if files.contains(d.as_bytes()) { + assert!( + matches!(store.current_share(), Ok(Some(_))), + "seed {seed} step {step}: pinned share not usable" + ); + } + } + // Rule 8: no temp files survive a boundary. + if let Ok(entries) = fs::read_dir(self.ndir()) { + for entry in entries.flatten() { + let name = entry.file_name(); + let name = name.to_str().unwrap_or(""); + assert!( + !name.ends_with(".tmp") && !name.contains(".tmp-"), + "seed {seed} step {step}: leftover temp {name}" + ); + } + } + } + + /// Fresh ceremony for the current pin with a never-used id. + fn fresh_ceremony(&mut self) -> AuthorizedCeremony { + let pin = self.model.pins.last().unwrap(); + let mut stmt = self + .signers + .body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + self.transcript = self.transcript.wrapping_add(1).max(64); + let mut id_bytes = [0x60u8; 32]; + id_bytes[0] = 0x60 ^ (self.transcript); + id_bytes[31] = self.transcript; + stmt.id = CeremonyId::new(id_bytes); + stmt.current = Some(pin.digest()); + stmt.epoch = pin.body().epoch().checked_next().unwrap(); + stmt.rekey = pin.body().rekey(); + AuthorizedCeremony::verify_for(pin, &self.signers.statement(&stmt), self.signers) + .unwrap() + } + + /// A signed successor certificate of the pin under `ceremony`. + fn fresh_signed( + &mut self, + ceremony: &AuthorizedCeremony, + ) -> (SignedCertificate, KeyShare) { + self.transcript = self.transcript.wrapping_add(1).max(64); + let pin = self.model.pins.last().unwrap(); + let (public, mut shares) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); + let body = match CertificateBody::successor( + pin, + ceremony, + &public, + &TranscriptHash::new([self.transcript; 32]), + ) { + Ok(body) => body, + Err(e) => panic!( + "fresh_signed: {e:?} pin={} cer_current={:?} cer_epoch={:?} pin_epoch={:?}", + hex(pin.digest().as_bytes()), + ceremony.statement().current.map(|d| hex(d.as_bytes())), + ceremony.statement().epoch, + pin.body().epoch(), + ), + }; + (cert::tests::sign(body, self.signers), shares.remove(0)) + } + + fn accept(&mut self, seed: u64, step: usize) -> Option { + let ceremony = self.fresh_ceremony(); + let outcome = self.store().accept_ceremony(&ceremony); + match outcome { + Ok(AcceptOutcome::Fresh) => {} + other => panic!("seed {seed} step {step}: fresh accept gave {other:?}"), + } + // C1: a fresh accept erases any unactivated candidate for + // the same predecessor. + if let Some((digest, _)) = self.model.pending.take() { + self.model.retired.insert(*digest.as_bytes()); + } + let id = ceremony.statement().id; + self.model.seen.push(id); + self.model.active = Some((id, ceremony.statement().current)); + self.model.ceremonies.push(ceremony.clone()); + let pred = self.model.pins.last().unwrap().digest(); + self.model + .ceremonies_by_pred + .insert(*pred.as_bytes(), ceremony.clone()); + Some(ceremony) + } + + fn install(&mut self, seed: u64, step: usize, ceremony: &AuthorizedCeremony) { + let (signed, raw) = self.fresh_signed(ceremony); + let candidate = mk_candidate(raw, &signed.body); + let signers = self.signers; + let digest = self + .store() + .install_candidate(&signed, candidate, ceremony, signers) + .unwrap_or_else(|e| panic!("seed {seed} step {step}: install failed: {e:?}")); + self.model.signed.insert(*digest.as_bytes(), signed); + assert!( + self.model.pending.is_none(), + "seed {seed} step {step}: model lost pending" + ); + self.model.pending = Some((digest, false)); + } + + /// Rule 9: after any crash the reopened store is consistent with + /// some crash-free projection of history — resync the model to + /// what the store actually exposes (truncating the pin chain and + /// forgetting the admissible id if the state reverted). + fn resync(&mut self, seed: u64, step: usize) { + let m = &mut self.model; + let store = self.store.as_ref().unwrap(); + m.frozen = store.equivocation(); + if m.frozen { + m.active = None; + m.pending = None; + return; + } + match store.pinned() { + Some(pin) => { + let pos = m + .pins + .iter() + .position(|p| p.digest() == pin.digest()) + .unwrap_or_else(|| { + panic!("seed {seed} step {step}: reopen pinned unknown certificate") + }); + m.pins.truncate(pos + 1); + } + None => m.pins.clear(), + } + m.pending = store.pending_candidate().map(|(d, _)| (d, false)); + // A reopened candidate must be the pin's successor — open's + // own convergence rule, asserted here too. + if let (Some((d, _)), Some(pin)) = (m.pending, m.pins.last()) { + let signed = &m.signed[d.as_bytes()]; + assert_eq!( + signed.body.previous(), + Some(&pin.digest()), + "seed {seed} step {step}: reopened candidate is not the pin's successor" + ); + } + // The admissible ceremony is persisted; resync reads what a + // crash-revert actually restored — an id whose recorded + // predecessor is again the pin may install, any other is + // contextually stale and must be refused. + m.active = store + .inner + .lock() + .unwrap() + .state + .active_ceremony + .as_ref() + .map(|a| (a.id, a.predecessor)); + } + + fn reopen(&mut self, seed: u64, step: usize) { + self.store = None; + match ShareStore::open( + self.dir, + &self.network, + self.signers.anchor().clone(), + Signers::did(1), + self.signers, + ) { + Ok(store) => { + self.store = Some(store); + self.resync(seed, step); + } + Err(e) => { + // The only lawful refusal: the recorded state + // promises a receipted share whose file is gone — + // real disk damage the store must not guess past. + let state = fs::read(self.ndir().join("state")) + .ok() + .and_then(|b| decode_state(&b).ok()); + let promised_missing = state.is_some_and(|s| { + s.candidate.is_some_and(|c| { + c.receipt_sent && !share_path(&self.ndir(), &c.digest).exists() + }) + }); + assert!( + matches!(e, StoreError::Corrupt) && promised_missing, + "seed {seed} step {step}: reopen failed: {e:?}" + ); + self.dead = true; + } + } + } + } + + /// Encode `state` in the pre-release version-1 layout (everything + /// except the `active_ceremony` option) to probe legacy decoding. + fn encode_state_v1(state: &State) -> Vec { + let mut e = Encoder::new(); + e.u8(1); + e.u8(u8::from(state.pin.is_some())); + if let Some(pin) = state.pin { + e.fixed(pin.as_bytes()); + } + e.u8(u8::from(state.candidate.is_some())); + if let Some(candidate) = &state.candidate { + e.fixed(candidate.digest.as_bytes()); + e.fixed(candidate.ceremony.as_bytes()); + e.u8(u8::from(candidate.predecessor.is_some())); + if let Some(predecessor) = candidate.predecessor { + e.fixed(predecessor.as_bytes()); + } + e.u8(u8::from(candidate.receipt_sent)); + } + e.u32(state.seen_ceremonies.len() as u32); + for id in &state.seen_ceremonies { + e.fixed(id.as_bytes()); + } + e.u32(state.retired.len() as u32); + for retired in &state.retired { + e.fixed(retired.digest.as_bytes()); + e.u64(retired.erased_at); + } + e.u64(state.seq); + e.finish() + } + + /// One seeded run: genesis activation, then `steps` weighted random + /// operations with boundary assertions after every single step. + fn run_seed(dir: &Path, signers: &Signers, seed: u64, steps: usize) { + let network = signers.anchor().network.clone(); + let mut runner = Runner { + signers, + dir, + network: network.clone(), + store: Some(open(dir, signers)), + model: Model::default(), + rng: Rng(seed), + snapshots: VecDeque::new(), + transcript: 63, + dead: false, + }; + let (signed, cert) = activated_genesis(runner.store.as_ref().unwrap(), signers); + runner.model.pins.push(cert); + runner + .model + .signed + .insert(*signed.body.digest().as_bytes(), signed); + let genesis_ceremony = signers.genesis(); + runner.model.seen.push(genesis_ceremony.statement().id); + runner.model.ceremonies.push(genesis_ceremony); + runner.snapshot(); + + for step in 0..steps { + if runner.dead { + break; + } + let frozen = runner.model.frozen; + let pending = runner.model.pending; + let roll = runner.rng.pick(100); + match roll { + // Fresh accept + install under the admissible id. + 0..=21 if !frozen => { + if let Some(ceremony) = runner.accept(seed, step) { + runner.install(seed, step, &ceremony); + } + } + // Accept only (candidate window stays empty). + 22..=29 if !frozen => { + runner.accept(seed, step); + } + // Replay or stale accept: a seen id whose context still + // matches the pin replays (informational — the stale + // install op proves it can never install); a contextually + // stale id is refused outright. + 30..=37 if !frozen => { + let n = runner.model.ceremonies.len(); + if n > 0 { + let ceremony = + runner.model.ceremonies[runner.rng.pick(n as u64) as usize].clone(); + let pin = runner.model.pins.last().unwrap().digest(); + let expect_replay = ceremony.statement().current == Some(pin); + match runner.store().accept_ceremony(&ceremony) { + Ok(AcceptOutcome::Replay) if expect_replay => {} + Err(e) if !expect_replay && matches!(e, StoreError::Crypto(_)) => {} + other => panic!( + "seed {seed} step {step}: stale/replay accept gave {other:?}" + ), + } + } + } + // Stale install: any non-admissible ceremony id — + // including retired ones — must fail. + 38..=43 if !frozen => { + let stale: Vec = runner + .model + .ceremonies + .iter() + .filter(|c| { + runner.model.active.map(|(id, _)| id) != Some(c.statement().id) + }) + .cloned() + .collect(); + if !stale.is_empty() { + let ceremony = + stale[runner.rng.pick(stale.len() as u64) as usize].clone(); + // The certificate only has to be a valid pin + // successor; the stale ceremony argument is what + // the admissible-id check refuses first. + let construction = runner.fresh_ceremony(); + let (signed, raw) = runner.fresh_signed(&construction); + let candidate = mk_candidate(raw, &signed.body); + assert!( + runner + .store() + .install_candidate(&signed, candidate, &ceremony, signers) + .is_err(), + "seed {seed} step {step}: stale ceremony id installed" + ); + } + } + // Install under the admissible id when none is pending. + // The restored record binds the pin it was accepted for: + // if a crash revert moved the pin past it the id is + // contextually stale and must be refused, never built. + 44..=51 if !frozen && pending.is_none() => { + if let Some((id, pred)) = runner.model.active { + let pin = runner.model.pins.last().unwrap().digest(); + let ceremony = runner + .model + .ceremonies + .iter() + .find(|c| c.statement().id == id) + .unwrap() + .clone(); + if pred == Some(pin) { + runner.install(seed, step, &ceremony); + } else { + let construction = runner.fresh_ceremony(); + let (signed, raw) = runner.fresh_signed(&construction); + let candidate = mk_candidate(raw, &signed.body); + assert!( + runner + .store() + .install_candidate(&signed, candidate, &ceremony, signers) + .is_err(), + "seed {seed} step {step}: pin-stale ceremony id installed" + ); + } + } + } + // Rule 6: receipt only over a durable share. + 52..=59 if !frozen && pending.is_some() => { + runner + .store() + .mark_receipt_sent() + .unwrap_or_else(|e| panic!("seed {seed} step {step}: receipt: {e:?}")); + runner.model.pending = pending.map(|(d, _)| (d, true)); + } + // Verified activation: the only legal pin advance. + 60..=73 if !frozen && pending.is_some() => { + let (digest, _) = pending.unwrap(); + let signed = runner.model.signed[digest.as_bytes()].clone(); + let cert = + verify_successor(runner.model.pins.last().unwrap(), &signed, signers) + .unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + runner + .store() + .record_activation(&signed, &activation, &receipts, signers) + .unwrap_or_else(|e| { + panic!("seed {seed} step {step}: activation: {e:?}") + }); + if let Some(old) = runner.model.pins.last() { + runner.model.retired.insert(*old.digest().as_bytes()); + } + runner.model.pins.push(cert); + runner.model.pending = None; + runner.model.active = None; + } + // Write an activated sibling proof for an interior + // predecessor: discovered at next open → frozen. + 74..=79 if !frozen && runner.model.pins.len() >= 2 => { + let i = 1 + runner.rng.pick((runner.model.pins.len() - 1) as u64) as usize; + let pred = runner.model.pins[i - 1].clone(); + let ceremony = + runner.model.ceremonies_by_pred[pred.digest().as_bytes()].clone(); + runner.transcript = runner.transcript.wrapping_add(1); + let signed = + sibling_successor(signers, &pred, &ceremony, runner.transcript); + let cert = verify_successor(&pred, &signed, signers).unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + let path = proof_path(&runner.ndir(), &cert.digest()); + fs::write(&path, encode_proof(&signed, Some(&activation), &receipts)) + .unwrap(); + runner + .model + .signed + .insert(*cert.digest().as_bytes(), signed); + runner.reopen(seed, step); + assert!( + runner.model.frozen, + "seed {seed} step {step}: activated fork did not freeze" + ); + } + // Rule 10: a second opener changes nothing. + 80..=85 => { + assert!(matches!( + ShareStore::open( + dir, + &network, + signers.anchor().clone(), + Signers::did(1), + signers + ), + Err(StoreError::Busy) + )); + } + // Clean reopen. + 86..=90 => runner.reopen(seed, step), + // Crash replays: revert `state` to a recent snapshot + // (commit-without-completion), delete a dealt marker + // (post-commit pre-marker crash), or leave temp debris; + // every variant ends in a reopen + resync. + _ => { + match runner.rng.pick(4) { + 0 if !runner.snapshots.is_empty() => { + runner.store = None; + let i = runner.rng.pick(runner.snapshots.len() as u64) as usize; + let bytes = runner.snapshots[i].clone(); + fs::write(runner.ndir().join("state"), &bytes).unwrap(); + } + 1 => { + let n = runner.model.seen.len(); + if n > 0 { + let id = runner.model.seen[runner.rng.pick(n as u64) as usize]; + let marker = + runner.ndir().join("dealt").join(hex(id.as_bytes())); + let _ = fs::remove_file(marker); + } + } + 2 => { + runner.store = None; + let debris = runner + .ndir() + .join(format!("share-{}.tmp-1-9", hex(&[0xAB; 32]))); + fs::write(&debris, [0x55u8; 67]).unwrap(); + } + _ => runner.store = None, + } + runner.reopen(seed, step); + } + } + runner.assert_boundary(seed, step); + runner.snapshot(); + } + eprintln!("model seed {seed}: {steps} steps ok"); + } + + /// Legacy `state` files must be handled explicitly: anything but the + /// current version is refused at decode, never silently upgraded. + /// Details: TC-2221 + #[test] + fn legacy_state_versions_are_refused_explicitly() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + { + let store = open(&dir, &signers); + activated_genesis(&store, &signers); + let state = store.inner.lock().unwrap().state.clone(); + drop(store); + // Version 1 (pre-release, no admissible-ceremony slot). + fs::write( + netdir(&dir, &network).join("state"), + encode_state_v1(&state), + ) + .unwrap(); + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); + // Unknown future version. + let mut future = encode_state(&state); + future[0] = 3; + fs::write(netdir(&dir, &network).join("state"), &future).unwrap(); + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); + } + } + + #[test] + fn seeded_model_invariants_hold_across_crashes() { + // Debug builds pay full fsync + verify cost per step, so CI runs + // a bounded slice; the release suite sweeps the wider seed space. + // Replay any failure with the printed seed. + let seeds: u64 = if cfg!(debug_assertions) { 4 } else { 64 }; + for seed in 0..seeds { + let signers = Signers::new(); + let dir = tmpdir(); + run_seed(&dir, &signers, seed, 220); + } + } + } } From 94733ae6c91316927bf02297663fbc395ece030d Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 10:19:33 +0000 Subject: [PATCH 5/8] fix(threshold): frozen reopen keeps recorded pin, retired digests refuse install and activation, state mutates after write; Details: TC-2221 --- tinycloud-threshold/src/store.rs | 1052 ++++++++++++++++++++++++------ 1 file changed, 861 insertions(+), 191 deletions(-) diff --git a/tinycloud-threshold/src/store.rs b/tinycloud-threshold/src/store.rs index fea5c021..25ca3396 100644 --- a/tinycloud-threshold/src/store.rs +++ b/tinycloud-threshold/src/store.rs @@ -317,6 +317,10 @@ pub enum StoreError { /// Another `ShareStore` already holds this network's lock. #[error("share store is already open for this network")] Busy, + /// The certificate belongs to a retired generation; its share was erased + /// under durable proof and must never come back. + #[error("certificate belongs to a retired generation")] + Retired, /// The filesystem cannot provide the exclusive open lock. #[error("share store lock unsupported on this filesystem")] LockUnsupported, @@ -390,10 +394,25 @@ fn write_atomic(path: &Path, bytes: &[u8]) -> Result<(), StoreError> { } fn erase_share_file(path: &Path) -> Result<(), StoreError> { - if !path.exists() { + // Never follow links or traverse directories: a store path that is not a + // regular file is removed as a bare name, never written through — a + // `*.tmp-*` symlink can otherwise make `open` zero a victim file + // outside the store. + let Ok(meta) = fs::symlink_metadata(path) else { + return Ok(()); + }; + if !meta.is_file() { + if meta.is_dir() { + fs::remove_dir(path)?; + } else { + fs::remove_file(path)?; + } + if let Some(parent) = path.parent() { + fs::File::open(parent)?.sync_all()?; + } return Ok(()); } - let length = usize::try_from(fs::metadata(path)?.len()).unwrap_or(usize::MAX); + let length = usize::try_from(meta.len()).unwrap_or(usize::MAX); { use std::io::Write; let mut file = fs::OpenOptions::new().write(true).open(path)?; @@ -502,6 +521,11 @@ impl ShareStore { v: &impl DidSignatureVerifier, ) -> Result { let network_dir = store_dir(dir, network); + // Fsync only what this call created — network_dir, proofs, dealt, + // and the first pre-existing parent that must record them. Ancestors + // above that belong to the caller; an execute-only one must not be + // required readable. + let existed = network_dir.exists(); fs::create_dir_all(network_dir.join("proofs"))?; fs::create_dir_all(network_dir.join("dealt"))?; #[cfg(unix)] @@ -509,13 +533,13 @@ impl ShareStore { use std::os::unix::fs::PermissionsExt; fs::set_permissions(&network_dir, fs::Permissions::from_mode(0o700))?; } - // Fsync every directory in the created hierarchy — including the - // parents of the caller's base directory — so the tree itself is - // durable even if power is lost before the first write completes. - let mut ancestor = Some(network_dir.as_path()); - while let Some(dir_path) = ancestor { - fs::File::open(dir_path)?.sync_all()?; - ancestor = dir_path.parent(); + fs::File::open(&network_dir)?.sync_all()?; + fs::File::open(network_dir.join("proofs"))?.sync_all()?; + fs::File::open(network_dir.join("dealt"))?.sync_all()?; + if !existed { + if let Some(parent) = network_dir.parent() { + fs::File::open(parent)?.sync_all()?; + } } // One opener per network directory, held for the store's lifetime. @@ -699,109 +723,132 @@ impl ShareStore { } } - // Candidate reconciliation. Chain membership and C2 losses are - // judged BEFORE the missing-file check: a candidate already retired - // by durable proof is settled even when its share is gone, while a - // still-live receipt-bearing candidate that lost its share is the - // only case that is truly corrupt. - let mut owed_erasures: Vec = Vec::new(); - if let Some(candidate) = state.candidate.clone() { - let share = share_path(&network_dir, &candidate.digest); - let proof = proof_path(&network_dir, &candidate.digest); - if chain.contains(&candidate.digest) { - // Already activated before the crash; the chain is the record. - state.candidate = None; - state_dirty = true; - } else { - // C2: the candidate lost if its predecessor is covered by the - // pin (any pin for a genesis predecessor) or the chain moved - // past it through another certificate. - let lost = match candidate.predecessor { - None => computed_pin.is_some(), - Some(p) => chain - .iter() - .position(|d| *d == p) - .map_or(computed_pin.is_some(), |i| { - computed_pin != Some(p) || i.saturating_add(1) < chain.len() - }), - }; - if lost { - // Commit the retirement first; the unlink is owed work a - // crash simply re-discovers as S1. - record_retired(&mut state, candidate.digest); + // Candidate reconciliation and R1/C2/S1 erasure run ONLY on an + // unequivocal chain. Once activated equivocation is on disk the + // store freezes in place: the recorded verified pin stands, every + // share file is kept (losing nothing is the whole point of the + // freeze), and only the operator's recovery path rewrites it. + if !equivocation { + // Candidate reconciliation. Chain membership and C2 losses are + // judged BEFORE the missing-file check: a candidate already + // retired by durable proof is settled even when its share is + // gone, while a still-live receipt-bearing candidate that lost + // its share is the only case that is truly corrupt. + let mut owed_erasures: Vec = Vec::new(); + if let Some(candidate) = state.candidate.clone() { + let share = share_path(&network_dir, &candidate.digest); + let proof = proof_path(&network_dir, &candidate.digest); + if chain.contains(&candidate.digest) { + // Already activated before the crash; the chain is the + // record. state.candidate = None; - state.pin = computed_pin; - write_atomic(&network_dir.join("state"), &encode_state(&state))?; - owed_erasures.push(share); - } else if !share.exists() || !proof.exists() { - if candidate.receipt_sent { - // A receipt was signed over a durable install; a - // missing file means the member promised what it - // cannot serve. - return Err(StoreError::Corrupt); + state_dirty = true; + } else { + // C2: the candidate lost if its predecessor is covered + // by the pin (any pin for a genesis predecessor) or the + // chain moved past it through another certificate. + let lost = match candidate.predecessor { + None => computed_pin.is_some(), + Some(p) => chain + .iter() + .position(|d| *d == p) + .map_or(computed_pin.is_some(), |i| { + computed_pin != Some(p) || i.saturating_add(1) < chain.len() + }), + }; + if lost { + // Commit the retirement first; the unlink is owed + // work a crash simply re-discovers as S1. + record_retired(&mut state, candidate.digest); + state.candidate = None; + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + owed_erasures.push(share); + } else if !share.exists() || !proof.exists() { + if candidate.receipt_sent { + // A receipt was signed over a durable install; + // a missing file means the member promised what + // it cannot serve. + return Err(StoreError::Corrupt); + } + state.candidate = None; + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + owed_erasures.push(share); } - state.candidate = None; - state.pin = computed_pin; - write_atomic(&network_dir.join("state"), &encode_state(&state))?; - owed_erasures.push(share); } } - } - // R1 and S1 over share files. Every retirement is committed to - // `state` before its unlink; files left by a crash re-enter here as - // owed work on the next open. - for entry in fs::read_dir(&network_dir)? { - let entry = entry?; - let name = entry.file_name(); - let Some(name) = name.to_str() else { continue }; - let Some(hex_digest) = name.strip_prefix("share-") else { - continue; - }; - let Some(digest_bytes) = decode_hex(hex_digest) else { - // An unnamed partial write (including .tmp leftovers) can - // never have been recorded; S1 erases it. - erase_share_file(&entry.path())?; - continue; - }; - let digest = CertDigest::new(digest_bytes); - let is_pin = computed_pin == Some(digest); - let is_candidate = state.candidate.as_ref().is_some_and(|c| c.digest == digest); - if is_pin || is_candidate { - continue; - } - if verified.contains_key(&digest) { - // Superseded activated generation (R1). - if !state.retired.iter().any(|r| r.digest == digest) { - record_retired(&mut state, digest); - state.pin = computed_pin; - write_atomic(&network_dir.join("state"), &encode_state(&state))?; - state_dirty = false; + // R1 and S1 over share files. Every retirement is committed to + // `state` before its unlink; files left by a crash re-enter here + // as owed work on the next open. + for entry in fs::read_dir(&network_dir)? { + let entry = entry?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + let Some(hex_digest) = name.strip_prefix("share-") else { + continue; + }; + let Some(digest_bytes) = decode_hex(hex_digest) else { + // An unnamed partial write (including .tmp leftovers) + // can never have been recorded; S1 erases it. + erase_share_file(&entry.path())?; + continue; + }; + let digest = CertDigest::new(digest_bytes); + let is_pin = computed_pin == Some(digest); + let is_candidate = state.candidate.as_ref().is_some_and(|c| c.digest == digest); + if is_pin || is_candidate { + continue; + } + if verified.contains_key(&digest) { + // Superseded activated generation (R1). + if !state.retired.iter().any(|r| r.digest == digest) { + record_retired(&mut state, digest); + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + state_dirty = false; + } + owed_erasures.push(entry.path()); + } else { + // Outside the activated chain and named in no record: + // S1 erases unconditionally (the pin's existence changes + // nothing — such a file never had a durable install + // record). + erase_share_file(&entry.path())?; } - owed_erasures.push(entry.path()); - } else { - // Outside the activated chain and named in no record: S1 - // erases unconditionally (the pin's existence changes - // nothing — such a file never had a durable install record). - erase_share_file(&entry.path())?; } - } - if state.pin != computed_pin || state_dirty { - state.pin = computed_pin; - write_atomic(&network_dir.join("state"), &encode_state(&state))?; - } + if state.pin != computed_pin || state_dirty { + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + } - // Owed unlinks run only after every corresponding `state` commit. - for path in &owed_erasures { - erase_share_file(path)?; - } + // Owed unlinks run only after every corresponding `state` + // commit. + for path in &owed_erasures { + erase_share_file(path)?; + } - if state.pin != computed_pin || state_dirty { - state.pin = computed_pin; - write_atomic(&network_dir.join("state"), &encode_state(&state))?; + if state.pin != computed_pin || state_dirty { + state.pin = computed_pin; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + } } + // The in-memory pin: normally the computed chain tip; on + // equivocation the recorded verified pin (a branch descendant is + // proven, not corruption), so a reopen is the same freeze the live + // store hit. + let pin = if equivocation { + state + .pin + .and_then(|d| verified.get(&d).cloned()) + .or_else(|| computed_pin.and_then(|d| verified.get(&d).cloned())) + } else { + pin + }; + Ok(Self { network_dir, anchor, @@ -921,6 +968,13 @@ impl ShareStore { return Err(StoreError::InvalidActivation); } let digest = verified.digest(); + // The certificate body carries no ceremony id, so a superseded + // ceremony's certificate could arrive under the fresh admissible + // label. The durable retirement record — not the label — decides: + // an erased share never resurrects. + if inner.state.retired.iter().any(|r| r.digest == digest) { + return Err(StoreError::Retired); + } let index = candidate.index(); write_atomic( &proof_path(&self.network_dir, &digest), @@ -938,13 +992,15 @@ impl ShareStore { return Err(StoreError::Corrupt); } let _bound = KeyShare::bind(reloaded, &verified)?; - inner.state.candidate = Some(CandidateRecord { + let mut next = inner.state.clone(); + next.candidate = Some(CandidateRecord { digest, ceremony: ceremony.statement().id, predecessor, receipt_sent: false, }); - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; + inner.state = next; Ok(digest) } @@ -954,11 +1010,14 @@ impl ShareStore { /// `receipt_sent` with a missing share file refuses to open. pub fn mark_receipt_sent(&self) -> Result<(), StoreError> { let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; - let Some(candidate) = &mut inner.state.candidate else { + let mut next = inner.state.clone(); + let Some(candidate) = &mut next.candidate else { return Err(StoreError::Corrupt); }; candidate.receipt_sent = true; - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state)) + write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; + inner.state = next; + Ok(()) } /// Record a verified activation; promotes the candidate when it matches @@ -1008,42 +1067,47 @@ impl ShareStore { }; verify_activation(&verified, receipts, ReadyQuorum::AllHolders, activation, v)?; let digest = verified.digest(); + // The durable retirement record rules out re-activation too: a + // losing candidate whose share was erased must never be re-pinned + // by a late `record_activation` — the erased share cannot serve. + if inner.state.retired.iter().any(|r| r.digest == digest) { + return Err(StoreError::Retired); + } write_atomic( &proof_path(&self.network_dir, &digest), &encode_proof(signed, Some(activation), receipts), )?; // `state` commits every removal BEFORE the unlink, so a crash is - // recovered as "erasure owed", never as a missing promised share. + // recovered as "erasure owed", never as a missing promised share; + // the in-memory record moves only after its durable copy exists. + let mut next = inner.state.clone(); let mut erasures: Vec = Vec::new(); // C2: a pending candidate that lost to this activation is erased; the // winning candidate's record is cleared (the chain is its record). - if let Some(candidate) = inner.state.candidate.take() { + if let Some(candidate) = next.candidate.take() { if candidate.digest != digest - && !inner - .state - .retired - .iter() - .any(|r| r.digest == candidate.digest) + && !next.retired.iter().any(|r| r.digest == candidate.digest) { - record_retired(&mut inner.state, candidate.digest); + record_retired(&mut next, candidate.digest); erasures.push(share_path(&self.network_dir, &candidate.digest)); } } // R1: the outgoing pin's share is retired. if let Some(old_pin) = &inner.pin { let old = old_pin.digest(); - if !inner.state.retired.iter().any(|r| r.digest == old) { - record_retired(&mut inner.state, old); + if !next.retired.iter().any(|r| r.digest == old) { + record_retired(&mut next, old); erasures.push(share_path(&self.network_dir, &old)); } } - inner.pin = Some(verified.clone()); - inner.verified.insert(digest, verified); - inner.state.pin = Some(digest); + next.pin = Some(digest); // Ceremonies accepted against the old pin are stale: the next // admissible ceremony must be accepted afresh for this pin. - inner.state.active_ceremony = None; - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + next.active_ceremony = None; + write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; + inner.state = next; + inner.pin = Some(verified.clone()); + inner.verified.insert(digest, verified); for path in &erasures { erase_share_file(path)?; } @@ -1080,22 +1144,27 @@ impl ShareStore { } // `state` commits the removal and the new admissible ceremony before // the unlink and before the dealt marker; a crash re-discovers the - // unlink as S1 and the marker's id as consumed. + // unlink as S1 and the marker's id as consumed. The in-memory record + // is touched only after its durable copy exists, so a failed write + // leaves the live state identical to disk — a retry sees the same + // ceremony as fresh, not a phantom replay. + let mut next = inner.state.clone(); let mut owed: Option = None; - if let Some(candidate) = &inner.state.candidate { + if let Some(candidate) = &next.candidate { if candidate.predecessor == statement.current { let digest = candidate.digest; - record_retired(&mut inner.state, digest); - inner.state.candidate = None; + record_retired(&mut next, digest); + next.candidate = None; owed = Some(share_path(&self.network_dir, &digest)); } } - inner.state.seen_ceremonies.push(statement.id); - inner.state.active_ceremony = Some(ActiveCeremony { + next.seen_ceremonies.push(statement.id); + next.active_ceremony = Some(ActiveCeremony { id: statement.id, predecessor: statement.current, }); - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; + inner.state = next; let marker = write_atomic( &self .network_dir @@ -1134,12 +1203,29 @@ mod tests { static TAG: AtomicU64 = AtomicU64::new(0); - fn tmpdir() -> PathBuf { + /// A temporary store directory removed on drop — tests leave nothing + /// under `/tmp` behind. + struct Tmp(PathBuf); + + impl std::ops::Deref for Tmp { + type Target = Path; + fn deref(&self) -> &Path { + &self.0 + } + } + + impl Drop for Tmp { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.0); + } + } + + fn tmpdir() -> Tmp { let tag = TAG.fetch_add(1, Ordering::Relaxed); let path = std::env::temp_dir().join(format!("tc2221-store-{}-{tag}", std::process::id())); let _ = fs::remove_dir_all(&path); fs::create_dir_all(&path).unwrap(); - path + Tmp(path) } fn netdir(dir: &Path, network: &NetworkUrn) -> PathBuf { @@ -2203,11 +2289,13 @@ mod tests { let store = open(&dir, &signers); activated_genesis(&store, &signers); } - // A directory where a temp file is expected cannot be overwritten or - // unlinked as a file — the sweep must refuse the open. + // A non-empty directory where a temp file is expected cannot be + // overwritten, unlinked as a file, or removed as an empty dir — + // the sweep must refuse the open. let ndir = netdir(&dir, &network); let stuck = ndir.join("state.tmp-1-1"); fs::create_dir(&stuck).unwrap(); + fs::write(stuck.join("content"), [0x55u8; 4]).unwrap(); assert!(ShareStore::open( &dir, &network, @@ -2216,7 +2304,7 @@ mod tests { &signers ) .is_err()); - fs::remove_dir(&stuck).unwrap(); + fs::remove_dir_all(&stuck).unwrap(); let _reopened = ShareStore::open( &dir, &network, @@ -2282,6 +2370,78 @@ mod tests { /// boundary in this harness (it never injects I/O failures). debt: BTreeSet<[u8; 32]>, frozen: bool, + /// The pin and share-file set recorded at the moment the store + /// froze: the freeze is terminal, so neither may ever move. + frozen_pin: Option, + frozen_files: Option>, + } + + /// The model's own parser for the on-disk v2 `state` record — the + /// observable contract a crash leaves behind, independent of the + /// store's internals. Returns `None` for any unreadable or + /// unparsable file. + struct ParsedState { + pin: Option, + candidate: Option<(CertDigest, bool)>, + active: Option<(CeremonyId, Option)>, + retired: BTreeSet<[u8; 32]>, + } + + fn parse_state(bytes: &[u8]) -> Option { + let mut cur = 0usize; + let take = |n: usize, cur: &mut usize| -> Option<&[u8]> { + let end = cur.checked_add(n)?; + let slice = bytes.get(*cur..end)?; + *cur = end; + Some(slice) + }; + if take(1, &mut cur)? != [2] { + return None; + } + let opt32 = |cur: &mut usize| -> Option> { + match take(1, cur)? { + [0] => Some(None), + [1] => take(32, cur).map(|s| Some(s.try_into().unwrap())), + _ => None, + } + }; + let pin = opt32(&mut cur)?.map(CertDigest::new); + let candidate = if take(1, &mut cur)? == [1] { + let digest = CertDigest::new(take(32, &mut cur)?.try_into().unwrap()); + take(32, &mut cur)?; // ceremony id + opt32(&mut cur)?; // predecessor + let receipt = take(1, &mut cur)? == [1]; + Some((digest, receipt)) + } else { + None + }; + let active = if take(1, &mut cur)? == [1] { + let id = CeremonyId::new(take(32, &mut cur)?.try_into().unwrap()); + let pred = opt32(&mut cur)?.map(CertDigest::new); + Some((id, pred)) + } else { + None + }; + let seen_len = u32::from_be_bytes(take(4, &mut cur)?.try_into().unwrap()) as usize; + for _ in 0..seen_len { + take(32, &mut cur)?; + } + let retired_len = u32::from_be_bytes(take(4, &mut cur)?.try_into().unwrap()) as usize; + let mut retired = BTreeSet::new(); + for _ in 0..retired_len { + retired.insert(take(32, &mut cur)?.try_into().unwrap()); + take(8, &mut cur)?; // erased_at + } + take(8, &mut cur)?; // seq + if cur != bytes.len() { + return None; + } + Some(ParsedState { + pin, + candidate, + active, + retired, + }) } struct Runner<'a> { @@ -2296,6 +2456,9 @@ mod tests { /// Set when the disk promises a receipted share it cannot serve: /// Corrupt is the store's only honest answer and the run ends. dead: bool, + /// Whether this seed may inject fork evidence; a quarter of + /// seeds never fork so most runs exercise long unfrozen chains. + fork_allowed: bool, transcript: u8, } @@ -2308,6 +2471,36 @@ mod tests { self.store.as_mut().unwrap() } + /// A cheap digest of every regular file under the network dir — + /// used to prove a refused second opener touches nothing. + fn disk_hash(&self) -> u64 { + let mut hash = 0xcbf29ce484222325u64; + for sub in [ + self.ndir(), + self.ndir().join("proofs"), + self.ndir().join("dealt"), + ] { + let mut entries: Vec<(String, Vec)> = fs::read_dir(&sub) + .map(|rd| { + rd.flatten() + .filter_map(|e| { + let name = e.file_name().to_str()?.to_owned(); + let bytes = fs::read(e.path()).ok()?; + Some((name, bytes)) + }) + .collect() + }) + .unwrap_or_default(); + entries.sort(); + for (name, bytes) in entries { + for b in name.bytes().chain(bytes) { + hash = (hash ^ u64::from(b)).wrapping_mul(0x100000001b3); + } + } + } + hash + } + fn share_files(&self) -> BTreeSet<[u8; 32]> { let mut out = BTreeSet::new(); if let Ok(entries) = fs::read_dir(self.ndir()) { @@ -2347,9 +2540,45 @@ mod tests { return; }; let files = self.share_files(); + // Rule 11: the state file is always exactly one v2 record. + let parsed = fs::read(self.ndir().join("state")) + .ok() + .and_then(|b| parse_state(&b)); + assert!( + parsed.is_some(), + "seed {seed} step {step}: state file does not parse as v2" + ); + // Rule 12: the model and the parsed state never diverge — + // pin, candidate (with its receipt flag, I6) and the + // admissible ceremony are exactly what the disk records. + if let Some(p) = &parsed { + assert_eq!( + p.pin, + m.pins.last().map(|p| p.digest()), + "seed {seed} step {step}: disk pin diverged from model" + ); + assert_eq!( + p.candidate, m.pending, + "seed {seed} step {step}: disk candidate diverged from model" + ); + assert_eq!( + p.active, m.active, + "seed {seed} step {step}: disk admissible ceremony diverged" + ); + // Rule 7 tail: `retired` never names a live generation. + let pin_d = m.pins.last().map(|p| p.digest()); + let pend_d = m.pending.map(|(d, _)| d); + for r in &p.retired { + assert!( + Some(CertDigest::new(*r)) != pin_d + && Some(CertDigest::new(*r)) != pend_d, + "seed {seed} step {step}: retired names a live share" + ); + } + } // Rule 4: once frozen, frozen forever — the recorded pin and - // candidate stay visible for the operator's audit, but no - // share material may leave the store. + // share set are immutable; nothing is ever erased or + // exported. if m.frozen { assert!( store.equivocation(), @@ -2359,6 +2588,18 @@ mod tests { matches!(store.current_share(), Err(StoreError::Equivocation)), "seed {seed} step {step}: frozen store exports a share" ); + // The pin never decreases and no share is lost after the + // freeze: both are pinned at the moment it happened. + assert_eq!( + store.pinned().map(|p| p.digest()), + m.frozen_pin, + "seed {seed} step {step}: frozen pin moved" + ); + assert_eq!( + files, + m.frozen_files.clone().unwrap_or_default(), + "seed {seed} step {step}: share files changed after freeze" + ); for digest in &files { assert!( m.signed.contains_key(digest), @@ -2527,14 +2768,24 @@ mod tests { /// what the store actually exposes (truncating the pin chain and /// forgetting the admissible id if the state reverted). fn resync(&mut self, seed: u64, step: usize) { - let m = &mut self.model; + let parsed = fs::read(self.ndir().join("state")) + .ok() + .and_then(|b| parse_state(&b)); let store = self.store.as_ref().unwrap(); - m.frozen = store.equivocation(); - if m.frozen { - m.active = None; - m.pending = None; + if store.equivocation() { + // The freeze point is recorded once: everything after + // is terminal immutability, checked at every boundary. + if self.model.frozen_pin.is_none() { + self.model.frozen_pin = store.pinned().map(|p| p.digest()); + self.model.frozen_files = Some(self.share_files()); + } + self.model.frozen = true; + self.model.active = None; + self.model.pending = None; return; } + self.model.frozen = false; + let m = &mut self.model; match store.pinned() { Some(pin) => { let pos = m @@ -2548,7 +2799,14 @@ mod tests { } None => m.pins.clear(), } - m.pending = store.pending_candidate().map(|(d, _)| (d, false)); + // The candidate record on disk is the durable truth: digest + // AND the receipt flag (I6 survives every reopen). + m.pending = parsed.as_ref().and_then(|p| p.candidate); + assert_eq!( + store.pending_candidate().map(|(d, _)| d), + m.pending.map(|(d, _)| d), + "seed {seed} step {step}: disk candidate ≠ exposed candidate" + ); // A reopened candidate must be the pin's successor — open's // own convergence rule, asserted here too. if let (Some((d, _)), Some(pin)) = (m.pending, m.pins.last()) { @@ -2559,18 +2817,10 @@ mod tests { "seed {seed} step {step}: reopened candidate is not the pin's successor" ); } - // The admissible ceremony is persisted; resync reads what a - // crash-revert actually restored — an id whose recorded - // predecessor is again the pin may install, any other is - // contextually stale and must be refused. - m.active = store - .inner - .lock() - .unwrap() - .state - .active_ceremony - .as_ref() - .map(|a| (a.id, a.predecessor)); + // The admissible ceremony is persisted on disk; a reverted + // id whose predecessor is again the pin may install, any + // other is contextually stale and must be refused. + m.active = parsed.and_then(|p| p.active); } fn reopen(&mut self, seed: u64, step: usize) { @@ -2585,23 +2835,49 @@ mod tests { Ok(store) => { self.store = Some(store); self.resync(seed, step); + // Open idempotence: while the store is not frozen, + // open∘open is byte-identical with open — the second + // open rewrites nothing on disk. + if !self.model.frozen && self.rng.pick(5) == 0 { + let before = fs::read(self.ndir().join("state")).unwrap(); + self.store = None; + self.store = Some( + ShareStore::open( + self.dir, + &self.network, + self.signers.anchor().clone(), + Signers::did(1), + self.signers, + ) + .unwrap_or_else(|e| { + panic!("seed {seed} step {step}: idempotent reopen: {e:?}") + }), + ); + let after = fs::read(self.ndir().join("state")).unwrap(); + assert_eq!( + before, after, + "seed {seed} step {step}: second open rewrote state" + ); + } } Err(e) => { - // The only lawful refusal: the recorded state - // promises a receipted share whose file is gone — - // real disk damage the store must not guess past. - let state = fs::read(self.ndir().join("state")) + // Two lawful refusals: (a) the recorded state itself + // is unreadable (a torn file is disk damage the store + // must never guess past), (b) the parsed record + // promises a receipted share whose file is gone. + let parsed = fs::read(self.ndir().join("state")) .ok() - .and_then(|b| decode_state(&b).ok()); - let promised_missing = state.is_some_and(|s| { - s.candidate.is_some_and(|c| { - c.receipt_sent && !share_path(&self.ndir(), &c.digest).exists() - }) - }); - assert!( - matches!(e, StoreError::Corrupt) && promised_missing, - "seed {seed} step {step}: reopen failed: {e:?}" - ); + .and_then(|b| parse_state(&b)); + let lawful = match parsed { + None => true, + Some(p) => { + matches!(e, StoreError::Corrupt) + && p.candidate.is_some_and(|(d, sent)| { + sent && !share_path(&self.ndir(), &d).exists() + }) + } + }; + assert!(lawful, "seed {seed} step {step}: reopen failed: {e:?}"); self.dead = true; } } @@ -2654,6 +2930,7 @@ mod tests { snapshots: VecDeque::new(), transcript: 63, dead: false, + fork_allowed: seed % 4 != 3, }; let (signed, cert) = activated_genesis(runner.store.as_ref().unwrap(), signers); runner.model.pins.push(cert); @@ -2672,6 +2949,7 @@ mod tests { } let frozen = runner.model.frozen; let pending = runner.model.pending; + let fork_allowed = runner.fork_allowed; let roll = runner.rng.pick(100); match roll { // Fresh accept + install under the admissible id. @@ -2706,7 +2984,7 @@ mod tests { } // Stale install: any non-admissible ceremony id — // including retired ones — must fail. - 38..=43 if !frozen => { + 38..=41 if !frozen => { let stale: Vec = runner .model .ceremonies @@ -2794,34 +3072,154 @@ mod tests { runner.model.pending = None; runner.model.active = None; } - // Write an activated sibling proof for an interior - // predecessor: discovered at next open → frozen. - 74..=79 if !frozen && runner.model.pins.len() >= 2 => { - let i = 1 + runner.rng.pick((runner.model.pins.len() - 1) as u64) as usize; - let pred = runner.model.pins[i - 1].clone(); - let ceremony = - runner.model.ceremonies_by_pred[pred.digest().as_bytes()].clone(); - runner.transcript = runner.transcript.wrapping_add(1); - let signed = - sibling_successor(signers, &pred, &ceremony, runner.transcript); - let cert = verify_successor(&pred, &signed, signers).unwrap(); - let receipts = receipts_for(&cert, signers); - let activation = activation_for(&cert, &receipts, signers); - let path = proof_path(&runner.ndir(), &cert.digest()); - fs::write(&path, encode_proof(&signed, Some(&activation), &receipts)) + // Fork evidence — rare, and gated per seed so most runs + // stay long-lived. Two paths: an activated sibling or + // second genesis proof written to disk (discovered at + // next open), or a live record_activation fork. + 68..=70 if !frozen && fork_allowed => { + let live = runner.rng.pick(3) == 0; + let genesis_fork = runner.model.pins.len() < 2 || runner.rng.pick(3) == 0; + let (signed, cert, receipts, activation) = if genesis_fork { + // A second activated genesis is equivocation at + // the chain root. + let ceremony = signers.genesis(); + let (public, _shares) = + cert_tests::material(17, Threshold::new(2, 3).unwrap()); + let body = CertificateBody::genesis( + &signers.anchor(), + &ceremony, + &public, + &TranscriptHash::new([0xF1; 32]), + ) .unwrap(); + let signed = cert::tests::sign(body, signers); + let cert = + verify_certificate(&signers.anchor(), &signed, signers).unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + (signed, cert, receipts, activation) + } else { + let i = + 1 + runner.rng.pick((runner.model.pins.len() - 1) as u64) as usize; + let pred = runner.model.pins[i - 1].clone(); + let ceremony = + runner.model.ceremonies_by_pred[pred.digest().as_bytes()].clone(); + runner.transcript = runner.transcript.wrapping_add(1); + let signed = + sibling_successor(signers, &pred, &ceremony, runner.transcript); + let cert = verify_successor(&pred, &signed, signers).unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + (signed, cert, receipts, activation) + }; runner .model .signed - .insert(*cert.digest().as_bytes(), signed); - runner.reopen(seed, step); + .insert(*cert.digest().as_bytes(), signed.clone()); + if live && !genesis_fork { + // The live path: the store itself detects and + // persists the fork, then freezes. + assert!(matches!( + runner.store().record_activation( + &signed, + &activation, + &receipts, + signers + ), + Err(StoreError::Equivocation) + )); + runner.reopen(seed, step); + } else { + let path = proof_path(&runner.ndir(), &cert.digest()); + fs::write(&path, encode_proof(&signed, Some(&activation), &receipts)) + .unwrap(); + runner.reopen(seed, step); + } assert!( runner.model.frozen, "seed {seed} step {step}: activated fork did not freeze" ); } - // Rule 10: a second opener changes nothing. - 80..=85 => { + // Proof-file loss: an activated certificate's proof + // disappears before reopen — convergence to an earlier + // crash-free point, never corruption. + 71..=73 => { + let proofs: Vec = fs::read_dir(runner.ndir().join("proofs")) + .map(|rd| rd.flatten().map(|e| e.path()).collect()) + .unwrap_or_default(); + if !proofs.is_empty() { + runner.store = None; + let path = + proofs[runner.rng.pick(proofs.len() as u64) as usize].clone(); + let _ = fs::remove_file(path); + runner.reopen(seed, step); + } + } + // Resurrection probe: a retired certificate must never + // reinstall, even under the admissible ceremony id. + 74..=77 if !frozen && runner.model.pending.is_none() => { + let pin = runner.model.pins.last().map(|p| p.digest()); + let candidates: Vec = runner + .model + .retired + .iter() + .filter_map(|d| runner.model.signed.get(d)) + .filter(|s| s.body.previous().copied() == pin) + .cloned() + .collect(); + if let (Some((id, _)), false) = (runner.model.active, candidates.is_empty()) + { + let signed = candidates + [runner.rng.pick(candidates.len() as u64) as usize] + .clone(); + let ceremony = runner + .model + .ceremonies + .iter() + .find(|c| c.statement().id == id) + .unwrap() + .clone(); + let (_, mut shares) = + cert_tests::material(11, Threshold::new(2, 3).unwrap()); + let candidate = mk_candidate(shares.remove(0), &signed.body); + assert!( + matches!( + runner + .store() + .install_candidate(&signed, candidate, &ceremony, signers), + Err(StoreError::Retired) + ), + "seed {seed} step {step}: retired certificate reinstalled" + ); + // The activation path must refuse a retired + // digest too (C1/C2): an erased share can never + // be re-pinned by a late record_activation. + let cert = verify_successor( + runner.model.pins.last().unwrap(), + &signed, + signers, + ) + .unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + assert!( + matches!( + runner.store().record_activation( + &signed, + &activation, + &receipts, + signers + ), + Err(StoreError::Retired) + ), + "seed {seed} step {step}: retired certificate reactivated" + ); + } + } + // Rule 10: a second opener is Busy and changes nothing + // on disk. + 78..=83 => { + let before = runner.disk_hash(); assert!(matches!( ShareStore::open( dir, @@ -2832,15 +3230,21 @@ mod tests { ), Err(StoreError::Busy) )); + assert_eq!( + runner.disk_hash(), + before, + "seed {seed} step {step}: Busy opener mutated the disk" + ); } // Clean reopen. - 86..=90 => runner.reopen(seed, step), - // Crash replays: revert `state` to a recent snapshot - // (commit-without-completion), delete a dealt marker - // (post-commit pre-marker crash), or leave temp debris; - // every variant ends in a reopen + resync. + 84..=88 => runner.reopen(seed, step), + // Crash replays and disk damage: revert `state` to a + // recent snapshot, delete a dealt marker, drop temp + // debris, truncate `state` to a torn prefix, delete a + // non-live share file (unrecorded disk damage), or just + // drop unclosed; every variant ends in a reopen + resync. _ => { - match runner.rng.pick(4) { + match runner.rng.pick(6) { 0 if !runner.snapshots.is_empty() => { runner.store = None; let i = runner.rng.pick(runner.snapshots.len() as u64) as usize; @@ -2863,6 +3267,63 @@ mod tests { .join(format!("share-{}.tmp-1-9", hex(&[0xAB; 32]))); fs::write(&debris, [0x55u8; 67]).unwrap(); } + 3 => { + // A torn `state`: atomic renames make this + // disk damage, so open must refuse — never + // adopt a truncated record. + runner.store = None; + let state = runner.ndir().join("state"); + if let Ok(bytes) = fs::read(&state) { + if bytes.len() > 4 { + fs::write(&state, &bytes[..bytes.len() / 2]).unwrap(); + } + } + assert!( + ShareStore::open( + dir, + &network, + signers.anchor().clone(), + Signers::did(1), + signers + ) + .is_err(), + "seed {seed} step {step}: torn state opened" + ); + } + 4 => { + // Delete a share file the state does not + // record: unrecorded disk damage. + runner.store = None; + let live: BTreeSet<[u8; 32]> = { + let mut s = BTreeSet::new(); + if let Some(p) = runner.model.pins.last() { + s.insert(*p.digest().as_bytes()); + } + if let Some((d, _)) = runner.model.pending { + s.insert(*d.as_bytes()); + } + s + }; + let victims: Vec = fs::read_dir(runner.ndir()) + .map(|rd| { + rd.flatten() + .filter(|e| { + e.file_name() + .to_str() + .and_then(|n| n.strip_prefix("share-")) + .and_then(decode_hex) + .is_some_and(|d| !live.contains(&d)) + }) + .map(|e| e.path()) + .collect() + }) + .unwrap_or_default(); + if !victims.is_empty() { + let _ = fs::remove_file( + &victims[runner.rng.pick(victims.len() as u64) as usize], + ); + } + } _ => runner.store = None, } runner.reopen(seed, step); @@ -2929,4 +3390,213 @@ mod tests { } } } + + /// P1-a: a frozen reopen keeps the recorded verified pin and erases + /// NOTHING — the member's last share is never lost to a fork. + /// Details: TC-2221 + #[test] + fn frozen_reopen_keeps_pin_and_every_share() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let digest_a; + let digest_z; + { + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + // A: activated candidate (retires G); Z: receipted candidate on A. + let (s_a, mut raw_a, c_a) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_a).unwrap(); + store + .install_candidate( + &s_a, + mk_candidate(raw_a.remove(0), &s_a.body), + &c_a, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); + let cert_a = verify_successor(&cert_g, &s_a, &signers).unwrap(); + digest_a = cert_a.digest(); + let receipts_a = receipts_for(&cert_a, &signers); + let activation_a = activation_for(&cert_a, &receipts_a, &signers); + store + .record_activation(&s_a, &activation_a, &receipts_a, &signers) + .unwrap(); + let mut stmt_z = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_z.id = CeremonyId::new([0x62; 32]); + stmt_z.current = Some(cert_a.digest()); + stmt_z.epoch = cert_a.body().epoch().checked_next().unwrap(); + stmt_z.rekey = cert_a.body().rekey(); + let c_z = + AuthorizedCeremony::verify_for(&cert_a, &signers.statement(&stmt_z), &signers) + .unwrap(); + let body_z = CertificateBody::successor( + &cert_a, + &c_z, + &cert_tests::material(11, Threshold::new(2, 3).unwrap()).0, + &TranscriptHash::new([0x5a; 32]), + ) + .unwrap(); + let s_z = cert::tests::sign(body_z, &signers); + let mut raw_z = cert_tests::material(11, Threshold::new(2, 3).unwrap()).1; + store.accept_ceremony(&c_z).unwrap(); + digest_z = store + .install_candidate( + &s_z, + mk_candidate(raw_z.remove(0), &s_z.body), + &c_z, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); + // Fork evidence: an activated sibling of A. + let s_fork = sibling_successor(&signers, &cert_g, &c_a, 0xf0); + let cert_f = verify_successor(&cert_g, &s_fork, &signers).unwrap(); + let receipts_f = receipts_for(&cert_f, &signers); + let activation_f = activation_for(&cert_f, &receipts_f, &signers); + assert!(matches!( + store.record_activation(&s_fork, &activation_f, &receipts_f, &signers), + Err(StoreError::Equivocation) + )); + } + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(store.equivocation()); + // The recorded pin stands — verified activated, never lowered. + assert_eq!(store.pinned().unwrap().digest(), digest_a); + // Nothing is erased while equivocated: A's share and the receipted + // candidate's share both survive. + let ndir = netdir(&dir, &network); + assert!(ndir + .join(format!("share-{}", hex(digest_a.as_bytes()))) + .exists()); + assert!(ndir + .join(format!("share-{}", hex(digest_z.as_bytes()))) + .exists()); + } + + /// P1-b: a retired candidate certificate can never be reinstalled, + /// even under the currently admissible ceremony. Details: TC-2221 + #[test] + fn retired_candidate_cert_cannot_reinstall() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + let (s_x, mut raw_x, c_x) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_x).unwrap(); + let digest_x = store + .install_candidate( + &s_x, + mk_candidate(raw_x.remove(0), &s_x.body), + &c_x, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); + // A fresh ceremony supersedes X (C1 retires and erases the share). + let mut stmt_y = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_y.id = CeremonyId::new([0x61; 32]); + stmt_y.current = Some(cert_g.digest()); + stmt_y.epoch = cert_g.body().epoch().checked_next().unwrap(); + stmt_y.rekey = cert_g.body().rekey(); + let ceremony_y = + AuthorizedCeremony::verify_for(&cert_g, &signers.statement(&stmt_y), &signers).unwrap(); + assert_eq!( + store.accept_ceremony(&ceremony_y).unwrap(), + AcceptOutcome::Fresh + ); + assert!(store.retired().iter().any(|r| r.digest == digest_x)); + // Reinstalling X's certificate — the same cert whose share C1 + // erased — under the now-admissible ceremony must be refused. + let (s_x2, mut raw_x2, _c) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + assert_eq!(s_x2.body.digest(), digest_x); + assert!(matches!( + store.install_candidate( + &s_x2, + mk_candidate(raw_x2.remove(0), &s_x2.body), + &ceremony_y, + &signers + ), + Err(StoreError::Retired) + )); + } + + /// The activation path obeys the retirement record too: a C1-retired + /// candidate's certificate — still a valid successor of the pin — must + /// never be re-pinned by a late `record_activation`. Details: TC-2221 + #[test] + fn retired_candidate_cert_cannot_reactivate() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + let (s_x, mut raw_x, c_x) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_x).unwrap(); + let digest_x = store + .install_candidate( + &s_x, + mk_candidate(raw_x.remove(0), &s_x.body), + &c_x, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); + // A fresh ceremony over the same pin retires and erases X (C1). + let mut stmt_y = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_y.id = CeremonyId::new([0x62; 32]); + stmt_y.current = Some(cert_g.digest()); + stmt_y.epoch = cert_g.body().epoch().checked_next().unwrap(); + stmt_y.rekey = cert_g.body().rekey(); + let ceremony_y = + AuthorizedCeremony::verify_for(&cert_g, &signers.statement(&stmt_y), &signers).unwrap(); + assert_eq!( + store.accept_ceremony(&ceremony_y).unwrap(), + AcceptOutcome::Fresh + ); + assert!(store.retired().iter().any(|r| r.digest == digest_x)); + // X's certificate is still a perfectly valid successor of the pin; + // the durable retirement record — not lineage — refuses it. + let cert_x = verify_successor(&cert_g, &s_x, &signers).unwrap(); + let receipts = receipts_for(&cert_x, &signers); + let activation = activation_for(&cert_x, &receipts, &signers); + assert!(matches!( + store.record_activation(&s_x, &activation, &receipts, &signers), + Err(StoreError::Retired) + )); + // The pin did not move and the erased share stays erased. + assert_eq!(store.pinned().map(|p| p.digest()), Some(cert_g.digest())); + assert!(!share_path(&netdir(&dir, &signers.anchor().network), &digest_x).exists()); + } } From 7ecc7b5027c1f16eecb34d4fa024148d94ad0f65 Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 11:33:32 +0000 Subject: [PATCH 6/8] fix(threshold): freeze on tip-in-retired, harden erasure against links, persist live forks, bound+compact retirement with predecessors; state v3; Details: TC-2221 - open: chain tip that is also a retired digest freezes (P3-1) - erase_share_file: unlink names with nlink>1, refuse links/symlinks before zeroing (P3-2) - live second activated genesis persists the proof and freezes (P3-3) - state v3: retired carries predecessor; dead entries compact at the 65536 bound, over-bound refuses (P3-4) - model: save/restore around open-path damage, recover-and-continue refusals, member-drop, pin-share and proof-loss faults, read-only dir, byte-frozen outcome check, op accounting --- tinycloud-threshold/SPEC.md | 2 +- tinycloud-threshold/src/store.rs | 1032 +++++++++++++++++++++++++++--- 2 files changed, 949 insertions(+), 85 deletions(-) diff --git a/tinycloud-threshold/SPEC.md b/tinycloud-threshold/SPEC.md index 4be4ff6e..fdc07dff 100644 --- a/tinycloud-threshold/SPEC.md +++ b/tinycloud-threshold/SPEC.md @@ -556,6 +556,6 @@ The `store` module keeps one member's share material for one network under `, /// Retirement sequence counter for this member. pub erased_at: u64, } +/// Hard bounds on the durable lists: the store REFUSES the operation that +/// would cross them rather than letting `state` grow past what a reopen +/// can parse. Details: TC-2221 +const MAX_RETIRED: usize = 65_536; +const MAX_SEEN_CEREMONIES: usize = 65_536; + /// The pending candidate record. #[derive(Clone, Debug)] struct CandidateRecord { @@ -108,7 +120,7 @@ struct State { fn encode_state(state: &State) -> Vec { let mut e = Encoder::new(); - e.u8(2); + e.u8(3); e.u8(u8::from(state.pin.is_some())); if let Some(pin) = state.pin { e.fixed(pin.as_bytes()); @@ -131,8 +143,8 @@ fn encode_state(state: &State) -> Vec { e.fixed(predecessor.as_bytes()); } } - // Ceremony ids and retired entries are bounded by the on-disk format cap - // enforced in decode_state; u32 cannot overflow below memory limits. + // Ceremony ids and retired entries are bounded by the write caps; u32 + // cannot overflow below memory limits. e.u32(state.seen_ceremonies.len() as u32); for id in &state.seen_ceremonies { e.fixed(id.as_bytes()); @@ -140,6 +152,10 @@ fn encode_state(state: &State) -> Vec { e.u32(state.retired.len() as u32); for retired in &state.retired { e.fixed(retired.digest.as_bytes()); + e.u8(u8::from(retired.predecessor.is_some())); + if let Some(predecessor) = retired.predecessor { + e.fixed(predecessor.as_bytes()); + } e.u64(retired.erased_at); } e.u64(state.seq); @@ -148,7 +164,7 @@ fn encode_state(state: &State) -> Vec { fn decode_state(bytes: &[u8]) -> Result { let mut d = Decoder::new(bytes)?; - if d.u8()? != 2 { + if d.u8()? != 3 { return Err(Error::InvalidEncoding.into()); } let pin = if d.u8()? == 1 { @@ -198,6 +214,11 @@ fn decode_state(bytes: &[u8]) -> Result { for _ in 0..retired_len { retired.push(RetiredGeneration { digest: CertDigest::new(d.fixed()?), + predecessor: if d.u8()? == 1 { + Some(CertDigest::new(d.fixed()?)) + } else { + None + }, erased_at: d.u64()?, }); } @@ -311,6 +332,10 @@ pub enum StoreError { /// The certificate does not extend the pin, or a genesis arrives after one. #[error("activation does not extend the pinned certificate")] InvalidActivation, + /// A durable list reached its bound; the operation is refused rather + /// than letting `state` grow past what a reopen can parse. + #[error("share store durable list is at capacity")] + Capacity, /// Activated equivocation froze the pin; further activations are refused. #[error("activated equivocation froze the pin")] Equivocation, @@ -413,6 +438,30 @@ fn erase_share_file(path: &Path) -> Result<(), StoreError> { return Ok(()); } let length = usize::try_from(meta.len()).unwrap_or(usize::MAX); + // Zeroing writes through the inode: a name with more than one link — + // or one that stops being a regular file between metadata and open — + // must be dropped as a bare name, never overwritten. O_NOFOLLOW keeps + // that check atomic with the open. + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt, OpenOptionsExt}; + if meta.nlink() > 1 { + fs::remove_file(path)?; + if let Some(parent) = path.parent() { + fs::File::open(parent)?.sync_all()?; + } + return Ok(()); + } + use std::io::Write; + const O_NOFOLLOW: i32 = 0x20000; + let mut file = fs::OpenOptions::new() + .write(true) + .custom_flags(O_NOFOLLOW) + .open(path)?; + file.write_all(&Zeroizing::new(vec![0u8; length]))?; + file.sync_all()?; + } + #[cfg(not(unix))] { use std::io::Write; let mut file = fs::OpenOptions::new().write(true).open(path)?; @@ -434,14 +483,32 @@ fn proof_path(dir: &Path, digest: &CertDigest) -> PathBuf { dir.join("proofs").join(hex(digest.as_bytes())) } -fn record_retired(state: &mut State, digest: CertDigest) { +/// Record a retirement durably before its unlink: the entry carries the +/// certificate the retired generation descended from, which is the only +/// pin context under which the digest can ever be presented again. +/// Details: TC-2221 +fn record_retired(state: &mut State, digest: CertDigest, predecessor: Option) { state.seq = state.seq.saturating_add(1); state.retired.push(RetiredGeneration { digest, + predecessor, erased_at: state.seq, }); } +/// Compact the retirement list only when it nears the bound: an entry is +/// reachable — and therefore must be kept — only while the pin equals its +/// recorded predecessor; dead entries are dropped first. Reachable entries +/// are never dropped: if the list still overflows, the operation must be +/// refused instead of resurrecting a retired share. Details: TC-2221 +fn compact_retired(state: &mut State, pin: Option) -> bool { + if state.retired.len() <= MAX_RETIRED { + return true; + } + state.retired.retain(|r| r.predecessor == pin); + state.retired.len() <= MAX_RETIRED +} + fn hex(bytes: &[u8]) -> String { const HEX: &[u8; 16] = b"0123456789abcdef"; let mut out = String::with_capacity(bytes.len().saturating_mul(2)); @@ -723,12 +790,39 @@ impl ShareStore { } } + // The chain tip being a retired digest is a durable contradiction: + // it means the member's usable share is precisely the one the + // record erased (a never-activated candidate that later gained an + // activated proof on disk — peer sync, operator restore). The live + // path refuses the same certificate as `Retired`; reopening freezes + // rather than letting R1 erase the still-live share underneath it. + // Superseded pins mid-chain are normal history: every outgoing pin + // keeps its proof and its retirement record. + if !equivocation + && computed_pin.is_some_and(|tip| state.retired.iter().any(|r| r.digest == tip)) + { + equivocation = true; + } + // Candidate reconciliation and R1/C2/S1 erasure run ONLY on an // unequivocal chain. Once activated equivocation is on disk the // store freezes in place: the recorded verified pin stands, every // share file is kept (losing nothing is the whole point of the // freeze), and only the operator's recovery path rewrites it. if !equivocation { + // The pin's share is this member's live key material: durable + // proof says it was installed and never retired, so its absence + // is disk damage the store must not guess past. + if let Some(pin) = computed_pin { + let holds_member = verified + .get(&pin) + .map(|c| c.body().members().iter().any(|m| m.node == member)) + .unwrap_or(false); + if holds_member && !share_path(&network_dir, &pin).exists() { + return Err(StoreError::Corrupt); + } + } + // Candidate reconciliation. Chain membership and C2 losses are // judged BEFORE the missing-file check: a candidate already // retired by durable proof is settled even when its share is @@ -759,7 +853,7 @@ impl ShareStore { if lost { // Commit the retirement first; the unlink is owed // work a crash simply re-discovers as S1. - record_retired(&mut state, candidate.digest); + record_retired(&mut state, candidate.digest, candidate.predecessor); state.candidate = None; state.pin = computed_pin; write_atomic(&network_dir.join("state"), &encode_state(&state))?; @@ -804,10 +898,10 @@ impl ShareStore { if verified.contains_key(&digest) { // Superseded activated generation (R1). if !state.retired.iter().any(|r| r.digest == digest) { - record_retired(&mut state, digest); + let predecessor = proofs[&digest].body.previous().copied(); + record_retired(&mut state, digest, predecessor); state.pin = computed_pin; write_atomic(&network_dir.join("state"), &encode_state(&state))?; - state_dirty = false; } owed_erasures.push(entry.path()); } else { @@ -1061,6 +1155,19 @@ impl ShareStore { return Err(StoreError::Equivocation); } } + // A second ACTIVATED genesis is the same equivocation reopening + // freezes on: verify it, persist the proof durably, then freeze — + // never silently drop the evidence. + if signed.body.previous().is_none() && inner.pin.is_some() { + let root = verify_certificate(&self.anchor, signed, v)?; + verify_activation(&root, receipts, ReadyQuorum::AllHolders, activation, v)?; + write_atomic( + &proof_path(&self.network_dir, &root.digest()), + &encode_proof(signed, Some(activation), receipts), + )?; + inner.equivocation = true; + return Err(StoreError::Equivocation); + } let verified = match &inner.pin { Some(pin) => verify_successor(pin, signed, v)?, None => verify_certificate(&self.anchor, signed, v)?, @@ -1088,7 +1195,7 @@ impl ShareStore { if candidate.digest != digest && !next.retired.iter().any(|r| r.digest == candidate.digest) { - record_retired(&mut next, candidate.digest); + record_retired(&mut next, candidate.digest, candidate.predecessor); erasures.push(share_path(&self.network_dir, &candidate.digest)); } } @@ -1096,11 +1203,18 @@ impl ShareStore { if let Some(old_pin) = &inner.pin { let old = old_pin.digest(); if !next.retired.iter().any(|r| r.digest == old) { - record_retired(&mut next, old); + record_retired(&mut next, old, old_pin.body().previous().copied()); erasures.push(share_path(&self.network_dir, &old)); } } next.pin = Some(digest); + // Retirement history is audit data and is kept; only when the list + // nears the bound are provably-dead entries (keyed to superseded + // pins) compacted away — reachable ones are never evicted, the + // operation is refused instead of resurrecting a share. + if !compact_retired(&mut next, Some(digest)) { + return Err(StoreError::Capacity); + } // Ceremonies accepted against the old pin are stale: the next // admissible ceremony must be accepted afresh for this pin. next.active_ceremony = None; @@ -1153,11 +1267,20 @@ impl ShareStore { if let Some(candidate) = &next.candidate { if candidate.predecessor == statement.current { let digest = candidate.digest; - record_retired(&mut next, digest); + let predecessor = candidate.predecessor; + record_retired(&mut next, digest, predecessor); next.candidate = None; owed = Some(share_path(&self.network_dir, &digest)); } } + // Refuse rather than let the durable lists grow past their bounds; + // dead retirement records are compacted first so legitimate + // long-lived stores never hit the wall. + if next.seen_ceremonies.len() >= MAX_SEEN_CEREMONIES + || !compact_retired(&mut next, inner.pin.as_ref().map(|p| p.digest())) + { + return Err(StoreError::Capacity); + } next.seen_ceremonies.push(statement.id); next.active_ceremony = Some(ActiveCeremony { id: statement.id, @@ -1231,6 +1354,38 @@ mod tests { fn netdir(dir: &Path, network: &NetworkUrn) -> PathBuf { store_dir(dir, network) } + fn copy_dir(src: &Path, dst: &Path) -> std::io::Result<()> { + fs::create_dir_all(dst)?; + for entry in fs::read_dir(src)? { + let entry = entry?; + let to = dst.join(entry.file_name()); + if entry.file_type()?.is_dir() { + copy_dir(&entry.path(), &to)?; + } else { + fs::copy(entry.path(), &to)?; + } + } + Ok(()) + } + + /// Make every entry under `dir` writable again — a read-only-dir fault + /// must be undone before the disk can be restored or cleaned up. + #[cfg(unix)] + fn make_writable(dir: &Path) { + use std::os::unix::fs::PermissionsExt; + if let Ok(entries) = fs::read_dir(dir) { + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + make_writable(&path); + } + } + } + let _ = fs::set_permissions(dir, fs::Permissions::from_mode(0o755)); + } + + #[cfg(not(unix))] + fn make_writable(_dir: &Path) {} fn open(dir: &Path, signers: &Signers) -> ShareStore { ShareStore::open( @@ -1667,13 +1822,23 @@ mod tests { let store = open(&dir, &signers); let (_signed, cert) = activated_genesis(&store, &signers); digest_old = cert.digest(); - let (s_succ, _raw, _ceremony) = cert_tests::successor( + let (s_succ, mut raw, ceremony_succ) = cert_tests::successor( &signers, &cert, CeremonyKind::Reshare, Threshold::new(2, 3).unwrap(), 11, ); + store.accept_ceremony(&ceremony_succ).unwrap(); + store + .install_candidate( + &s_succ, + mk_candidate(raw.remove(0), &s_succ.body), + &ceremony_succ, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); let succ = verify_successor(&cert, &s_succ, &signers).unwrap(); digest_succ = succ.digest(); // Crash simulation: activated proof written, state never updated. @@ -2117,6 +2282,11 @@ mod tests { encode_proof(&s_w, Some(&activation), &receipts), ) .unwrap(); + // The winner's share was installed before activation — the + // durable-install state its receipts imply. + let (_pub_w, mut raw_w) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); + let winner_share = mk_candidate(raw_w.remove(0), &s_w.body).to_bytes(); + fs::write(share_path(&ndir, &succ_digest), &winner_share).unwrap(); fs::remove_file(ndir.join(format!("share-{}", hex(digest_c.as_bytes())))).unwrap(); } // Restart: the durable retirement proof settles the candidate even @@ -2322,6 +2492,7 @@ mod tests { /// A failing run prints its seed so it can be replayed. Details: TC-2221 mod model { use super::super::tests::*; + use crate::ceremony::CeremonyRecipient; use std::collections::{BTreeMap, BTreeSet, VecDeque}; /// splitmix64; cheap and stable across platforms for reproducing runs. @@ -2369,6 +2540,9 @@ mod tests { /// Shares whose erasure is owed; empty at every operation /// boundary in this harness (it never injects I/O failures). debt: BTreeSet<[u8; 32]>, + /// The pin's member set no longer names this node (a legal + /// member-drop reshare): no live share is owed. + dropped: bool, frozen: bool, /// The pin and share-file set recorded at the moment the store /// froze: the freeze is terminal, so neither may ever move. @@ -2376,10 +2550,11 @@ mod tests { frozen_files: Option>, } - /// The model's own parser for the on-disk v2 `state` record — the + /// The model's own parser for the on-disk v3 `state` record — the /// observable contract a crash leaves behind, independent of the /// store's internals. Returns `None` for any unreadable or /// unparsable file. + #[derive(Debug)] struct ParsedState { pin: Option, candidate: Option<(CertDigest, bool)>, @@ -2395,7 +2570,7 @@ mod tests { *cur = end; Some(slice) }; - if take(1, &mut cur)? != [2] { + if take(1, &mut cur)? != [3] { return None; } let opt32 = |cur: &mut usize| -> Option> { @@ -2423,13 +2598,20 @@ mod tests { None }; let seen_len = u32::from_be_bytes(take(4, &mut cur)?.try_into().unwrap()) as usize; + if seen_len > 1_000_000 { + return None; + } for _ in 0..seen_len { take(32, &mut cur)?; } let retired_len = u32::from_be_bytes(take(4, &mut cur)?.try_into().unwrap()) as usize; + if retired_len > 1_000_000 { + return None; + } let mut retired = BTreeSet::new(); for _ in 0..retired_len { retired.insert(take(32, &mut cur)?.try_into().unwrap()); + opt32(&mut cur)?; // predecessor (v3) take(8, &mut cur)?; // erased_at } take(8, &mut cur)?; // seq @@ -2453,9 +2635,15 @@ mod tests { rng: Rng, /// Recent `state` file contents for crash replays. snapshots: VecDeque>, - /// Set when the disk promises a receipted share it cannot serve: - /// Corrupt is the store's only honest answer and the run ends. - dead: bool, + /// How many operations the scheduler actually executed — a run + /// that stalls on damage and never recovers fails this count. + executed: usize, + /// How many lawful refusals were rolled back to continue. + recoveries: usize, + /// Executions per operation kind, for the run summary. + ops: BTreeMap<&'static str, u64>, + /// Pre-fault disk backup used to recover after a lawful refusal. + backup: Option, /// Whether this seed may inject fork evidence; a quarter of /// seeds never fork so most runs exercise long unfrozen chains. fork_allowed: bool, @@ -2501,6 +2689,54 @@ mod tests { hash } + /// Copy the member's entire share dir for a later restore — + /// called before every open-path damage op so the scheduler can + /// roll the disk back and continue the run after a lawful + /// refusal. + fn save_disk(&mut self) { + let backup = self.dir.join("model-backup"); + if backup.exists() { + fs::remove_dir_all(&backup).unwrap(); + } + copy_dir(&self.ndir(), &backup).unwrap(); + self.backup = Some(backup); + } + + /// Roll the share dir back to the snapshot `save_disk` took. + fn restore_disk(&mut self) { + let backup = self.backup.take().expect("restore needs a snapshot"); + let ndir = self.ndir(); + if ndir.exists() { + make_writable(&ndir); + fs::remove_dir_all(&ndir).unwrap(); + } + copy_dir(&backup, &ndir).unwrap(); + fs::remove_dir_all(&backup).unwrap(); + } + + fn mark_op(&mut self, op: &'static str) { + self.executed += 1; + *self.ops.entry(op).or_default() += 1; + } + + /// Whether `proofs/` can still prove `pin`: its own proof file + /// plus every ancestor's up to genesis must exist. Used only to + /// classify a `Corrupt` refusal — the store's own verification + /// stays authoritative. + fn pin_is_provable(&self, pin: &CertDigest) -> bool { + let mut current = Some(*pin); + while let Some(digest) = current { + let Ok(bytes) = fs::read(proof_path(&self.ndir(), &digest)) else { + return false; + }; + let Ok((signed, _)) = decode_proof(&bytes) else { + return false; + }; + current = signed.body.previous().copied(); + } + true + } + fn share_files(&self) -> BTreeSet<[u8; 32]> { let mut out = BTreeSet::new(); if let Ok(entries) = fs::read_dir(self.ndir()) { @@ -2532,13 +2768,10 @@ mod tests { /// called after EVERY scheduler step. Details: TC-2221 fn assert_boundary(&self, seed: u64, step: usize) { let m = &self.model; - let Some(store) = self.store.as_ref() else { - assert!( - self.dead, - "seed {seed} step {step}: closed store without Corrupt" - ); - return; - }; + let store = self + .store + .as_ref() + .expect("seed: closed store outside a crash arm"); let files = self.share_files(); // Rule 11: the state file is always exactly one v2 record. let parsed = fs::read(self.ndir().join("state")) @@ -2546,7 +2779,7 @@ mod tests { .and_then(|b| parse_state(&b)); assert!( parsed.is_some(), - "seed {seed} step {step}: state file does not parse as v2" + "seed {seed} step {step}: state file does not parse as v3" ); // Rule 12: the model and the parsed state never diverge — // pin, candidate (with its receipt flag, I6) and the @@ -2579,6 +2812,8 @@ mod tests { // Rule 4: once frozen, frozen forever — the recorded pin and // share set are immutable; nothing is ever erased or // exported. + // Rule 4: once frozen, frozen forever — the recorded pin is + // immutable and nothing is exported or erased past it. if m.frozen { assert!( store.equivocation(), @@ -2588,18 +2823,23 @@ mod tests { matches!(store.current_share(), Err(StoreError::Equivocation)), "seed {seed} step {step}: frozen store exports a share" ); - // The pin never decreases and no share is lost after the - // freeze: both are pinned at the moment it happened. + // The recorded pin stands — it may be an older + // generation than the model's tip when a late fault + // moved the model past the durable write. assert_eq!( store.pinned().map(|p| p.digest()), m.frozen_pin, "seed {seed} step {step}: frozen pin moved" ); - assert_eq!( - files, - m.frozen_files.clone().unwrap_or_default(), - "seed {seed} step {step}: share files changed after freeze" - ); + // Nothing past the recorded pin is ever erased: the + // share set at freeze is a floor, not an equality — a + // crash can add candidates the freeze then keeps. + for d in m.frozen_files.clone().unwrap_or_default() { + assert!( + files.contains(&d), + "seed {seed} step {step}: a share frozen in place was lost" + ); + } for digest in &files { assert!( m.signed.contains_key(digest), @@ -2626,8 +2866,15 @@ mod tests { // live except as recorded debt — the on-disk share set is // exactly {pin?} ∪ {pending?} ∪ debt. let mut live: BTreeSet<[u8; 32]> = BTreeSet::new(); + // A member-drop pin owes no live share to this node — the + // new cohort's share was never installed and the old pin's + // is retired. A pending candidate, though, is this node's + // own share (install only ever binds our index): it is owed + // regardless of the pin's member set. if let Some(d) = pin { - live.insert(*d.as_bytes()); + if !m.dropped { + live.insert(*d.as_bytes()); + } } if let Some((d, _)) = m.pending { live.insert(*d.as_bytes()); @@ -2641,7 +2888,14 @@ mod tests { for digest in &files { assert!( live.contains(digest) || m.debt.contains(digest), - "seed {seed} step {step}: share file outside live set and debt" + "seed {seed} step {step}: share file {} outside live {:?} and debt \ + (pins={} pending={:?} dropped={} ops={:?})", + hex(digest), + live.iter().map(|d| hex(d)).collect::>(), + m.pins.len(), + m.pending, + m.dropped, + self.ops ); } assert!( @@ -2660,7 +2914,7 @@ mod tests { // Rule 6 + usable share: with a pin file on disk the bound // share is exported. if let Some(d) = pin { - if files.contains(d.as_bytes()) { + if !m.dropped && files.contains(d.as_bytes()) { assert!( matches!(store.current_share(), Ok(Some(_))), "seed {seed} step {step}: pinned share not usable" @@ -2775,29 +3029,69 @@ mod tests { if store.equivocation() { // The freeze point is recorded once: everything after // is terminal immutability, checked at every boundary. + // The recorded pin may sit before the model's tip — a + // freeze discovered at open truncates the model to the + // recorded position, not the crash-free projection. + match store.pinned() { + Some(pin) => { + let pos = self + .model + .pins + .iter() + .position(|p| p.digest() == pin.digest()); + if let Some(pos) = pos { + self.model.pins.truncate(pos + 1); + } + } + None => self.model.pins.clear(), + } if self.model.frozen_pin.is_none() { self.model.frozen_pin = store.pinned().map(|p| p.digest()); self.model.frozen_files = Some(self.share_files()); } self.model.frozen = true; - self.model.active = None; - self.model.pending = None; + // Frozen resync still reads what the disk records: the + // state file is the durable truth for pending/active + // too — assertions below only check their tails. + self.model.active = parsed.as_ref().and_then(|p| p.active); + self.model.pending = parsed.as_ref().and_then(|p| p.candidate); return; } - self.model.frozen = false; let m = &mut self.model; match store.pinned() { Some(pin) => { - let pos = m - .pins - .iter() - .position(|p| p.digest() == pin.digest()) - .unwrap_or_else(|| { - panic!("seed {seed} step {step}: reopen pinned unknown certificate") - }); - m.pins.truncate(pos + 1); + if let Some(pos) = m.pins.iter().position(|p| p.digest() == pin.digest()) { + m.pins.truncate(pos + 1); + } else if let Some(signed) = m.signed.get(pin.digest().as_bytes()).cloned() + { + // A pin discovered on disk (proof written while + // the member was offline — e.g. a member-drop): + // extend the model chain to the verified tip. + let cert = + verify_successor(m.pins.last().unwrap(), &signed, self.signers) + .unwrap_or_else(|e| { + panic!( + "seed {seed} step {step}: disk pin not a chain successor: {e:?}" + ) + }); + m.pins.push(cert); + } else { + panic!("seed {seed} step {step}: reopen pinned unknown certificate") + } + } + None => { + m.pins.clear(); + m.dropped = false; } - None => m.pins.clear(), + } + // Member-drop: when the tip's member set no longer names + // this node, the pin owes no usable share. + if let Some(pin) = m.pins.last() { + m.dropped = !pin + .body() + .members() + .iter() + .any(|cm| cm.node == Signers::did(1)); } // The candidate record on disk is the durable truth: digest // AND the receipt flag (I6 survives every reopen). @@ -2823,7 +3117,45 @@ mod tests { m.active = parsed.and_then(|p| p.active); } + /// The disk must be byte-for-byte what open found when the + /// outcome is a freeze: open freezes in place — nothing is + /// erased or rewritten once equivocation is discovered. + fn assert_frozen_outcome( + &self, + seed: u64, + step: usize, + files_before: &BTreeSet<[u8; 32]>, + state_before: &Option>, + ) { + let store = self.store.as_ref().unwrap(); + if !store.equivocation() { + return; + } + let files_after = self.share_files(); + for d in files_before { + assert!( + files_after.contains(d), + "seed {seed} step {step}: frozen open erased a share" + ); + } + let state_after = fs::read(self.ndir().join("state")).ok(); + if let Some(before) = state_before { + assert_eq!( + before.as_slice(), + state_after.as_deref().unwrap_or(&[]), + "seed {seed} step {step}: frozen open rewrote state" + ); + } + } + } + + impl<'a> Runner<'a> { + /// Reopen the store. The snapshot is taken BEFORE open so the + /// frozen path can prove nothing present before was lost and a + /// lawful refusal can roll the disk back entirely. fn reopen(&mut self, seed: u64, step: usize) { + let files_before = self.share_files(); + let state_before = fs::read(self.ndir().join("state")).ok(); self.store = None; match ShareStore::open( self.dir, @@ -2834,6 +3166,11 @@ mod tests { ) { Ok(store) => { self.store = Some(store); + self.assert_frozen_outcome(seed, step, &files_before, &state_before); + // Undo any permission fault the injected damage + // left: open repaired what it needed; the rest is + // the harness's job. + make_writable(&self.ndir()); self.resync(seed, step); // Open idempotence: while the store is not frozen, // open∘open is byte-identical with open — the second @@ -2861,24 +3198,84 @@ mod tests { } } Err(e) => { - // Two lawful refusals: (a) the recorded state itself - // is unreadable (a torn file is disk damage the store - // must never guess past), (b) the parsed record - // promises a receipted share whose file is gone. + // Lawful refusals: disk damage the store must not + // guess past — a torn or unparsable `state`, a + // receipted candidate whose share file is gone, an + // activated pin whose share is missing (the member + // promised a share it cannot serve), or a pin's + // proof lost with it (equivocation evidence must + // not be silently dropped either). let parsed = fs::read(self.ndir().join("state")) .ok() .and_then(|b| parse_state(&b)); - let lawful = match parsed { + let lawful = match &parsed { None => true, Some(p) => { - matches!(e, StoreError::Corrupt) - && p.candidate.is_some_and(|(d, sent)| { - sent && !share_path(&self.ndir(), &d).exists() + if let StoreError::Io(io) = &e { + // A read-only-dir (or similar) fault: + // the store may fail anywhere on the + // path it can no longer write. + matches!( + io.kind(), + std::io::ErrorKind::PermissionDenied + | std::io::ErrorKind::ReadOnlyFilesystem + ) + } else if !matches!(e, StoreError::Corrupt) { + false + } else if p.candidate.is_some_and(|(d, sent)| { + sent && (!share_path(&self.ndir(), &d).exists() + || !proof_path(&self.ndir(), &d).exists()) + }) { + true + } else { + // A recorded pin the store can no + // longer serve — its share is gone — + // or prove: its own proof or any + // ancestor's proof up to genesis was + // lost while the member was offline. + p.pin.is_some_and(|d| { + !share_path(&self.ndir(), &d).exists() + || !self.pin_is_provable(&d) }) + } } }; - assert!(lawful, "seed {seed} step {step}: reopen failed: {e:?}"); - self.dead = true; + assert!( + lawful, + "seed {seed} step {step}: reopen failed: {e:?} \ + parsed={:?} shares={:?} proofs={:?}", + parsed, + self.share_files() + .iter() + .map(|d| hex(d)) + .collect::>(), + fs::read_dir(self.ndir().join("proofs")) + .map(|rd| rd + .flatten() + .filter_map(|e| e.file_name().to_str().map(str::to_owned)) + .collect::>()) + .unwrap_or_default() + ); + // The store refused honestly; a real deployment + // recovers from backup. Roll the disk back to the + // pre-fault snapshot and continue the run. + if self.backup.is_some() { + self.restore_disk(); + self.recoveries += 1; + } + self.store = Some( + ShareStore::open( + self.dir, + &self.network, + self.signers.anchor().clone(), + Signers::did(1), + self.signers, + ) + .unwrap_or_else(|e| { + panic!("seed {seed} step {step}: restored disk refused: {e:?}") + }), + ); + self.resync(seed, step); } } } @@ -2929,7 +3326,10 @@ mod tests { rng: Rng(seed), snapshots: VecDeque::new(), transcript: 63, - dead: false, + executed: 0, + recoveries: 0, + ops: BTreeMap::new(), + backup: None, fork_allowed: seed % 4 != 3, }; let (signed, cert) = activated_genesis(runner.store.as_ref().unwrap(), signers); @@ -2944,9 +3344,8 @@ mod tests { runner.snapshot(); for step in 0..steps { - if runner.dead { - break; - } + // Every step executes a real op or a deliberate no-op; + // a lawful refusal is rolled back and the run continues. let frozen = runner.model.frozen; let pending = runner.model.pending; let fork_allowed = runner.fork_allowed; @@ -2954,12 +3353,14 @@ mod tests { match roll { // Fresh accept + install under the admissible id. 0..=21 if !frozen => { + runner.mark_op("accept_install"); if let Some(ceremony) = runner.accept(seed, step) { runner.install(seed, step, &ceremony); } } // Accept only (candidate window stays empty). 22..=29 if !frozen => { + runner.mark_op("accept"); runner.accept(seed, step); } // Replay or stale accept: a seen id whose context still @@ -2967,6 +3368,7 @@ mod tests { // install op proves it can never install); a contextually // stale id is refused outright. 30..=37 if !frozen => { + runner.mark_op("accept_stale"); let n = runner.model.ceremonies.len(); if n > 0 { let ceremony = @@ -2985,6 +3387,7 @@ mod tests { // Stale install: any non-admissible ceremony id — // including retired ones — must fail. 38..=41 if !frozen => { + runner.mark_op("install_stale"); let stale: Vec = runner .model .ceremonies @@ -3017,6 +3420,7 @@ mod tests { // if a crash revert moved the pin past it the id is // contextually stale and must be refused, never built. 44..=51 if !frozen && pending.is_none() => { + runner.mark_op("install_admissible"); if let Some((id, pred)) = runner.model.active { let pin = runner.model.pins.last().unwrap().digest(); let ceremony = runner @@ -3044,6 +3448,7 @@ mod tests { } // Rule 6: receipt only over a durable share. 52..=59 if !frozen && pending.is_some() => { + runner.mark_op("receipt"); runner .store() .mark_receipt_sent() @@ -3052,6 +3457,7 @@ mod tests { } // Verified activation: the only legal pin advance. 60..=73 if !frozen && pending.is_some() => { + runner.mark_op("activate"); let (digest, _) = pending.unwrap(); let signed = runner.model.signed[digest.as_bytes()].clone(); let cert = @@ -3069,6 +3475,17 @@ mod tests { runner.model.retired.insert(*old.digest().as_bytes()); } runner.model.pins.push(cert); + // The new pin may re-admit or still exclude this + // node — recompute, never carry stale. + runner.model.dropped = !runner + .model + .pins + .last() + .unwrap() + .body() + .members() + .iter() + .any(|cm| cm.node == Signers::did(1)); runner.model.pending = None; runner.model.active = None; } @@ -3077,6 +3494,7 @@ mod tests { // second genesis proof written to disk (discovered at // next open), or a live record_activation fork. 68..=70 if !frozen && fork_allowed => { + runner.mark_op("fork"); let live = runner.rng.pick(3) == 0; let genesis_fork = runner.model.pins.len() < 2 || runner.rng.pick(3) == 0; let (signed, cert, receipts, activation) = if genesis_fork { @@ -3102,8 +3520,17 @@ mod tests { let i = 1 + runner.rng.pick((runner.model.pins.len() - 1) as u64) as usize; let pred = runner.model.pins[i - 1].clone(); - let ceremony = - runner.model.ceremonies_by_pred[pred.digest().as_bytes()].clone(); + // A disk-discovered pin (member-drop) may have + // no locally-accepted ceremony to fork under — + // skip when the scheduler can't build one. + let Some(ceremony) = runner + .model + .ceremonies_by_pred + .get(pred.digest().as_bytes()) + .cloned() + else { + continue; + }; runner.transcript = runner.transcript.wrapping_add(1); let signed = sibling_successor(signers, &pred, &ceremony, runner.transcript); @@ -3144,11 +3571,13 @@ mod tests { // disappears before reopen — convergence to an earlier // crash-free point, never corruption. 71..=73 => { + runner.mark_op("proof_loss"); let proofs: Vec = fs::read_dir(runner.ndir().join("proofs")) .map(|rd| rd.flatten().map(|e| e.path()).collect()) .unwrap_or_default(); if !proofs.is_empty() { runner.store = None; + runner.save_disk(); let path = proofs[runner.rng.pick(proofs.len() as u64) as usize].clone(); let _ = fs::remove_file(path); @@ -3158,6 +3587,7 @@ mod tests { // Resurrection probe: a retired certificate must never // reinstall, even under the admissible ceremony id. 74..=77 if !frozen && runner.model.pending.is_none() => { + runner.mark_op("resurrect_probe"); let pin = runner.model.pins.last().map(|p| p.digest()); let candidates: Vec = runner .model @@ -3219,6 +3649,7 @@ mod tests { // Rule 10: a second opener is Busy and changes nothing // on disk. 78..=83 => { + runner.mark_op("busy"); let before = runner.disk_hash(); assert!(matches!( ShareStore::open( @@ -3237,14 +3668,76 @@ mod tests { ); } // Clean reopen. - 84..=88 => runner.reopen(seed, step), - // Crash replays and disk damage: revert `state` to a - // recent snapshot, delete a dealt marker, drop temp - // debris, truncate `state` to a torn prefix, delete a - // non-live share file (unrecorded disk damage), or just - // drop unclosed; every variant ends in a reopen + resync. + 84..=88 => { + runner.mark_op("reopen"); + runner.reopen(seed, step); + } + // Member-drop: a reshared certificate that no longer + // names this node arrives as activated proof. The pin + // must advance to it — the member's share is then owed + // to nobody and nothing about it is corruption. + 89..=91 if !frozen => { + runner.mark_op("member_drop"); + let pin = runner.model.pins.last().unwrap().clone(); + let mut stmt = + signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + runner.transcript = runner.transcript.wrapping_add(1).max(64); + let mut id_bytes = [0x70u8; 32]; + id_bytes[0] = 0x70 ^ runner.transcript; + id_bytes[31] = runner.transcript; + stmt.id = CeremonyId::new(id_bytes); + stmt.current = Some(pin.digest()); + stmt.epoch = pin.body().epoch().checked_next().unwrap(); + stmt.rekey = pin.body().rekey(); + stmt.recipients = (1..=3u16) + .map(|i| CeremonyRecipient { + index: MemberIndex::new(i).unwrap(), + node: Signers::did(i as u8 + 1), + hpke_public_key: [(i as u8) + 1; 32], + }) + .collect(); + let ceremony = AuthorizedCeremony::verify_for( + &pin, + &signers.statement(&stmt), + signers, + ) + .unwrap(); + let body = CertificateBody::successor( + &pin, + &ceremony, + &cert_tests::material(11, Threshold::new(2, 3).unwrap()).0, + &TranscriptHash::new([runner.transcript; 32]), + ) + .unwrap(); + let signed = cert::tests::sign(body, signers); + let cert = verify_successor(&pin, &signed, signers).unwrap(); + let receipts = receipts_for(&cert, signers); + let activation = activation_for(&cert, &receipts, signers); + runner + .model + .signed + .insert(*cert.digest().as_bytes(), signed.clone()); + fs::write( + proof_path(&runner.ndir(), &cert.digest()), + encode_proof(&signed, Some(&activation), &receipts), + ) + .unwrap(); + let old = pin.digest(); + runner.reopen(seed, step); + runner.model.retired.insert(*old.as_bytes()); + assert!( + runner.model.dropped || runner.model.frozen, + "seed {seed} step {step}: member-drop pin not adopted" + ); + } + // Crash replays and disk damage. The disk is backed up + // BEFORE the fault so a lawful refusal can roll it back + // and the run can continue (a real deployment restores + // from backup the same way). _ => { - match runner.rng.pick(6) { + runner.mark_op("crash"); + runner.save_disk(); + match runner.rng.pick(9) { 0 if !runner.snapshots.is_empty() => { runner.store = None; let i = runner.rng.pick(runner.snapshots.len() as u64) as usize; @@ -3278,17 +3771,6 @@ mod tests { fs::write(&state, &bytes[..bytes.len() / 2]).unwrap(); } } - assert!( - ShareStore::open( - dir, - &network, - signers.anchor().clone(), - Signers::did(1), - signers - ) - .is_err(), - "seed {seed} step {step}: torn state opened" - ); } 4 => { // Delete a share file the state does not @@ -3324,6 +3806,68 @@ mod tests { ); } } + 5 => { + // The pin's share file is lost: the store + // promised a share it can no longer serve — + // open must refuse, then restore. + runner.store = None; + if let Some(p) = runner.model.pins.last() { + let _ = + fs::remove_file(share_path(&runner.ndir(), &p.digest())); + } + if runner.model.frozen { + runner.model.frozen_files = Some(runner.share_files()); + } + } + 6 => { + // The pin's proof AND share are lost + // together: the activated record cannot be + // proven or served — refuse, then restore. + runner.store = None; + if let Some(p) = runner.model.pins.last() { + let _ = + fs::remove_file(share_path(&runner.ndir(), &p.digest())); + let _ = + fs::remove_file(proof_path(&runner.ndir(), &p.digest())); + } + if runner.model.frozen { + runner.model.frozen_files = Some(runner.share_files()); + } + } + #[cfg(unix)] + 7 => { + // Read-only directories: writes fail + // anywhere on the path; a refusal is lawful + // and the disk rolls back. + use std::os::unix::fs::PermissionsExt; + runner.store = None; + for sub in [ + runner.ndir(), + runner.ndir().join("proofs"), + runner.ndir().join("dealt"), + ] { + let _ = fs::set_permissions( + &sub, + fs::Permissions::from_mode(0o500), + ); + } + } + 8 => { + // Compound damage: `state` reverted AND the + // pin's share file lost — whichever the + // store sees first, the outcome stays + // lawful. + runner.store = None; + if !runner.snapshots.is_empty() { + let i = runner.rng.pick(runner.snapshots.len() as u64) as usize; + let bytes = runner.snapshots[i].clone(); + fs::write(runner.ndir().join("state"), &bytes).unwrap(); + } + if let Some(p) = runner.model.pins.last() { + let _ = + fs::remove_file(share_path(&runner.ndir(), &p.digest())); + } + } _ => runner.store = None, } runner.reopen(seed, step); @@ -3332,7 +3876,15 @@ mod tests { runner.assert_boundary(seed, step); runner.snapshot(); } - eprintln!("model seed {seed}: {steps} steps ok"); + eprintln!( + "model seed {seed}: {} steps executed ({} recoveries) ops={:?} frozen={}", + runner.executed, runner.recoveries, runner.ops, runner.model.frozen + ); + assert!( + runner.executed >= steps * 2 / 3, + "seed {seed}: only {} of {steps} steps executed", + runner.executed + ); } /// Legacy `state` files must be handled explicitly: anything but the @@ -3362,9 +3914,21 @@ mod tests { &signers ) .is_err()); + // Version 2 (no predecessor on retired entries). + let mut v2 = encode_state(&state); + v2[0] = 2; + fs::write(netdir(&dir, &network).join("state"), &v2).unwrap(); + assert!(ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers + ) + .is_err()); // Unknown future version. let mut future = encode_state(&state); - future[0] = 3; + future[0] = 4; fs::write(netdir(&dir, &network).join("state"), &future).unwrap(); assert!(ShareStore::open( &dir, @@ -3599,4 +4163,304 @@ mod tests { assert_eq!(store.pinned().map(|p| p.digest()), Some(cert_g.digest())); assert!(!share_path(&netdir(&dir, &signers.anchor().network), &digest_x).exists()); } + + /// P3-1: an ACTIVATED proof for a retired certificate arriving on disk + /// (peer sync, operator restore) while the store is closed must freeze + /// the next open — the live path already refuses the same proof as + /// `Retired`; reopening must never R1-erase the live pin share. + /// Details: TC-2221 + #[test] + fn activated_proof_of_retired_candidate_freezes_reopen() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let (s_x, digest_x, receipts_x, activation_x, cert_g) = { + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + let (s_x, mut raw_x, c_x) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_x).unwrap(); + let digest_x = store + .install_candidate( + &s_x, + mk_candidate(raw_x.remove(0), &s_x.body), + &c_x, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); + // A fresh ceremony retires X (C1); the share is erased. + let mut stmt_y = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_y.id = CeremonyId::new([0x63; 32]); + stmt_y.current = Some(cert_g.digest()); + stmt_y.epoch = cert_g.body().epoch().checked_next().unwrap(); + stmt_y.rekey = cert_g.body().rekey(); + let ceremony_y = + AuthorizedCeremony::verify_for(&cert_g, &signers.statement(&stmt_y), &signers) + .unwrap(); + store.accept_ceremony(&ceremony_y).unwrap(); + assert!(store.retired().iter().any(|r| r.digest == digest_x)); + let cert_x = verify_successor(&cert_g, &s_x, &signers).unwrap(); + let receipts = receipts_for(&cert_x, &signers); + let activation = activation_for(&cert_x, &receipts, &signers); + (s_x, digest_x, receipts, activation, cert_g) + }; + // An activated X proof lands on disk while the store is closed — + // indistinguishable from peer sync or an operator restore. + fs::write( + proof_path(&netdir(&dir, &network), &digest_x), + encode_proof(&s_x, Some(&activation_x), &receipts_x), + ) + .unwrap(); + // Reopen: the chain now runs through a durably retired digest — the + // store freezes; it must NOT advance the pin to X and erase G's + // still-live share. + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(store.equivocation()); + assert!(matches!( + store.current_share(), + Err(StoreError::Equivocation) + )); + assert_eq!(store.pinned().map(|p| p.digest()), Some(cert_g.digest())); + assert!(share_path(&netdir(&dir, &network), &cert_g.digest()).exists()); + } + + /// P3-2: a `*.tmp-*` name that is a HARDLINK — not a symlink — must be + /// unlinked, never zeroed: writing zeros through the inode rewrites the + /// file it shares storage with outside the store. Details: TC-2221 + #[test] + #[cfg(unix)] + fn hardlinked_temp_name_is_unlinked_never_zeroed() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let victim = dir.join("victim-share-backup"); + fs::write(&victim, [0x5Au8; 64]).unwrap(); + { + let _store = open(&dir, &signers); + activated_genesis(&_store, &signers); + let link = netdir(&dir, &network).join("state.tmp-7-7"); + fs::hard_link(&victim, &link).unwrap(); + } + // The sweep removes the link; the victim keeps its bytes. + let _store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(!netdir(&dir, &network).join("state.tmp-7-7").exists()); + assert_eq!(fs::read(&victim).unwrap(), vec![0x5Au8; 64]); + } + + /// P3-3: a second ACTIVATED genesis presented to the live store is the + /// same equivocation evidence reopening freezes on — persist it and + /// freeze; do not silently drop it. Details: TC-2221 + #[test] + fn live_second_activated_genesis_persists_and_freezes() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + // A second genesis certificate over the same ceremony. + let ceremony = signers.genesis(); + let (public, _shares) = cert_tests::material(17, Threshold::new(2, 3).unwrap()); + let body = CertificateBody::genesis( + &signers.anchor(), + &ceremony, + &public, + &TranscriptHash::new([0xF2; 32]), + ) + .unwrap(); + let signed = cert::tests::sign(body, &signers); + let cert_g2 = verify_certificate(&signers.anchor(), &signed, &signers).unwrap(); + let receipts = receipts_for(&cert_g2, &signers); + let activation = activation_for(&cert_g2, &receipts, &signers); + assert!(matches!( + store.record_activation(&signed, &activation, &receipts, &signers), + Err(StoreError::Equivocation) + )); + assert!(store.equivocation()); + // The evidence is durable and the pin does not move. + assert!(proof_path(&netdir(&dir, &network), &cert_g2.digest()).exists()); + drop(store); + let store = ShareStore::open( + &dir, + &network, + signers.anchor().clone(), + Signers::did(1), + &signers, + ) + .unwrap(); + assert!(store.equivocation()); + assert_eq!(store.pinned().map(|p| p.digest()), Some(cert_g.digest())); + } + + /// P3-4a: retirement history is audit data — it survives a pin advance + /// (nothing is silently dropped), yet a retired digest still cannot + /// come back. Compaction only evicts provably-dead entries when the + /// list nears the bound. Details: TC-2221 + #[test] + fn pin_advance_keeps_retirement_audit_without_resurrecting() { + let signers = Signers::new(); + let dir = tmpdir(); + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + // C1-retire candidate X under the genesis pin. + let (s_x, mut raw_x, c_x) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_x).unwrap(); + let digest_x = store + .install_candidate( + &s_x, + mk_candidate(raw_x.remove(0), &s_x.body), + &c_x, + &signers, + ) + .unwrap(); + let mut stmt_y = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_y.id = CeremonyId::new([0x64; 32]); + stmt_y.current = Some(cert_g.digest()); + stmt_y.epoch = cert_g.body().epoch().checked_next().unwrap(); + stmt_y.rekey = cert_g.body().rekey(); + let ceremony_y = + AuthorizedCeremony::verify_for(&cert_g, &signers.statement(&stmt_y), &signers).unwrap(); + store.accept_ceremony(&ceremony_y).unwrap(); + assert!(store.retired().iter().any(|r| r.digest == digest_x)); + // Activate successor Y under the same pin (reshare preserves mpk, + // so Y reuses the genesis key material with a fresh transcript). + let s_y = sibling_successor(&signers, &cert_g, &ceremony_y, 0x71); + let (_pub_y, mut raw_y) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); + store + .install_candidate( + &s_y, + mk_candidate(raw_y.remove(0), &s_y.body), + &ceremony_y, + &signers, + ) + .unwrap(); + store.mark_receipt_sent().unwrap(); + let cert_y = verify_successor(&cert_g, &s_y, &signers).unwrap(); + let receipts = receipts_for(&cert_y, &signers); + let activation = activation_for(&cert_y, &receipts, &signers); + store + .record_activation(&s_y, &activation, &receipts, &signers) + .unwrap(); + // Audit survives the pin advance; the record also carries the + // predecessor it was retired against (needed by bounded pruning). + let entry = store + .retired() + .into_iter() + .find(|r| r.digest == digest_x) + .expect("retirement audit must survive a pin advance"); + assert_eq!(entry.predecessor, Some(cert_g.digest())); + // Reinstalling X is refused either way: the retirement record when + // reachable, stale lineage after the pin moved. + assert!(store + .install_candidate( + &s_x, + mk_candidate(raw_x.remove(0), &s_x.body), + &ceremony_y, + &signers + ) + .is_err()); + } + + /// P3-4c: at the bound, dead retirement records are compacted away + /// while every reachable one is kept; a list of only reachable records + /// reports overflow so the operation is refused. Details: TC-2221 + #[test] + fn compact_retired_drops_only_dead_entries_at_the_bound() { + let pin = CertDigest::new([0x10; 32]); + let mut state = State { + pin: Some(pin), + candidate: None, + active_ceremony: None, + seen_ceremonies: Vec::new(), + retired: Vec::new(), + seq: 0, + }; + // One reachable entry + a pile of dead ones. + record_retired(&mut state, CertDigest::new([0xAA; 32]), Some(pin)); + for i in 0..(MAX_RETIRED + 8) { + let mut bytes = [0u8; 32]; + bytes[31] = (i % 250) as u8; + bytes[30] = (i / 250) as u8; + record_retired( + &mut state, + CertDigest::new(bytes), + Some(CertDigest::new([0xEE; 32])), + ); + } + assert!(compact_retired(&mut state, Some(pin))); + assert_eq!(state.retired.len(), 1); + assert_eq!(state.retired[0].predecessor, Some(pin)); + // Nothing but reachable entries left → the bound reports overflow. + let mut full = state.clone(); + full.retired = (0..=MAX_RETIRED) + .map(|i| { + let mut bytes = [0u8; 32]; + bytes[31] = (i % 250) as u8; + bytes[30] = (i / 250) as u8; + bytes[29] = 0xBB; + RetiredGeneration { + digest: CertDigest::new(bytes), + predecessor: Some(pin), + erased_at: i as u64, + } + }) + .collect(); + assert!(!compact_retired(&mut full, Some(pin))); + } + + /// P3-4b: the retired and seen-ceremony lists are bounded: decoding a + /// state file claiming absurd lengths refuses rather than allocating, + /// and the store itself never writes past its own caps. Details: TC-2221 + #[test] + fn state_lists_are_bounded_at_decode() { + // More retired entries than the decoder bound → refuse, never parse. + let mut e = Encoder::new(); + e.u8(3); + e.u8(0); // pin + e.u8(0); // candidate + e.u8(0); // active ceremony + e.u32(0); // seen + e.u32(1_000_001); // retired_count above the bound + for _ in 0..100 { + e.fixed(&[0xAB; 32]); + e.u8(0); + e.u64(0); + } + e.u64(0); + assert!(decode_state(&e.finish()).is_err()); + let mut e = Encoder::new(); + e.u8(3); + e.u8(0); + e.u8(0); + e.u8(0); + e.u32(1_000_001); // seen above the bound + e.u32(0); + e.u64(0); + assert!(decode_state(&e.finish()).is_err()); + } } From 887d6ae87ee907416ab2428716c479894ea9889f Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 12:13:59 +0000 Subject: [PATCH 7/8] fix(threshold): record the equivocation freeze durably so losing fork evidence cannot thaw the store; model resync reconciles rollback timelines; Details: TC-2221 --- tinycloud-threshold/README.md | 2 +- tinycloud-threshold/SPEC.md | 2 +- tinycloud-threshold/src/store.rs | 120 +++++++++++++++++++++++++------ 3 files changed, 102 insertions(+), 22 deletions(-) diff --git a/tinycloud-threshold/README.md b/tinycloud-threshold/README.md index 46fbd38b..2c62e4b4 100644 --- a/tinycloud-threshold/README.md +++ b/tinycloud-threshold/README.md @@ -18,7 +18,7 @@ The crate is a workspace member with `publish = false`. No other workspace crate - Genesis and re-key DKG (GJKR New-DKG) over owner-closed rounds, the re-key under an owner-authorized `re-key` ceremony pinned to the current certificate with the genesis fault budget: consume-self member states for dealing, complaints, answers, Feldman extraction, the common R5 round and R6 reconstruction, and `dkg_verify_transcript`, which recomputes the outcome from the raw signed round records. - Redistribution for reshare and refresh over owner-closed rounds R1–R3: Feldman dealings from the current certificate's members to an authorized cohort with a new threshold, complaints and answers, deterministic selection of the first t eligible dealers, and `reshare_verify_transcript`. The master public key is unchanged; the output is a candidate for the successor certificate. - A real-ceremony chain test: genesis 2-of-3, reshare to 3-of-4 and back to 2-of-3, refresh, re-key, and further reshares and re-keys, with a certificate, readiness receipts and activation at every generation and each retired generation's shares refused in the next context. -- A durable share store: `CandidateShare::to_bytes`/`from_bytes` (canonical bytes; the codec guards its own secret intermediates), one file per generation under a per-network directory, an exclusive per-network open lock, install before any readiness receipt, a pin recomputed from activations verified on every open, and retirement whose `state` commit always precedes the unlink. +- A durable share store: `CandidateShare::to_bytes`/`from_bytes` (canonical bytes; the codec guards its own secret intermediates), one file per generation under a per-network directory, an exclusive per-network open lock, install before any readiness receipt, a pin recomputed from activations verified on every open, retirement whose `state` commit always precedes the unlink, and a durably-recorded freeze on activated equivocation that survives the loss of the fork evidence itself. `VerifiedCertificate` supplies `KeyContext` and `PublicKeySet`; `KeyShare::bind` validates a candidate against its certificate. Raw candidate-scalar construction remains crate-private, and the dealer exists only under `#[cfg(test)]`. Redistribution does not erase earlier shares; old and new shares carry different key contexts. The `store` module implements the file-volume durable share store and retirement; node-runtime wiring remains a separate slice. Their contracts and the implemented canonical byte layouts are recorded in [SPEC.md](SPEC.md). diff --git a/tinycloud-threshold/SPEC.md b/tinycloud-threshold/SPEC.md index fdc07dff..dc8adac4 100644 --- a/tinycloud-threshold/SPEC.md +++ b/tinycloud-threshold/SPEC.md @@ -556,6 +556,6 @@ The `store` module keeps one member's share material for one network under `, retired: Vec, seq: u64, + /// Activated equivocation froze the store; terminal and durable — it + /// outlives the loss of the fork evidence itself. + frozen: bool, } fn encode_state(state: &State) -> Vec { @@ -159,6 +162,7 @@ fn encode_state(state: &State) -> Vec { e.u64(retired.erased_at); } e.u64(state.seq); + e.u8(u8::from(state.frozen)); e.finish() } @@ -223,6 +227,7 @@ fn decode_state(bytes: &[u8]) -> Result { }); } let seq = d.u64()?; + let frozen = d.u8()? == 1; d.finish()?; Ok(State { pin, @@ -231,6 +236,7 @@ fn decode_state(bytes: &[u8]) -> Result { seen_ceremonies, retired, seq, + frozen, }) } @@ -636,6 +642,10 @@ impl ShareStore { Err(e) if e.kind() == std::io::ErrorKind::NotFound => State::default(), Err(e) => return Err(e.into()), }; + // The decoded record before any in-memory bookkeeping: if a freeze + // is discovered the persist write must carry only the flag flip — + // never incidental mutations like orphan dealt-marker recording. + let decoded_state = state.clone(); let mut state_dirty = false; // Orphan dealt markers: a marker whose id was never recorded consumed @@ -684,7 +694,10 @@ impl ShareStore { // pin at the fork. let mut verified: BTreeMap = BTreeMap::new(); let mut chain: Vec = Vec::new(); - let mut equivocation = false; + // A durably-frozen store stays frozen even when the fork evidence + // itself is later lost; the freeze is terminal by record, not by + // recomputation. + let mut equivocation = state.frozen; // Every flagged activation is verified before it can extend the pin // or count toward a fork; an activation flag alone proves nothing. let mut geneses: Vec = Vec::new(); @@ -783,10 +796,14 @@ impl ShareStore { // A recorded pin must be a verified activated certificate: either on // the surviving chain or an equivocated branch descendant. Anything - // else is corruption. - if let Some(recorded) = state.pin { - if !verified.contains_key(&recorded) { - return Err(StoreError::Corrupt); + // else is corruption. A durably frozen store skips the check: the + // freeze is terminal and the pin's own proof may legitimately be + // the evidence that was lost. + if !state.frozen { + if let Some(recorded) = state.pin { + if !verified.contains_key(&recorded) { + return Err(StoreError::Corrupt); + } } } @@ -803,6 +820,15 @@ impl ShareStore { { equivocation = true; } + // Persist a newly-discovered freeze before honoring it: the freeze + // is terminal, so it must outlive the evidence that produced it. + if equivocation && !state.frozen { + state = State { + frozen: true, + ..decoded_state + }; + write_atomic(&network_dir.join("state"), &encode_state(&state))?; + } // Candidate reconciliation and R1/C2/S1 erasure run ONLY on an // unequivocal chain. Once activated equivocation is on disk the @@ -898,7 +924,12 @@ impl ShareStore { if verified.contains_key(&digest) { // Superseded activated generation (R1). if !state.retired.iter().any(|r| r.digest == digest) { - let predecessor = proofs[&digest].body.previous().copied(); + let predecessor = proofs + .get(&digest) + .ok_or(StoreError::Corrupt)? + .body + .previous() + .copied(); record_retired(&mut state, digest, predecessor); state.pin = computed_pin; write_atomic(&network_dir.join("state"), &encode_state(&state))?; @@ -1151,6 +1182,8 @@ impl ShareStore { &proof_path(&self.network_dir, &fork.digest()), &encode_proof(signed, Some(activation), receipts), )?; + inner.state.frozen = true; + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; inner.equivocation = true; return Err(StoreError::Equivocation); } @@ -1165,6 +1198,8 @@ impl ShareStore { &proof_path(&self.network_dir, &root.digest()), &encode_proof(signed, Some(activation), receipts), )?; + inner.state.frozen = true; + write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; inner.equivocation = true; return Err(StoreError::Equivocation); } @@ -2615,6 +2650,7 @@ mod tests { take(8, &mut cur)?; // erased_at } take(8, &mut cur)?; // seq + take(1, &mut cur)?; // frozen (v3) if cur != bytes.len() { return None; } @@ -2785,9 +2821,15 @@ mod tests { // pin, candidate (with its receipt flag, I6) and the // admissible ceremony are exactly what the disk records. if let Some(p) = &parsed { + // Under freeze the disk pin is the recorded freeze + // point — the model's own tip may project further. + let expected_pin = if m.frozen { + m.frozen_pin + } else { + m.pins.last().map(|p| p.digest()) + }; assert_eq!( - p.pin, - m.pins.last().map(|p| p.digest()), + p.pin, expected_pin, "seed {seed} step {step}: disk pin diverged from model" ); assert_eq!( @@ -2799,11 +2841,10 @@ mod tests { "seed {seed} step {step}: disk admissible ceremony diverged" ); // Rule 7 tail: `retired` never names a live generation. - let pin_d = m.pins.last().map(|p| p.digest()); let pend_d = m.pending.map(|(d, _)| d); for r in &p.retired { assert!( - Some(CertDigest::new(*r)) != pin_d + Some(CertDigest::new(*r)) != expected_pin && Some(CertDigest::new(*r)) != pend_d, "seed {seed} step {step}: retired names a live share" ); @@ -2825,9 +2866,12 @@ mod tests { ); // The recorded pin stands — it may be an older // generation than the model's tip when a late fault - // moved the model past the durable write. + // moved the model past the durable write. The durable + // record is the truth: when a fault also destroyed the + // pin's own proof there is no cert to expose, and + // `pinned()` can only offer the chain tip. assert_eq!( - store.pinned().map(|p| p.digest()), + parsed.and_then(|p| p.pin), m.frozen_pin, "seed {seed} step {step}: frozen pin moved" ); @@ -3025,6 +3069,12 @@ mod tests { let parsed = fs::read(self.ndir().join("state")) .ok() .and_then(|b| parse_state(&b)); + // A rollback may land on a backup that predates a recorded + // retirement: the disk's retired set replaces the model's, + // matching what the restored store actually refuses. + if let Some(p) = &parsed { + self.model.retired = p.retired.clone(); + } let store = self.store.as_ref().unwrap(); if store.equivocation() { // The freeze point is recorded once: everything after @@ -3041,14 +3091,28 @@ mod tests { .position(|p| p.digest() == pin.digest()); if let Some(pos) = pos { self.model.pins.truncate(pos + 1); + } else { + // The recorded pin is off the model chain — a + // rollback to an older backup that already + // named a chain the model never built. Keep + // the longest prefix whose tip is still on + // disk; nothing can extend the foreign tip. + let keep = self + .model + .pins + .iter() + .position(|p| !proof_path(&self.ndir(), &p.digest()).exists()) + .unwrap_or(self.model.pins.len()); + self.model.pins.truncate(keep); } } None => self.model.pins.clear(), } - if self.model.frozen_pin.is_none() { - self.model.frozen_pin = store.pinned().map(|p| p.digest()); - self.model.frozen_files = Some(self.share_files()); - } + // A rollback can land before the old freeze — the + // freeze point is per-timeline: re-record it on every + // frozen resync, not only the first. + self.model.frozen_pin = parsed.as_ref().and_then(|p| p.pin); + self.model.frozen_files = Some(self.share_files()); self.model.frozen = true; // Frozen resync still reads what the disk records: the // state file is the durable truth for pending/active @@ -3057,7 +3121,15 @@ mod tests { self.model.pending = parsed.as_ref().and_then(|p| p.candidate); return; } + // Thaw is only honest when the operator's rollback removed + // the freeze's basis — restore to a pre-fork world. Reset + // the model's freeze bookkeeping for the new timeline. let m = &mut self.model; + if m.frozen { + m.frozen = false; + m.frozen_pin = None; + m.frozen_files = None; + } match store.pinned() { Some(pin) => { if let Some(pos) = m.pins.iter().position(|p| p.digest() == pin.digest()) { @@ -3138,11 +3210,18 @@ mod tests { "seed {seed} step {step}: frozen open erased a share" ); } + // `state` may differ only in the trailing durable-freeze + // byte — the one write an open that newly discovers + // equivocation is required to make. let state_after = fs::read(self.ndir().join("state")).ok(); - if let Some(before) = state_before { - assert_eq!( - before.as_slice(), - state_after.as_deref().unwrap_or(&[]), + if let (Some(before), Some(after)) = (state_before, state_after) { + let freeze_write = before.len() == after.len() + && before[..before.len().saturating_sub(1)] + == after[..after.len().saturating_sub(1)] + && *before.last().unwrap_or(&1) == 0 + && *after.last().unwrap_or(&0) == 1; + assert!( + *before == after || freeze_write, "seed {seed} step {step}: frozen open rewrote state" ); } @@ -4399,6 +4478,7 @@ mod tests { seen_ceremonies: Vec::new(), retired: Vec::new(), seq: 0, + frozen: false, }; // One reachable entry + a pile of dead ones. record_retired(&mut state, CertDigest::new([0xAA; 32]), Some(pin)); From 65bcded9af1dd907212017f13d482644b87b4bed Mon Sep 17 00:00:00 2001 From: "tinycloud-bot[bot]" <348376441+tinycloud-bot[bot]@users.noreply.github.com> Date: Tue, 10 Nov 2026 13:11:10 +0000 Subject: [PATCH 8/8] fix(threshold): gate receipt flag on equivocation, freeze-state writes assign after durable commit, wide seeded-harness ceremony ids, compact-bounded recovery retirement; Details: TC-2221 --- tinycloud-threshold/SPEC.md | 4 +- tinycloud-threshold/src/store.rs | 190 ++++++++++++++++++++++++++----- 2 files changed, 162 insertions(+), 32 deletions(-) diff --git a/tinycloud-threshold/SPEC.md b/tinycloud-threshold/SPEC.md index dc8adac4..cd939205 100644 --- a/tinycloud-threshold/SPEC.md +++ b/tinycloud-threshold/SPEC.md @@ -556,6 +556,6 @@ The `store` module keeps one member's share material for one network under `, + pin: Option, +) { + if state.retired.len() >= MAX_RETIRED { + state.retired.retain(|r| r.predecessor == pin); + } + record_retired(state, digest, predecessor); +} + /// Compact the retirement list only when it nears the bound: an entry is /// reachable — and therefore must be kept — only while the pin equals its /// recorded predecessor; dead entries are dropped first. Reachable entries @@ -879,7 +895,12 @@ impl ShareStore { if lost { // Commit the retirement first; the unlink is owed // work a crash simply re-discovers as S1. - record_retired(&mut state, candidate.digest, candidate.predecessor); + recover_retired( + &mut state, + candidate.digest, + candidate.predecessor, + computed_pin, + ); state.candidate = None; state.pin = computed_pin; write_atomic(&network_dir.join("state"), &encode_state(&state))?; @@ -930,7 +951,7 @@ impl ShareStore { .body .previous() .copied(); - record_retired(&mut state, digest, predecessor); + recover_retired(&mut state, digest, predecessor, computed_pin); state.pin = computed_pin; write_atomic(&network_dir.join("state"), &encode_state(&state))?; } @@ -1135,6 +1156,11 @@ impl ShareStore { /// `receipt_sent` with a missing share file refuses to open. pub fn mark_receipt_sent(&self) -> Result<(), StoreError> { let mut inner = self.inner.lock().map_err(|_| StoreError::Corrupt)?; + // A frozen store accepts nothing new and mutates nothing — the + // receipt flag is part of the record the freeze preserves. + if inner.equivocation { + return Err(StoreError::Equivocation); + } let mut next = inner.state.clone(); let Some(candidate) = &mut next.candidate else { return Err(StoreError::Corrupt); @@ -1182,8 +1208,12 @@ impl ShareStore { &proof_path(&self.network_dir, &fork.digest()), &encode_proof(signed, Some(activation), receipts), )?; - inner.state.frozen = true; - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + // Mutate only after the durable write exists: a failed + // write leaves memory identical to disk. + let mut next = inner.state.clone(); + next.frozen = true; + write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; + inner.state = next; inner.equivocation = true; return Err(StoreError::Equivocation); } @@ -1198,8 +1228,10 @@ impl ShareStore { &proof_path(&self.network_dir, &root.digest()), &encode_proof(signed, Some(activation), receipts), )?; - inner.state.frozen = true; - write_atomic(&self.network_dir.join("state"), &encode_state(&inner.state))?; + let mut next = inner.state.clone(); + next.frozen = true; + write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; + inner.state = next; inner.equivocation = true; return Err(StoreError::Equivocation); } @@ -1215,13 +1247,12 @@ impl ShareStore { if inner.state.retired.iter().any(|r| r.digest == digest) { return Err(StoreError::Retired); } - write_atomic( - &proof_path(&self.network_dir, &digest), - &encode_proof(signed, Some(activation), receipts), - )?; // `state` commits every removal BEFORE the unlink, so a crash is // recovered as "erasure owed", never as a missing promised share; // the in-memory record moves only after its durable copy exists. + // The candidate next-state is built before the proof write so a + // Capacity refusal leaves disk untouched — a refused call must not + // durably advance the chain on reopen. let mut next = inner.state.clone(); let mut erasures: Vec = Vec::new(); // C2: a pending candidate that lost to this activation is erased; the @@ -1253,6 +1284,10 @@ impl ShareStore { // Ceremonies accepted against the old pin are stale: the next // admissible ceremony must be accepted afresh for this pin. next.active_ceremony = None; + write_atomic( + &proof_path(&self.network_dir, &digest), + &encode_proof(signed, Some(activation), receipts), + )?; write_atomic(&self.network_dir.join("state"), &encode_state(&next))?; inner.state = next; inner.pin = Some(verified.clone()); @@ -1484,22 +1519,26 @@ mod tests { (signed, cert) } + /// A transcript hash carrying a wide counter: never collides, so a + /// long seeded run always exercises a fresh certificate. + fn transcript_hash(n: u64) -> TranscriptHash { + let mut b = [0u8; 32]; + b[..8].copy_from_slice(&n.to_be_bytes()); + TranscriptHash::new(b) + } + /// A successor certificate over the same ceremony and shares, distinct /// only by transcript; used where two candidates compete. fn sibling_successor( signers: &Signers, prev: &VerifiedCertificate, ceremony: &AuthorizedCeremony, - transcript: u8, + transcript: u64, ) -> SignedCertificate { let (public, _shares) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); - let body = CertificateBody::successor( - prev, - ceremony, - &public, - &TranscriptHash::new([transcript; 32]), - ) - .unwrap(); + let body = + CertificateBody::successor(prev, ceremony, &public, &transcript_hash(transcript)) + .unwrap(); cert::tests::sign(body, signers) } @@ -2683,7 +2722,7 @@ mod tests { /// Whether this seed may inject fork evidence; a quarter of /// seeds never fork so most runs exercise long unfrozen chains. fork_allowed: bool, - transcript: u8, + transcript: u64, } impl<'a> Runner<'a> { @@ -2984,10 +3023,10 @@ mod tests { let mut stmt = self .signers .body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); - self.transcript = self.transcript.wrapping_add(1).max(64); - let mut id_bytes = [0x60u8; 32]; - id_bytes[0] = 0x60 ^ (self.transcript); - id_bytes[31] = self.transcript; + self.transcript = self.transcript.wrapping_add(1); + let mut id_bytes = [0u8; 32]; + id_bytes[..8].copy_from_slice(&self.transcript.to_be_bytes()); + id_bytes[8] = 0x60; stmt.id = CeremonyId::new(id_bytes); stmt.current = Some(pin.digest()); stmt.epoch = pin.body().epoch().checked_next().unwrap(); @@ -3001,14 +3040,14 @@ mod tests { &mut self, ceremony: &AuthorizedCeremony, ) -> (SignedCertificate, KeyShare) { - self.transcript = self.transcript.wrapping_add(1).max(64); + self.transcript = self.transcript.wrapping_add(1); let pin = self.model.pins.last().unwrap(); let (public, mut shares) = cert_tests::material(11, Threshold::new(2, 3).unwrap()); let body = match CertificateBody::successor( pin, ceremony, &public, - &TranscriptHash::new([self.transcript; 32]), + &transcript_hash(self.transcript), ) { Ok(body) => body, Err(e) => panic!( @@ -3760,10 +3799,10 @@ mod tests { let pin = runner.model.pins.last().unwrap().clone(); let mut stmt = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); - runner.transcript = runner.transcript.wrapping_add(1).max(64); - let mut id_bytes = [0x70u8; 32]; - id_bytes[0] = 0x70 ^ runner.transcript; - id_bytes[31] = runner.transcript; + runner.transcript = runner.transcript.wrapping_add(1); + let mut id_bytes = [0u8; 32]; + id_bytes[..8].copy_from_slice(&runner.transcript.to_be_bytes()); + id_bytes[8] = 0x70; stmt.id = CeremonyId::new(id_bytes); stmt.current = Some(pin.digest()); stmt.epoch = pin.body().epoch().checked_next().unwrap(); @@ -3785,7 +3824,7 @@ mod tests { &pin, &ceremony, &cert_tests::material(11, Threshold::new(2, 3).unwrap()).0, - &TranscriptHash::new([runner.transcript; 32]), + &transcript_hash(runner.transcript), ) .unwrap(); let signed = cert::tests::sign(body, signers); @@ -4032,6 +4071,20 @@ mod tests { run_seed(&dir, &signers, seed, 220); } } + + /// The extended sweep: fresh 64-wide ceremony ids must survive the + /// whole run — a u8 counter wrapped at step ~290 and a "fresh" + /// accept returned Replay (reviewer seed 67). Release-only: this + /// run is the evidence for the widened id space. Details: TC-2221 + #[test] + #[cfg(not(debug_assertions))] + fn seeded_model_invariants_extended_sweep() { + for seed in 64..128u64 { + let signers = Signers::new(); + let dir = tmpdir(); + run_seed(&dir, &signers, seed, 400); + } + } } /// P1-a: a frozen reopen keeps the recorded verified pin and erases @@ -4131,6 +4184,83 @@ mod tests { .exists()); } + /// A frozen store refuses `mark_receipt_sent` and leaves `state` + /// byte-identical — the receipt flag is part of the record the freeze + /// preserves. Details: TC-2221 + #[test] + fn frozen_store_refuses_receipt_flag_and_keeps_state() { + let signers = Signers::new(); + let dir = tmpdir(); + let network = signers.anchor().network.clone(); + let store = open(&dir, &signers); + let (_signed_g, cert_g) = activated_genesis(&store, &signers); + let (s_a, mut raw_a, c_a) = cert_tests::successor( + &signers, + &cert_g, + CeremonyKind::Reshare, + Threshold::new(2, 3).unwrap(), + 11, + ); + store.accept_ceremony(&c_a).unwrap(); + store + .install_candidate( + &s_a, + mk_candidate(raw_a.remove(0), &s_a.body), + &c_a, + &signers, + ) + .unwrap(); + // Activate A so the pin moves; a candidate pending under A still + // accepts a receipt flag, which the freeze must refuse. + let cert_a = verify_successor(&cert_g, &s_a, &signers).unwrap(); + let receipts_a = receipts_for(&cert_a, &signers); + let activation_a = activation_for(&cert_a, &receipts_a, &signers); + store + .record_activation(&s_a, &activation_a, &receipts_a, &signers) + .unwrap(); + let mut stmt_z = signers.body(CeremonyKind::Reshare, Threshold::new(2, 3).unwrap()); + stmt_z.id = CeremonyId::new([0x62; 32]); + stmt_z.current = Some(cert_a.digest()); + stmt_z.epoch = cert_a.body().epoch().checked_next().unwrap(); + stmt_z.rekey = cert_a.body().rekey(); + let c_z = + AuthorizedCeremony::verify_for(&cert_a, &signers.statement(&stmt_z), &signers).unwrap(); + let body_z = CertificateBody::successor( + &cert_a, + &c_z, + &cert_tests::material(11, Threshold::new(2, 3).unwrap()).0, + &TranscriptHash::new([0x5a; 32]), + ) + .unwrap(); + let s_z = cert::tests::sign(body_z, &signers); + let mut raw_z = cert_tests::material(11, Threshold::new(2, 3).unwrap()).1; + store.accept_ceremony(&c_z).unwrap(); + store + .install_candidate( + &s_z, + mk_candidate(raw_z.remove(0), &s_z.body), + &c_z, + &signers, + ) + .unwrap(); + // Fork evidence: an activated sibling of A under the genesis pin. + let s_fork = sibling_successor(&signers, &cert_g, &c_a, 0xf0); + let cert_f = verify_successor(&cert_g, &s_fork, &signers).unwrap(); + let receipts_f = receipts_for(&cert_f, &signers); + let activation_f = activation_for(&cert_f, &receipts_f, &signers); + assert!(matches!( + store.record_activation(&s_fork, &activation_f, &receipts_f, &signers), + Err(StoreError::Equivocation) + )); + let ndir = netdir(&dir, &network); + let before = fs::read(ndir.join("state")).unwrap(); + assert!(matches!( + store.mark_receipt_sent(), + Err(StoreError::Equivocation) + )); + assert_eq!(fs::read(ndir.join("state")).unwrap(), before); + } + /// P1-b: a retired candidate certificate can never be reinstalled, /// even under the currently admissible ceremony. Details: TC-2221 #[test]