From 806da371341225ee90cd9da1e0f71f9f3b5564e6 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 11:55:00 +0200 Subject: [PATCH 01/37] Capture power-loss recovery and administrative replay direction --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 55 +++++++++++++++++++++++ .spiral/sources/SRC-20260919-TTZ7C-18.ttl | 11 +++++ 2 files changed, 66 insertions(+) create mode 100644 .spiral/sources/SRC-20260919-TTZ7C-18.md create mode 100644 .spiral/sources/SRC-20260919-TTZ7C-18.ttl diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md new file mode 100644 index 0000000..6aa647d --- /dev/null +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -0,0 +1,55 @@ +--- +id: SRC-20260919-TTZ7C-18 +--- + +# Source: Power-Loss Recovery And Administrative Replay + +## Origin and availability + +Maintainer conversation on 2026-09-19. The operative human statements below +are retained verbatim; the proposed cycle boundary is summarized separately. +There is no external conversation locator. This source supplements SRC-001. + +## Retained human direction + +> If we fail to recover the committed state, we need to have a plausible way for an administrator to recover. + +> We have a command log, if that is complete, shows an accepted command, but its data files are missing.... we can't just assume we can run the command again, it might have secondary effects. But an administrator should be able to judge whether it can be rerun. + +> The commands stored in the log should have all required information. If that is not so, that is a bug. Another thing, I don't think we should allow a command to be rerun, unless there are no accepted commands with their datasets after it. If all commands after it are either waiting or also miss their datasets, then the administrator can run them again. + +> There is one exception. I designed the system to theoretically be able to rerun all commands, useful if you alter the code the command executes and want to create a new dataset consistent with that new code. So I think we should have an option the adminstrator could set to signal to the commands not to trigger side effects, only update the data. + +After a proposed next-cycle plan, the maintainer committed: + +> Lets make a cycle to do all this work. + +## Proposal to which the commitment responded + +The assistant proposed one cycle for durable commits and an administrative +recovery path, explicitly revising the immediate roadmap order: + +- failure matrix and baseline tests before runtime hardening; +- durable acceptance and completion, including file/directory sync ordering; +- simulated lost/reordered/partial writes and sync failures, with acknowledgment + evidence outside the damaged store; +- inspection and administrator-authorized ordered recovery on a copy, preserving + the original store and preventing automatic replay from bypassing the rules; +- command-log input completeness and fixes for demonstrated omissions; +- a tested consistent backup/restore procedure; +- a specified Linux/local-filesystem envelope, latency measurements, and a + disposable block-level power-loss exercise before claiming that envelope; +- recording the rebuild-mode contract now, with the full rebuild workflow in + a following cycle. That mode produces a new store using selected code and + communicates that external side effects must be suppressed. + +This boundary is the assistant's proposal confirmed by the final human message; +it is not a quotation authored by the maintainer. The side-effect flag is a +cooperative command/hook contract, not proof that arbitrary code is sandboxed. + +## Limits + +Exact recovery CLI syntax, durable command ordering, log-format implications, +filesystem test configuration, and handling of ambiguous/corrupt evidence need +design and executable evidence. No runtime outcome or cycle acceptance is +claimed by this source. Creating the cycle is distinct from completing it. diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.ttl b/.spiral/sources/SRC-20260919-TTZ7C-18.ttl new file mode 100644 index 0000000..c56f5b3 --- /dev/null +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:SRC-20260919-TTZ7C-18 + a sd:Source ; + dcterms:identifier "SRC-20260919-TTZ7C-18" ; + sd:repositoryPath ".spiral/sources/SRC-20260919-TTZ7C-18.md" ; + sd:status sd:Active ; + sd:sourceAvailability sd:Retained ; + sd:provenanceConfidence sd:Explicit . From d522359dee1621267df8e550237a9767cb72fccd Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 11:56:09 +0200 Subject: [PATCH 02/37] Establish power-loss and administrative recovery understanding --- .../understandings/UND-20260919-TTZ7C-19.md | 84 +++++++++++++++++++ .../understandings/UND-20260919-TTZ7C-19.ttl | 19 +++++ 2 files changed, 103 insertions(+) create mode 100644 .spiral/understandings/UND-20260919-TTZ7C-19.md create mode 100644 .spiral/understandings/UND-20260919-TTZ7C-19.ttl diff --git a/.spiral/understandings/UND-20260919-TTZ7C-19.md b/.spiral/understandings/UND-20260919-TTZ7C-19.md new file mode 100644 index 0000000..20aa607 --- /dev/null +++ b/.spiral/understandings/UND-20260919-TTZ7C-19.md @@ -0,0 +1,84 @@ +--- +id: UND-20260919-TTZ7C-19 +--- + +# Understanding: Durable Commits With Recoverable Failure + +## My understanding + +The durability objective includes both preventing acknowledged data loss under +the supported power-loss model and giving administrators a practical recovery +route when automatic reconstruction fails. Refusing startup remains appropriate +for ambiguous storage, but refusal alone no longer satisfies the outcome. + +Reconstruction applies surviving committed changesets without executing command +code. Rerunning a command is different: its external effects may already have +happened. Missing data does not authorize automatic re-execution. + +For ordinary recovery, a command may be rerun only with administrator approval +and only when no later accepted command has a dataset. The eligible trailing +commands are executed in their original order from the verified preceding +state. This predicate must consider all accepted commands, not just current +`done` statuses. Corrupted or ambiguous files are not silently treated as absent. + +Logged commands must preserve all execution inputs required beyond the preceding +dataset. Any demonstrated omission is a defect, not an acceptable recovery +limitation. Missing inputs in historical records cannot be invented by tooling. + +Full rebuild is a separate operation: execute selected history using selected +code against a starting dataset and write a new store. It may deliberately +produce different data and bypass ordinary recovery's later-dataset restriction. +Commands and hooks must receive a context instructing them to suppress external +side effects; canonical and derived output writes are still required. A flag +alone does not prevent arbitrary code from making external changes. The contract +is retained now; the full rebuild workflow follows this cycle. + +## Current effective behavior and evidenced gap + +Read-only inspection of baseline `44ea91717a328ad7b5e049c6c0845b5e3bcc1718` +established the following before the human's cycle commitment: + +| Boundary | Current behavior | Gap / remaining evidence | +|---|---|---| +| `src/util.mjs#appendFile` | Appends and awaits `datasync()` | No explicit parent-directory sync for newly created files; failure propagation and torn records need tests | +| Command changesets / final offsets | Installed `write-file-atomic` syncs the temporary file and renames it | No parent-directory sync after publication | +| Base conversion / ID indexes | Ordinary `writeFileSync` | No explicit persistence barrier; classify recovery-required versus safely rebuildable artifacts | +| `src/server.mjs#checkCommand` / completion | Log/status appends awaited before acceptance; `done` awaited before query updates | Complete cross-file persistence dependency and error handling need verification | +| `src/recovery.mjs` | Reads statuses/log; some interrupted active commands become accepted for automatic replay | No administrative uncertainty decision or later-surviving-dataset eligibility guard | +| Recovery order | Log iteration queues accepted commands; status-map iteration selects committed changesets | Establish the authoritative execution order, including concurrent acceptance, before enforcing suffix recovery | +| Command inputs | Normal queue includes request context; replay uses `request: null`; default worker does not forward request to `runCommand` | The FIXME is not proof of differing default-handler inputs; test normal/replay equivalence, including configured workers | +| Existing test suites | Process-crash, corruption, retries, offsets, handler behavior and timeouts | Process termination does not establish loss of kernel/device caches or a tested administrator recovery procedure | + +The earlier conversation's fresh 59-test run is baseline context, not new +power-loss evidence. No runtime changes or new fault tests are claimed here. + +## Material assumptions and boundaries + +- First persistence envelope: a named Linux/local-filesystem configuration with + storage honoring flush requests. Record actual filesystem/mount/virtual-disk + settings in evidence; do not claim arbitrary hardware or remote filesystems. +- The intended ownership model is one writer. Verify it or enforce an appropriate + exclusion mechanism; administrator recovery cannot race ordinary queue writes. +- Recovery works on a copy and preserves the original command log and artifacts. + Revalidate a preview's inputs before applying it so a stale plan cannot bypass + the ordering guard. Promotion is a separate explicit administrator decision. +- A consistent backup plus surviving committed data supplies a recovery route + when rerunning is unsafe or disallowed. Report verified coverage, loss, and + uncertainty; do not promise to reconstruct irretrievably missing information. +- A failed sync can leave commit status uncertain. Prevent false success and + further mutations until recovery establishes a safe state; do not invent a + definitive failed outcome merely because persisting `done` returned an error. +- Preserve disk/query/REST compatibility by default. Any required durable-format + change must explicitly cover migration and legacy-store behavior. + +## Commitment and plan continuity + +The maintainer confirmed the preceding investigation and proposed cycle with +"Lets make a cycle to do all this work" on 2026-09-19. This is the execution +commitment for the agreed goal, not acceptance of its future implementation. + +Revise the immediate SRC-001 / REQ-001 / DES-001 sequence: crash/retry/handler +characterization and optional integrity exist, while file publication and +administrative recovery undermine broader confidence. Address these before +general randomized/soak work. Preserve the remaining roadmap and the recorded +full-rebuild follow-up. Reopen only material assumptions falsified by evidence. diff --git a/.spiral/understandings/UND-20260919-TTZ7C-19.ttl b/.spiral/understandings/UND-20260919-TTZ7C-19.ttl new file mode 100644 index 0000000..0620928 --- /dev/null +++ b/.spiral/understandings/UND-20260919-TTZ7C-19.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:UND-20260919-TTZ7C-19 + a sd:Understanding ; + dcterms:identifier "UND-20260919-TTZ7C-19" ; + sd:repositoryPath ".spiral/understandings/UND-20260919-TTZ7C-19.md" ; + sd:status sd:Active ; + sd:provenanceConfidence sd:Evidenced ; + sd:interprets [ + a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "806da371341225ee90cd9da1e0f71f9f3b5564e6" + ], [ + a sd:ArtifactReference ; + sd:artifact project:SRC-001 ; + sd:gitCommit "44ea91717a328ad7b5e049c6c0845b5e3bcc1718" + ] . From b1f15b5964986a9e3cab9025f08829b88294974c Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 11:57:28 +0200 Subject: [PATCH 03/37] Define power-loss durability and recovery acceptance criteria --- .spiral/requests/REQ-20260919-TTZ7C-20.md | 87 ++++++++++++++++++++++ .spiral/requests/REQ-20260919-TTZ7C-20.ttl | 23 ++++++ 2 files changed, 110 insertions(+) create mode 100644 .spiral/requests/REQ-20260919-TTZ7C-20.md create mode 100644 .spiral/requests/REQ-20260919-TTZ7C-20.ttl diff --git a/.spiral/requests/REQ-20260919-TTZ7C-20.md b/.spiral/requests/REQ-20260919-TTZ7C-20.md new file mode 100644 index 0000000..b4bc552 --- /dev/null +++ b/.spiral/requests/REQ-20260919-TTZ7C-20.md @@ -0,0 +1,87 @@ +--- +id: REQ-20260919-TTZ7C-20 +--- + +# Request: Power-Loss Durability And Administrative Recovery + +## Outcome and audience + +Developers and administrators need a bounded, executable power-loss durability +claim and a practical recovery route when committed storage cannot be loaded. +Complete this as one cycle, from fault evidence through sync hardening, +administrative recovery, and a verified backup/restore exercise. + +This supplements REQ-001 and refines its failure-only allowance: refusing unsafe +startup remains necessary, but a tested administrative route is also required. +It does not retrospectively invalidate the earlier bounded test results. + +## Required behavior and acceptance evidence + +| ID | Requirement | Discriminating evidence | +|---|---|---| +| P1 | Durable command acceptance | The complete execution record and acceptance survive supported power-loss cuts after `202`; a retained command is not mistaken for proof of safe re-execution | +| P2 | Durable completion | Before `done` is observable, the changeset and every required recovery artifact, including new/replaced directory entries, are durable; acknowledged completed state survives modeled cache loss | +| P3 | Explicit I/O uncertainty | Write, short-write, sync, rename, and close failures never report false acceptance/completion; uncertain commit failures halt further mutation pending recovery | +| P4 | Independent crash oracle | Keep issued-command order and observed acknowledgments outside the damaged store; independently reconstruct surviving committed data and detect silent rollback, duplication, and partial state | +| P5 | Read-only recovery inspection | Identify relevant command order/status/attempts, missing/corrupt/ambiguous artifacts, verified preceding state, and known/unknown external effects; produce a preview without changing original evidence | +| P6 | Administrator-authorized ordered recovery | Rerun only when explicitly authorized and no later accepted command has a dataset; execute eligible trailing missing/waiting commands in original order from verified preceding state; do not infer the rule from `done` alone | +| P7 | Recovery cannot be bypassed | Automatic retry and normal queue processing cannot bypass P6 or overtake recovery; revalidate evidence before applying a plan; corrupt/unknown files are not treated as absent | +| P8 | Complete execution inputs | Verify original and recovered invocations have all required logged inputs beyond the preceding dataset, including relevant configured-worker behavior; fix actual omissions, and diagnose incomplete legacy records rather than inventing values | +| P9 | Consistent backup and restore | Provide and test a usable consistent backup procedure and restoration into a separate store; show recovered command coverage, known loss, and uncertainty when reruns are prohibited or unsafe | +| P10 | Preserved evidence and deliberate promotion | Recovery preserves the original store/log and records administrator decisions and execution outcomes; validation precedes explicit promotion; recovery interruption itself must leave an inspectable safe state | +| P11 | Bounded filesystem claim | Name the tested Linux filesystem, mount options, storage/cache assumptions, and writer ownership; validate with a disposable block-level power-loss exercise, distinguish simulation from physical hardware evidence, and measure commit latency | +| P12 | Rebuild exception retained | Document separate new-store rebuild using selected code/history and context instructing commands/hooks to suppress external effects; this cannot become an ordinary recovery override | + +## Mandatory recovery examples + +In established command order, `A:dataset, B:missing, C:dataset` blocks rerunning +B, including when C is accepted but not currently marked done. + +`A:dataset, B:missing, C:waiting, D:missing` permits an administrator-authorized +ordered recovery of B/C/D after checking secondary effects. It is permission +eligibility, not a claim that replay is safe or automatically approved. + +A file that exists but is corrupt requires diagnosis. A later dataset appearing +after preview invalidates that plan. Missing command-log evidence also prevents +asserting eligibility from filenames alone. Failed/unsafe/duplicate records and +concurrent acceptance must receive explicit design/test dispositions, not a +silent choice of a convenient map iteration order. + +## Implementation direction and compatibility + +Start with a failure matrix and baseline tests. Reuse the current file-based +architecture and reconstruction oracle; choose small durable-write primitives +and explicit sequencing over a new storage framework. Determine which indexes +must persist before commit and which can be rebuilt safely from canonical data. +Include initial base/log creation and recovery writes in the persistence audit. + +The supported first envelope assumes a single writer and storage that honors +flushes on a named Linux/local-filesystem configuration. If that envelope cannot +be established in the test environment, report the missing evidence and keep +the power-loss claim outstanding. Do not relabel process-kill tests as proof. + +Preserve public REST/query APIs and disk formats by default. Any necessary log +or recovery format extension must explicitly address old stores, migration, +failure behavior, and downstream compatibility before implementation. + +Full rebuild tooling is follow-up work under P12's retained contract. Side-effect +suppression is cooperative unless separately enforced; actual command and hook +implementations need verification before being called rebuild-safe. Changes to +command code are explicit rebuild inputs, not ordinary recovery assumptions. + +## Scope exclusions + +- General randomized/destructive soak campaigns beyond the focused fault matrix + and disposable filesystem acceptance exercise. +- Full rebuild CLI/workflow implementation in this cycle. +- Automatic rollback or exactly-once guarantees for arbitrary external effects. +- General message buses, transaction frameworks, remote-filesystem guarantees, + sandbox replacement, package publishing, or production data recovery/deployment. + +## Evaluation + +Each P1-P12 row needs traceable implementation/documentation and verification +evidence appropriate to its claim. P12 is a recorded contract, not a tested +rebuild feature. Run relevant existing startup/offset, handler, ACID, and crash +suites as regression checks. A failed or unavailable required exercise remains +an open acceptance item; creation of the cycle is not acceptance of its outcome. diff --git a/.spiral/requests/REQ-20260919-TTZ7C-20.ttl b/.spiral/requests/REQ-20260919-TTZ7C-20.ttl new file mode 100644 index 0000000..22f1303 --- /dev/null +++ b/.spiral/requests/REQ-20260919-TTZ7C-20.ttl @@ -0,0 +1,23 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260919-TTZ7C-20 + a sd:Request ; + dcterms:identifier "REQ-20260919-TTZ7C-20" ; + sd:repositoryPath ".spiral/requests/REQ-20260919-TTZ7C-20.md" ; + sd:status sd:Active ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:UND-20260919-TTZ7C-19 ; + sd:gitCommit "d522359dee1621267df8e550237a9767cb72fccd" + ] ; + sd:constrainedBy [ + a sd:ArtifactReference ; + sd:artifact project:REQ-001 ; + sd:gitCommit "44ea91717a328ad7b5e049c6c0845b5e3bcc1718" + ], [ + a sd:ArtifactReference ; + sd:artifact project:DES-001 ; + sd:gitCommit "44ea91717a328ad7b5e049c6c0845b5e3bcc1718" + ] . From 9fbdf89dcca8d7a0b18e9ddfc41c524f2ee854e2 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 11:59:36 +0200 Subject: [PATCH 04/37] Open power-loss durability and administrative recovery cycle --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 154 +++++++++++++++++++++++ .spiral/cycles/CYC-20260919-TTZ7C-17.ttl | 14 +++ .spiral/project-context.md | 12 +- AGENTS.md | 11 ++ 4 files changed, 188 insertions(+), 3 deletions(-) create mode 100644 .spiral/cycles/CYC-20260919-TTZ7C-17.md create mode 100644 .spiral/cycles/CYC-20260919-TTZ7C-17.ttl diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md new file mode 100644 index 0000000..317ed50 --- /dev/null +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -0,0 +1,154 @@ +# Cycle: Power-Loss Durability And Administrative Recovery + +Branch: `spiral/CYC-20260919-TTZ7C-17-power-loss-recovery`. +Created from authoritative `master` at +`44ea91717a328ad7b5e049c6c0845b5e3bcc1718` on 2026-09-19. +Branch creation and correspondence were verified with Git. The pre-existing +untracked `package.json~` is unrelated and remains untouched. + +Status: **Active**. The maintainer committed to this cycle on 2026-09-19: +"Lets make a cycle to do all this work." This accepts the proposed work boundary, +not the eventual runtime result. Creation records the plan; implementation and +acceptance evidence remain outstanding. + +## Analyze + +### Goal + +Make command acceptance and completion durable across the supported power-loss +model, and provide an administrator with a tested, evidence-preserving recovery +route when committed storage cannot be reconstructed automatically. + +### Current effective behavior and gap + +The project has crash/retry, malformed-storage, handler, timeout, integrity, and +persisted-offset evidence. The latest offset cycle is accepted and merged. +Log appends already await `datasync`; changesets/final offsets already use file +sync plus rename. Directory entries, initial base/ID-index writes, sync-error +handling, and administrator recovery remain incomplete. Existing retry evidence +does not establish safety for secondary effects. The detailed inspected baseline +and input-completeness caveat are in UND-20260919-TTZ7C-19. + +### Why this boundary and why now + +A narrower sync-only change would leave administrators stranded after explicit +startup failure and would retain unsafe replay assumptions. One cycle therefore +covers failure evidence, persistence ordering, recovery decisions, and backup +restoration. Full rebuild has a separate evaluation boundary: intentionally +changed code and cooperative suppression of effects need their own workflow. + +### Plan continuity: revise + +Re-read SRC-001, REQ-001, and DES-001. After accepted crash/retry/handler and +integrity slices and the offset correction, bring file publication and +administrative recovery ahead of general randomized/soak testing. The new human +direction strengthens failure-only recovery expectations and preserves the +existing small architecture. It does not replace the remaining roadmap. + +### Adopted profiles + +The project's Muze engineering and library stewardship profiles apply: preserve +inspectability, small replaceable mechanisms, and explicit compatibility. +WPF-HUMAN-001 applies to acknowledgment evidence, administrator authority, and +recovery options. No new profile or independent veto is introduced. The concrete +scope is grounded in the maintainer's requirements. + +## Plan + +### Committed requirements + +REQ-20260919-TTZ7C-20 is the acceptance matrix (P1-P12), interpreted from retained +human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. + +- Complete command inputs must be logged; demonstrated omissions are bugs. +- Ordinary rerun requires administrator approval and no later accepted command + with a dataset. Recover the eligible trailing sequence in original order. +- Missing data is not proof that external effects did not happen. Existing + corrupt/ambiguous data is not silently classified as missing. +- Automatic retries and ordinary queue activity cannot bypass recovery rules. +- Preserve original evidence, work on a copy, validate and explicitly promote. +- Provide a tested consistent backup/restore path for unsafe/disallowed reruns. +- Retain separate rebuild semantics: selected code/history, new output store, + side-effect suppression context for commands/hooks; full workflow follows. + +### Work sequence + +1. **Failure matrix and baseline:** enumerate durable artifacts and dependencies, + establish command ordering/ownership, model lost/reordered/partial writes and + lost directory entries, and record acknowledgments independently of the store. + Preserve failing evidence before correcting runtime behavior. +2. **Design:** specify acceptance/completion barriers, required versus rebuildable + indexes, initial-file publication, I/O uncertainty handling, recovery preview + and apply behavior, input completeness, and compatibility. Create DES/IMP/EVD + artifacts when those decisions/results crystallize, not empty placeholders. +3. **Durable writes:** complete file/directory sync sequencing before acceptance + or completion becomes visible; ensure errors cannot falsely report success + or allow unsafe further mutations. +4. **Administrative recovery:** implement inspect/preview and explicitly authorized + ordered recovery on a copy; enforce later-dataset guards, stop on uncertain + effects, revalidate stale plans, and record outcomes without erasing history. + Cover interruption of recovery itself and normal/automatic queue interactions. +5. **Inputs and backup/restore:** test normal versus recovered invocation inputs, + correct demonstrated log omissions, diagnose incomplete legacy records, and + exercise consistent backup creation and restoration with reported coverage. +6. **Filesystem validation and evaluation:** exercise a disposable block-level + power-loss setup on a documented Linux/local filesystem, measure commit + latency, run relevant regression suites and checks, and provide a runnable + administrator guide with bounded durability claims and known limitations. + +### Acceptance examples + +| Scenario | Required result | +|---|---| +| Power cut after observed `accepted` | Complete accepted command survives under the supported model; replay safety is assessed separately | +| Power cut after observed `done` | Recovered state includes that completed transition, with no partial state, duplication, or silent rollback | +| Write/sync/rename failure | No false acknowledgment; uncertain outcomes prevent further mutation pending recovery | +| A has data, B missing, C has data | Rerunning B is blocked, even if C is not currently `done` | +| A has data, B missing, C waiting, D missing | Administrator can authorize B/C/D in order after judging external effects | +| Later dataset appears after preview | Stale recovery plan is rejected/recomputed before execution | +| Corrupt artifact or incomplete log evidence | Explicit diagnosis; do not infer eligibility from absence of valid parsing | +| Rerun unsafe or prohibited | Demonstrate backup restoration, disclose verified coverage, loss, and uncertainty | +| Interrupted administrative recovery | Original evidence remains intact; subsequent inspection can explain the partial recovery attempt | + +All P1-P12 criteria remain open until evidenced; P12 is documentation of a +follow-up contract, not acceptance of an implemented rebuild feature. Existing +startup/offset, handler, ACID, and crash suites must remain green with any +intentional retry-policy changes reflected by justified new tests. + +### Non-goals + +Full rebuild tooling, broad randomized/soak campaigns, arbitrary external-effect +rollback or exactly-once delivery, sandbox replacement, general database/message +frameworks, remote-filesystem promises, package release, and production recovery +or deployment. The focused block-level fault exercise remains in scope. + +### Material assumptions and replan conditions + +The first envelope is a named Linux/local filesystem with storage honoring +flushes and one writer; exact environment and ownership checks are design work. +Destructive tests use disposable storage only. Missing test infrastructure leaves +the required exercise outstanding; it does not authorize a weaker claim. + +Replan affected work if authoritative command order cannot be established, +necessary legacy inputs cannot be recovered, a breaking format/API change is +required without an accepted compatibility path, or the smallest safe solution +contradicts the stated architecture/administrator rules. Storage irrecoverability +must be reported honestly, not hidden by an invented replay outcome. + +## Act + +Cycle opening captured source, understanding, and request as separate immutable +causal commits, with companion Turtle references to prior versions. Project +context and repository instructions now point to this refinement. + +Next executable work: the failure matrix and baseline tests in step 1. +No new runtime implementation or power-loss test results exist at opening. + +## Evaluate + +Pending. Cycle creation is complete; the engineering cycle is Active. +Before review, verify the acceptance matrix, runtime evidence, administrator +procedure, commit latency, and remaining limits. Validate staged provenance before +each semantic commit and range/tree provenance before integration. Only human +acceptance can mark the cycle Accepted; then validate the exact prospective +integration against current `master`. Do not merge this Active cycle outward. diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl new file mode 100644 index 0000000..4661c22 --- /dev/null +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl @@ -0,0 +1,14 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:CYC-20260919-TTZ7C-17 + a sd:Cycle ; + dcterms:identifier "CYC-20260919-TTZ7C-17" ; + sd:repositoryPath ".spiral/cycles/CYC-20260919-TTZ7C-17.md" ; + sd:status sd:Active ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" + ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 9c333d1..a5f4597 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,7 +18,9 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Current active Spiral cycle: none. Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18. Conversion and command changesets now finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. Integration must still be validated against the current target. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16. +Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-Loss Durability And Administrative Recovery`), committed by the maintainer on 2026-09-19 and opened on `spiral/CYC-20260919-TTZ7C-17-power-loss-recovery` from `master` at `44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. The cycle is Active; runtime work and acceptance evidence remain outstanding. Its source, understanding, and acceptance matrix are `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20`. + +Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. ## Intake State @@ -96,7 +98,11 @@ The first production-readiness priority is proving more of SimplyStore's ACID cl The current durability/extensibility source is `.spiral/sources/SRC-001.md`. The first ordered slice is baseline archaeology, followed by a crash/fault-injection harness, an independent reconstruction oracle, adversarial storage tests, retry/idempotency hardening, after-change handler contracts, handler failure evidence, optional integrity roots, integrity acceptance tests, property/randomized durability tests, destructive soak tests, a minimal post-commit extension seam, a demonstration derived store, and `DURABILITY.md`. -Process continuity rule: before proposing the next durability/production-readiness cycle, re-read `.spiral/sources/SRC-001.md`, `.spiral/requests/REQ-001.md`, and `.spiral/designs/DES-001.md`; identify the current position in that ordered plan; compare the latest evidence with it; and explicitly classify the proposal as continue, revise, or deliberate deviation. +**2026-09-19 refinement β€” revise immediate order:** `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20` bring durable file publication and administrative recovery ahead of general randomized/soak work. These refine the earlier failure-only allowance: explicit startup refusal must be accompanied by a tested recovery route. Accepted/completed acknowledgments need complete file/directory persistence barriers; storage-fault evidence precedes hardening, followed by recovery tooling, command-input completeness, consistent backup/restore, latency measurements, and a disposable filesystem power-loss exercise. The rest of the original roadmap remains. + +**Recovery contract (required, not yet implemented):** missing datasets do not prove that external effects never happened. Ordinary rerun requires administrator approval and no later accepted command with a dataset; execute eligible trailing missing/waiting commands in original order from verified preceding state. Automatic retries must not bypass this rule. Preserve original evidence, preview recovery on a copy, and validate before explicit promotion. Required command inputs must be logged; omissions are bugs. Separate full rebuild may replay selected history with changed code into a new store while instructing commands/hooks to suppress external side effects. Record that cooperative contract in this cycle; full rebuild tooling is follow-up work. + +Process continuity rule: before proposing the next durability/production-readiness cycle, re-read `.spiral/sources/SRC-001.md`, `.spiral/requests/REQ-001.md`, `.spiral/designs/DES-001.md`, and the 2026-09-19 refinement in `.spiral/sources/SRC-20260919-TTZ7C-18.md` / `.spiral/requests/REQ-20260919-TTZ7C-20.md`; identify the current position in that ordered plan; compare the latest evidence with it; and explicitly classify the proposal as continue, revise, or deliberate deviation. Current Spiral collaboration rule: treat planning, evaluation, and ambiguous human suggestions as discourse until a sufficiently explicit commitment exists. Human confirmation of a cycle goal is the commitment boundary for execution within that goal; later tentative comments should not silently become scope expansion. Before commitment, surface material ambiguities, contradictions, unsupported premises, or alternative framings when resolving them differently could materially change downstream work. @@ -269,7 +275,7 @@ This is not yet a claim that SimplyStore is production-safe for all workloads. I - Continue guided brownfield intake as concrete durability, security, API, and downstream questions arise. - Add a developer/operator script for resetting or cleaning local/example datasets after explicit recovery failures. -- Decide later whether SimplyStore should ever attempt durable artifact recovery/repair, or only provide inspection and reset tooling. +- Administrative inspection, authorized ordered recovery, and consistent backup/restore are now committed in CYC-20260919-TTZ7C-17. More general automatic repair remains a later decision. - Characterize the VM2 replacement risk as a candidate early Spiral cycle. - Extend process-level fault tests around command acceptance, changeset write, status append, and restart recovery. - Define larger-system integration options after the durability priority is better bounded. diff --git a/AGENTS.md b/AGENTS.md index e8351d7..3665a8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,6 +41,17 @@ SimplyStore has a governing durability/extensibility direction in: - `.spiral/requests/REQ-001.md` - `.spiral/designs/DES-001.md` +The maintainer refined that direction on 2026-09-19 in: + +- `.spiral/sources/SRC-20260919-TTZ7C-18.md` +- `.spiral/understandings/UND-20260919-TTZ7C-19.md` +- `.spiral/requests/REQ-20260919-TTZ7C-20.md` + +Read this refinement alongside the original direction. It requires an +administrative recovery path, complete logged command inputs, and ordered rerun +eligibility; it records separate new-store rebuild semantics without making +full rebuild tooling part of the current power-loss cycle. + Before proposing a next durability/production-readiness cycle, re-read those durable references, identify the current position in the ordered plan, reconcile the latest evidence with that plan, and state whether the proposal continues, revises, or deliberately deviates from it. Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifacts and companion Turtle resources only when they are causally useful for an actual cycle, decision, implementation, evidence, acceptance, or lesson. From 88fbfadffe40474590b777cd735996314a8b59c0 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 12:18:26 +0200 Subject: [PATCH 05/37] Design independent power-loss baseline and fault matrix --- .spiral/designs/DES-20260919-TTZ7C-21.md | 69 +++++++++++++++++++++++ .spiral/designs/DES-20260919-TTZ7C-21.ttl | 11 ++++ 2 files changed, 80 insertions(+) create mode 100644 .spiral/designs/DES-20260919-TTZ7C-21.md create mode 100644 .spiral/designs/DES-20260919-TTZ7C-21.ttl diff --git a/.spiral/designs/DES-20260919-TTZ7C-21.md b/.spiral/designs/DES-20260919-TTZ7C-21.md new file mode 100644 index 0000000..4a191dd --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-21.md @@ -0,0 +1,69 @@ +# Design: Power-Loss Baseline Harness + +## Purpose and boundary + +Implement the first work item of CYC-20260919-TTZ7C-17: discriminating baseline +tests and a failure matrix before persistence/recovery fixes. The design follows +REQ-20260919-TTZ7C-20. Product behavior remains unchanged in this slice. + +Use actual server/worker writes with a test-only Node preload. Trace completed +file operations and syncs outside the disposable store; capture synchronized +bytes for the small fixture logs. Independently record client-issued commands, +observed acceptance, and observed completion. The expected state comes from +fixture semantics and external observations, never solely from damaged logs. + +The preload is explicitly enabled only by a test configuration and is inherited +by worker threads. Scope interception to the disposable store. Capture filesystem +function references before wrapping so tracing cannot recurse. Controls must +prove that instrumentation observes real appends, file syncs, and renames. + +## Fault model + +- Trace file sync and namespace publication separately. After abrupt process + termination, remove a newly created/renamed filename only when its trace shows + no subsequent directory sync. Such image edits model a permitted persistence + outcome; they are not physical power cuts or measured filesystem behavior. +- At an explicit pause before a log sync, restore its last successfully synced + byte snapshot. Only unsynced data may be lost in this model. Preserve external + acknowledgments and side-effect witnesses outside the crash image. +- Inject I/O errors before a selected operation, and a real partial callback + write where applicable. Assert that fault injection was actually reached. +- Torn/corrupt durable artifacts and synthetic recovery-status combinations are + separate adversarial/policy fixtures. Do not present arbitrary mutation of + previously synced bytes as a valid cache-loss model. +- A controlled append-completion gate can exercise concurrent acceptance order + without relying on timing races. Test that subsequent recovery does not derive + its expected order from the same log it is checking. + +## Reporting and tests + +Ordinary assertions verify setup, interception, completed operations, observed +responses, and current classifications. Known requirement violations appear in +nested tests marked TODO. `SIMPLYSTORE_BASELINE_STRICT=1` removes only those TODO +annotations, exposing reproducible failing invariant assertions with a nonzero +exit. A fixture/harness failure outside that final invariant remains a real +failure in normal baseline mode. Remove TODO annotations as later fixes earn it. + +Cover durable acceptance/completion, missing directory entries, derived index +loss, interrupted status append, explicit sync/rename/write failures, automatic +replay with secondary effects, later-dataset rerun restrictions, command input +completeness, and concurrent queue order. Name untested matrix rows explicitly; +this baseline is not acceptance of the whole cycle or administrator tooling. + +Add an isolated `test:power-loss` command and a human-readable matrix with test +names, expected/current results, and limitations. Existing suites stay separate +and are run as regression evidence. Only new test files, package test scripts, +and Spiral/documentation artifacts belong to this implementation slice. + +## Independence and limits + +External observations and side-effect witnesses survive every modeled cut and +are not reconstructed from the store under test. Fixture command behavior is +simple and known. The OD-JSONTag parser remains shared for reading actual bytes; +compare its output against independently expected values/order. + +The trace covers the explicit filesystem APIs used by these fixtures, not all +Node/native/custom-module I/O. It cannot prove hardware flush honesty, torn-sector +semantics, mount behavior, cross-process writer exclusion, consistent backups, +or full administration/rebuild workflows. A disposable block-level exercise and +the remaining P1-P12 work are still required later in the active cycle. diff --git a/.spiral/designs/DES-20260919-TTZ7C-21.ttl b/.spiral/designs/DES-20260919-TTZ7C-21.ttl new file mode 100644 index 0000000..88b678e --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-21.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260919-TTZ7C-21 a sd:Design ; + dcterms:identifier "DES-20260919-TTZ7C-21" ; + sd:repositoryPath ".spiral/designs/DES-20260919-TTZ7C-21.md" ; + sd:status sd:Active ; + sd:satisfies [ a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] . From 0766186d24e973c47bc78865c3b71a6272effde0 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 12:28:54 +0200 Subject: [PATCH 06/37] Add power-loss fault matrix and independent baseline tests --- .../implementations/IMP-20260919-TTZ7C-22.md | 30 ++ .../implementations/IMP-20260919-TTZ7C-22.ttl | 17 + docs/power-loss-baseline.md | 112 ++++++ package.json | 4 +- test/power-loss-baseline.mjs | 331 ++++++++++++++++++ test/power-loss-helpers.mjs | 118 +++++++ test/power-loss-preload.mjs | 129 +++++++ 7 files changed, 740 insertions(+), 1 deletion(-) create mode 100644 .spiral/implementations/IMP-20260919-TTZ7C-22.md create mode 100644 .spiral/implementations/IMP-20260919-TTZ7C-22.ttl create mode 100644 docs/power-loss-baseline.md create mode 100644 test/power-loss-baseline.mjs create mode 100644 test/power-loss-helpers.mjs create mode 100644 test/power-loss-preload.mjs diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-22.md b/.spiral/implementations/IMP-20260919-TTZ7C-22.md new file mode 100644 index 0000000..abc85bf --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-22.md @@ -0,0 +1,30 @@ +# Implementation: Independent Power-Loss Baseline + +Implements DES-20260919-TTZ7C-21 as a new isolated test suite. No `src/` or +production script behavior changes are included. + +- `test/power-loss-preload.mjs`: explicitly enabled, store-scoped I/O tracing and + faults, inherited by disposable server workers; synced-byte snapshots and + namespace publication observations live outside the failed storage domain. +- `test/power-loss-helpers.mjs`: external client observations, bounded waits, + abrupt shutdown, copied crash images, trace-constrained filename loss, and + isolated expected-failure invariant reporting. +- `test/power-loss-baseline.mjs`: 21 scenarios covering normal persistence, + missing publications, corruption, I/O errors and short writes, effects on + replay, later-dataset guards, input completeness, concurrent order, and a + positive directory-sync model control. +- `package.json`: `test:power-loss` and `test:power-loss:strict` scripts. Strict + mode exposes requirement failures; normal baseline uses explicit nested TODO + assertions while keeping harness/control failures fatal. +- `docs/power-loss-baseline.md`: fault matrix, commands, current observations, + independence limits, and remaining cycle work. + +The test-only custom finalizer in PL21 supplies a directory sync as a positive +control; it is not a production fix. Policy fixtures and torn-record corruption +are explicitly distinguished from permitted unsynced-data/name-loss models. +The partial-write injection actually writes a shorter buffer through the current +dependency; it does not merely fabricate an error return. + +This new suite adds a separate evidence mechanism without changing the existing +durability helper or runtime implementation units. Verification results belong +in EVD-20260919-TTZ7C-23 after this implementation has a committed version. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-22.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-22.ttl new file mode 100644 index 0000000..e34c9ff --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-22.ttl @@ -0,0 +1,17 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260919-TTZ7C-22 a sd:Implementation ; + dcterms:identifier "IMP-20260919-TTZ7C-22" ; + sd:repositoryPath ".spiral/implementations/IMP-20260919-TTZ7C-22.md" ; + sd:status sd:Active ; + sd:implementationLocation "test/power-loss-preload.mjs" ; + sd:implementationLocation "test/power-loss-helpers.mjs" ; + sd:implementationLocation "test/power-loss-baseline.mjs" ; + sd:implementationLocation "package.json#scripts.test:power-loss" ; + sd:implementationLocation "package.json#scripts.test:power-loss:strict" ; + sd:implementationLocation "docs/power-loss-baseline.md" ; + sd:implements [ a sd:ArtifactReference ; + sd:artifact project:DES-20260919-TTZ7C-21 ; + sd:gitCommit "88fbfadffe40474590b777cd735996314a8b59c0" ] . diff --git a/docs/power-loss-baseline.md b/docs/power-loss-baseline.md new file mode 100644 index 0000000..b67f57c --- /dev/null +++ b/docs/power-loss-baseline.md @@ -0,0 +1,112 @@ +# Power-loss failure matrix and baseline + +This is the first evidence slice of CYC-20260919-TTZ7C-17, against runtime +baseline `44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. It does not change production +persistence or recovery. The cycle's administrative tooling, sync fixes, and +filesystem acceptance exercise remain outstanding. + +## Run and interpret + +```sh +npm run test:power-loss +npm run test:power-loss:strict +``` + +The tests need permission to bind localhost ports and use disposable directories. +The first command characterizes current behavior: fixture checks and controls +must pass, while known violated requirements are explicit nested TODO tests. +The strict command removes those annotations and must currently exit nonzero. +Neither a green baseline command nor its TODO count means power-loss safety. +Remove the corresponding TODO when later implementation satisfies an invariant; +also revise any characterization assumptions that intentionally describe the +old defect. Do not suppress a new harness/setup failure with TODO. + +## Persistence boundaries + +| Artifact | Current writer / barrier | Required role | +|---|---|---| +| Base dataset | Converter uses `writeFileSync` | Durable starting state before operating or backing up a store; untested conversion publication remains open | +| Command log | `appendFile` then `datasync` | Complete inputs before accepted; persist first-created filename | +| Command status | `appendFile` then `datasync` | Accepted/active/done ordering; persist first-created filename; distinguish sync error from guaranteed rollback | +| Changeset | `write-file-atomic`: file sync then rename | Complete bytes and durable filename before done | +| Final offset index | `write-file-atomic`: file sync then rename | Current startup reconstructs without it; custom consumers still need an explicit contract | +| ID index | `writeFileSync` | Current startup reconstructs without it; no universal custom-index conclusion | +| Integrity manifest | Append then `datasync` | Ordered metadata for integrity-enabled stores; dedicated power-loss cases remain open | + +File sync and directory sync are distinct barriers. The namespace-loss model +uses this distinction from the [Linux fsync contract](https://man7.org/linux/man-pages/man2/fsync.2.html). +It does not assert that every tested filesystem would actually lose an entry. + +## Executable matrix + +| Test | Fault or question | Observed baseline | Requirement disposition | +|---|---|---|---| +| PL01 | Instrumented normal commit, then process crash | Existing log contents sync; changeset temp file sync precedes rename; restart equals external expected state | Passing control, not physical power-loss proof | +| PL02 | Drop newly created status filename lacking a directory barrier after observed done | Startup serves empty base state despite external completion evidence | P2 violated: silent loss | +| PL03 | Drop newly created command-log filename after observed accepted | Server starts with pending status but no reconstructible command task | P1 violated: accepted inputs lost | +| PL04 | Drop renamed changeset filename lacking directory barrier after observed done | Startup refuses with missing changeset | P2 durability violated; explicit failure is preferable to silent state | +| PL05 | Remove default ID/offset sidecars from copied store | Startup reconstructs expected canonical state | Current default recovery does not require persisted sidecars | +| PL06 | Append a malformed status tail to a copied store | Startup refuses; original store preserved | Passing corruption classification; administrator path remains absent | +| PL07 | Changeset file fsync throws EIO | Command fails, no done or query update | Passing bounded error propagation | +| PL08 | Changeset rename throws EIO | Command fails, no done or query update | Passing bounded error propagation | +| PL09 | Changeset write throws ENOSPC | Command fails, no done or query update | Passing bounded error propagation | +| PL10 | Callback write succeeds with only half the changeset bytes | Command becomes done and live query looks correct; restart refuses truncated data | P3 violated: short write is not retried/validated | +| PL11 | Done-status datasync throws EIO | Server exits; unsynced done bytes remain visible in the file | Passing conservative termination; failure is not proof of rollback | +| PL12 | Pause before done sync, lose unsynced status tail and unpublished changeset name | Restart repeats an external side-effect witness, then commits one dataset transition | P7 violated: dataset idempotency is insufficient | +| PL13 | B accepted/missing; later accepted C has real dataset, with current status accepted or done | Recovery queues B in both cases | Two P6 violations: later-dataset guard absent | +| PL14 | JSONTag command with date, nested data, false/zero/null and Unicode | Logged/reconstructed body preserves the input | Passing body-completeness control | +| PL15 | Configured worker uses request query parameter | Normal execution uses it; log/reconstructed task omits it | P8 violated for this supported worker configuration | +| PL16 | A log sync completes but its acceptance is gated; B accepted while a blocker runs | External acceptance and original waiting queue B/A; command log A/B; restart executes A/B | P4/P6 violated: recovery order differs from waiting-queue order | +| PL17 | Active A and done C with missing C dataset | Startup appends accepted for A before refusing C; old records remain intact | P7 violated: uncertain active work reclassified without review | +| PL18 | Command-log datasync error | HTTP 500, no accepted response or query mutation | Passing acceptance barrier | +| PL19 | Accepted-status datasync error | HTTP 500, no accepted response or query mutation | Passing acceptance barrier | +| PL20 | Error reported after command-log close | HTTP 500, no accepted response or query mutation | Passing conservative close-error propagation | +| PL21 | Test custom finalizer explicitly syncs the parent directory | Trace observes the barrier; loss model preserves filenames; restart matches independent expectation | Positive control for directory tracing and loss-model discrimination | + +PL02/03/04/12 alter a stopped copy and leave the source store and observer files +untouched. PL06 is deliberate corruption, not cache loss. PL13 and PL17 are +synthetic policy/startup inputs, not claims that a power cut produced those +exact histories. PL15/16 use actual HTTP calls and the configured/default worker +paths respectively. Concurrent PL16 uses a controlled gate instead of a race. + +## Independent observations and model limits + +Each fixture has a store directory and a separate observer directory. Client +issued commands, acceptance responses, and polled completion responses go into +an external journal. Expected fixture values are derived from those inputs and +observations, not the store's command log/status. Actual data reconstruction +still uses OD-JSONTag parsing, so format decoding is not an independent parser. + +An explicitly loaded test-only preload traces promises FileHandle append/sync/ +close operations and callback open/write/fsync/rename/close used by the current +runtime and installed atomic-write dependency. Worker threads inherit it. +It records synchronized fixture-log bytes and observed namespace publication. +The loss model can drop a new filename only if no later directory sync appears +in the trace. PL12 restores the last completed status sync in a controlled +single-writer interval. It never removes acknowledged synced content arbitrarily. + +This is a bounded permitted-outcome model, not a general virtual filesystem: +it does not intercept every Node/native/custom I/O API or enumerate every +possible persistence ordering. The observer is assumed outside the failed +storage domain. It does not model sector corruption of earlier synced records, +hardware lies about flushes, real filesystem mount behavior, or external systems. +The external-effect witness proves repeated invocation only, not exactly-once +behavior of any real email/payment/network service. + +## Still required in this cycle + +- Initial base/log publication and integrity-enabled power-loss combinations. +- Append partial-write/ENOSPC details, other close failures, interrupted rename + completion, and sync-error recovery beyond the named cases above. +- Explicit authoritative ordering, failed/unsafe histories, duplicated IDs, + writer ownership, and exclusion of concurrent administrative/normal writes. +- Administrative inspect/preview/apply, stale-plan rejection, allowed trailing + recovery, interrupted recovery, verified promotion, and loss reporting. +- Complete log inputs across relevant configured integrations and compatibility + for legacy records; a FIXME alone is not the input-loss evidence. +- Consistent backup creation and restore, including cases where rerun is unsafe. +- Measured latency, a named Linux filesystem/storage configuration, and an actual + disposable block-level exercise before claiming filesystem power-loss safety. + +Full changed-code rebuild tooling remains follow-up work under the cycle's +recorded side-effect-suppression contract. diff --git a/package.json b/package.json index 9dfad8a..61d5e11 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,9 @@ "test": "node --test test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs", "test:acid": "node --test test/acid-baseline.mjs", "test:durability": "node --test test/durability-crash.mjs", - "test:handlers": "node --test test/after-change-handlers.mjs" + "test:handlers": "node --test test/after-change-handlers.mjs", + "test:power-loss": "node --test --test-reporter=tap test/power-loss-baseline.mjs", + "test:power-loss:strict": "SIMPLYSTORE_BASELINE_STRICT=1 node --test --test-reporter=tap test/power-loss-baseline.mjs" }, "author": "auke@muze.nl", "license": "MIT", diff --git a/test/power-loss-baseline.mjs b/test/power-loss-baseline.mjs new file mode 100644 index 0000000..9960405 --- /dev/null +++ b/test/power-loss-baseline.mjs @@ -0,0 +1,331 @@ +import assert from 'node:assert/strict' +import test from 'node:test' +import fs from 'node:fs/promises' +import path from 'node:path' +import {Buffer} from 'node:buffer' +import JSONTag from '@muze-nl/jsontag' +import {getChangesetPath, loadCommandLog, loadCommandStatus} from '../src/recovery.mjs' +import {getOpenPort, waitForServer, waitForExit, queryPersons, readCommandLogRecords, + readCommandStatusRecords, reconstructCommittedPersonNames} from './durability-helpers.mjs' +import {fixture, rows, submit, completion, eventually, launch, crash, copyImage, + loseUnpublishedName, requirement} from './power-loss-helpers.mjs' + +async function ready(t, store, options = {}) { + const port = await getOpenPort() + const server = await launch(t, store, port, options) + await waitForServer(server.child, server.getOutput, port) + return {...server, port} +} + +async function commit(store, server, id = 'A', value = id) { + const result = await submit(store, server.port, {id, name: 'addPerson', value: {name: value}}) + assert.equal(result.http, 202) + assert.equal((await completion(store, server.port, id)).status, 'done') +} + +async function restartedFailure(t, image) { + const server = await launch(t, image, await getOpenPort()) + const exit = await waitForExit(server.child) + assert.notEqual(exit.code, 0) + assert.doesNotMatch(server.getOutput(), /SimplyStore listening/) + return server.getOutput() +} + +async function expectedNames(store) { + // Fixture oracle uses only externally observed completions and original inputs. + const journal = await rows(store.journal) + const issued = new Map(journal.filter(e => e.event === 'issued').map(e => [e.command.id, e.command])) + return journal.filter(e => e.event === 'completion' && e.body.status === 'done') + .map(e => issued.get(e.id).value.name) +} + +test('PL01 control: trace observes actual file syncs before completed state survives restart', async t => { + const store = await fixture(t) + // Existing log names let this control isolate synchronized contents. + await fs.writeFile(store.commandLog, '') + await fs.writeFile(store.commandStatus, '') + const server = await ready(t, store) + await commit(store, server) + const trace = await rows(store.trace) + assert.ok(trace.some(e => e.op === 'datasync' && e.file === store.commandLog)) + assert.ok(trace.some(e => e.op === 'datasync' && e.file === store.commandStatus)) + const changeset = getChangesetPath(store.datafile, 'A') + const rename = trace.findIndex(e => e.op === 'rename' && e.file === changeset) + assert.ok(rename > 0) + assert.ok(trace.slice(0, rename).some(e => e.op === 'sync' && e.file === trace[rename].from)) + assert.ok(trace[rename].threadId > 0, 'preload also instruments command worker threads') + await crash(server.child) + const image = await copyImage(t, store) + const restarted = await ready(t, image) + assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), await expectedNames(store)) + assert.deepEqual(await reconstructCommittedPersonNames(image), await expectedNames(store)) +}) + +test('PL02 new status filename can be lost after externally observed done', async t => { + const store = await fixture(t) + const server = await ready(t, store) + await commit(store, server) + await crash(server.child) + const expected = await expectedNames(store) + assert.deepEqual(expected, ['A']) + const originalStatus = await fs.readFile(store.commandStatus) + const image = await copyImage(t, store) + await loseUnpublishedName(store, image, store.commandStatus) + const restarted = await ready(t, image) + const actual = (await queryPersons(restarted.port)).map(p => p.name) + assert.deepEqual(await fs.readFile(store.commandStatus), originalStatus, 'original evidence is preserved') + await requirement(t, 'P2', actual, expected, 'completed command must not silently disappear with its status filename') +}) + +test('PL03 new command-log filename can be lost after observed acceptance', async t => { + const store = await fixture(t, {commandsSource: 'export default { hang() { while (true) {} } }'}) + const server = await ready(t, store) + assert.equal((await submit(store, server.port, {id: 'waiting', name: 'hang'})).http, 202) + await eventually(async () => (await rows(store.trace)).some(e => e.file === store.commandStatus && + e.op === 'datasync' && Buffer.from(e.bytes, 'base64').toString().includes('active')), 'active is synced') + await crash(server.child) + const image = await copyImage(t, store) + await loseUnpublishedName(store, image, store.commandLog) + const restarted = await ready(t, image) + assert.deepEqual(await queryPersons(restarted.port), []) + const status = loadCommandStatus(image.commandStatus) + const queued = loadCommandLog(status, image.commandLog) + const witness = (await rows(store.journal)).filter(e => e.event === 'response' && e.http === 202).map(e => e.id) + assert.deepEqual(witness, ['waiting']) + await requirement(t, 'P1', queued.map(c => c.id), witness, 'accepted execution inputs must remain recoverable') +}) + +test('PL04 unpersisted changeset rename causes explicit failure after observed done', async t => { + const store = await fixture(t) + const server = await ready(t, store) + await commit(store, server) + await crash(server.child) + const image = await copyImage(t, store) + const lost = await loseUnpublishedName(store, image, getChangesetPath(store.datafile, 'A')) + assert.equal(lost, true, 'baseline changeset publication lacks a directory barrier') + assert.match(await restartedFailure(t, image), /Missing changeset|missing changeset/) + assert.deepEqual(await expectedNames(store), ['A']) + await requirement(t, 'P2', lost, false, 'completed changeset filename must survive the supported cut') +}) + +test('PL05 default index sidecars are currently rebuildable at startup', async t => { + const store = await fixture(t, {indexSource: `export {default} from ${JSON.stringify(new URL('../src/index.mjs', import.meta.url).href)}`}) + const server = await ready(t, store) + await commit(store, server) + await crash(server.child) + const image = await copyImage(t, store) + const sidecars = (await fs.readdir(image.dir)).filter(name => /^index\.(id|offset).*\.json$/.test(name)) + assert.ok(sidecars.length >= 2) + for (const name of sidecars) await fs.rm(path.join(image.dir, name)) + const restarted = await ready(t, image) + assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), await expectedNames(store)) +}) + +test('PL06 adversarial torn status tail fails explicitly and preserves the original', async t => { + const store = await fixture(t) + const server = await ready(t, store) + await commit(store, server) + await crash(server.child) + const original = await fs.readFile(store.commandStatus) + const image = await copyImage(t, store) + // Corruption fixture, NOT an allowed loss of previously synced bytes. + await fs.appendFile(image.commandStatus, '{"command":"next","status":') + assert.match(await restartedFailure(t, image), /Invalid command status record/) + assert.deepEqual(await fs.readFile(store.commandStatus), original) +}) + +for (const [label, fault] of [ + ['PL07 changeset fsync error', {op: 'fsync', file: 'data.A.jsontag.', action: 'error'}], + ['PL08 changeset rename error', {op: 'rename', file: 'data.A.jsontag', action: 'error'}], + ['PL09 changeset write ENOSPC', {op: 'write', file: 'data.A.jsontag.', action: 'error', code: 'ENOSPC'}] +]) { + test(`${label} prevents command completion`, async t => { + const store = await fixture(t) + const server = await ready(t, store, {fault}) + assert.equal((await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}})).http, 202) + assert.equal((await completion(store, server.port, 'A')).status, 'failed') + assert.ok((await rows(store.trace)).some(e => e.op === 'injected' && e.operation === fault.op)) + assert.deepEqual(await queryPersons(server.port), []) + assert.ok(!(await readCommandStatusRecords(store)).some(e => e.status === 'done')) + }) +} + +test('PL10 a successful short changeset write can still be acknowledged as done', async t => { + const store = await fixture(t) + const server = await ready(t, store, {fault: {op: 'write', file: 'data.A.jsontag.', action: 'short-write'}}) + await commit(store, server) + assert.ok((await rows(store.trace)).some(e => e.op === 'injected' && e.action === 'short-write')) + assert.deepEqual((await queryPersons(server.port)).map(p => p.name), ['A'], 'live workers receive intact memory bytes') + await crash(server.child) + const image = await copyImage(t, store) + assert.match(await restartedFailure(t, image), /truncated|record length|record payload/i) + await requirement(t, 'P3', (await expectedNames(store)).length, 0, 'a partial changeset write must not be acknowledged as complete') +}) + +test('PL11 done-status datasync failure exits with an uncertain on-disk outcome', async t => { + const store = await fixture(t) + const server = await ready(t, store, {fault: {op: 'datasync', file: 'command-status.jsontag', contains: '"status":"done"', action: 'error'}}) + assert.equal((await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}})).http, 202) + const exit = await waitForExit(server.child) + assert.notEqual(exit.code, 0) + assert.match(server.getOutput(), /Injected EIO/) + const trace = await rows(store.trace) + const injection = trace.findIndex(e => e.op === 'injected') + assert.ok(injection >= 0) + assert.ok(!trace.slice(injection + 1).some(e => e.file === store.commandStatus && e.op === 'datasync')) + assert.deepEqual(await expectedNames(store), []) + // The written done bytes may nevertheless exist. Failure is not proof of rollback. + assert.ok((await readCommandStatusRecords(store)).some(e => e.status === 'done')) +}) + +test('PL12 loss before done sync causes automatic replay of an external effect', async t => { + const store = await fixture(t) + const effects = path.join(store.audit, 'effects.jsonl') + await fs.writeFile(effects, '') + await fs.writeFile(store.commandsFile, `import fs from 'node:fs' +export default { addPerson(data, command) { + fs.appendFileSync(${JSON.stringify(effects)}, JSON.stringify(command.id) + '\\n') + data.persons.push(command.value) +} }`) + // Persisted initial names are part of the fixture's starting state. + await fs.writeFile(store.commandStatus, '') + const server = await ready(t, store, {fault: {op: 'datasync', file: 'command-status.jsontag', contains: '"status":"done"', action: 'pause'}}) + assert.equal((await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}})).http, 202) + await eventually(async () => (await rows(store.trace)).some(e => e.op === 'injected'), 'pause before done sync') + await crash(server.child) + const trace = await rows(store.trace) + const stable = trace.findLast(e => e.file === store.commandStatus && e.op === 'datasync') + assert.ok(stable) + assert.doesNotMatch(Buffer.from(stable.bytes, 'base64').toString(), /"status":"done"/) + const image = await copyImage(t, store) + await fs.writeFile(image.commandStatus, Buffer.from(stable.bytes, 'base64')) + await loseUnpublishedName(store, image, getChangesetPath(store.datafile, 'A')) + assert.deepEqual(await rows(effects), ['A']) + const restarted = await ready(t, image) + assert.equal((await completion(image, restarted.port, 'A')).status, 'done') + assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), ['A']) + await requirement(t, 'P7', await rows(effects), ['A'], 'uncertain execution needs administrator approval before repeating external effects') +}) + +test('PL13 recovery queue ignores later accepted datasets (policy fixture)', async t => { + const store = await fixture(t) + const server = await ready(t, store) + await commit(store, server, 'C') + await crash(server.child) + const image = await copyImage(t, store) + const commands = [{id: 'B', name: 'addPerson', value: {name: 'B'}}, {id: 'C', name: 'addPerson', value: {name: 'C'}}] + await fs.writeFile(image.commandLog, commands.map(c => JSONTag.stringify(c)).join('\n') + '\n') + // Synthetic administrative input: do not claim this is an observed power-cut image. + for (const laterStatus of ['accepted', 'done']) { + const status = new Map([['B', {status: 'accepted'}], ['C', {status: laterStatus}]]) + assert.ok((await fs.stat(getChangesetPath(image.datafile, 'C'))).isFile()) + const queued = loadCommandLog(status, image.commandLog) + await requirement(t, 'P6', queued.some(c => c.id === 'B'), false, + `block B when later C has a dataset and status ${laterStatus}`) + } +}) + +test('PL14 complete command body survives JSONTag logging and queue reconstruction', async t => { + const store = await fixture(t) + const command = {id: 'tagged', name: 'addPerson', value: {name: 'caf\u00e9 \u6f22\u5b57'}, + when: new Date('2026-09-19T10:00:00.000Z'), nested: {zero: 0, flag: false, values: [null, 'x']}} + const server = await ready(t, store) + assert.equal((await submit(store, server.port, command)).http, 202) + assert.equal((await completion(store, server.port, command.id)).status, 'done') + await crash(server.child) + const queued = loadCommandLog(new Map([[command.id, {status: 'accepted'}]]), store.commandLog) + assert.equal(queued.length, 1) + assert.equal(JSONTag.stringify(JSONTag.parse(queued[0].command)), JSONTag.stringify(command)) +}) + +test('PL15 configured worker can use request inputs that are absent from the log', async t => { + const store = await fixture(t, {commandsSource: 'export default { addPerson(data, command, request) { data.persons.push({name: request.query.name}) } }'}) + const worker = path.join(store.dir, 'request-worker.mjs') + await fs.writeFile(worker, `import {parentPort} from 'node:worker_threads' +import runCommand, {initialize} from ${JSON.stringify(new URL('../src/command-worker-module.mjs', import.meta.url).href)} +parentPort.on('message', async task => { + await initialize(task) + parentPort.postMessage(await runCommand(task.command, task.request)) +})`) + const server = await ready(t, store, {server: {commandWorker: worker}}) + assert.equal((await submit(store, server.port, {id: 'request', name: 'addPerson'}, '?name=from-request')).http, 202) + assert.equal((await completion(store, server.port, 'request')).status, 'done') + assert.deepEqual((await queryPersons(server.port)).map(p => p.name), ['from-request']) + await crash(server.child) + const [queued] = loadCommandLog(new Map([['request', {status: 'accepted'}]]), store.commandLog) + assert.equal((await rows(store.journal)).find(e => e.event === 'issued').query, '?name=from-request') + await requirement(t, 'P8', queued.request?.query?.name, 'from-request', 'recovery must retain the request input used by the configured worker') +}) + +test('PL16 concurrent acceptance can reorder waiting commands on restart', async t => { + const store = await fixture(t, {commandsSource: `export default { + block() { while (true) {} }, + addPerson(data, command) { data.persons.push(command.value) } +}`}) + const server = await ready(t, store, {fault: {op: 'datasync:after', file: 'command-log.jsontag', contains: '"id":"A"', action: 'pause'}, + server: {maxCommandCrashAttempts: 1}}) + assert.equal((await submit(store, server.port, {id: 'block', name: 'block'})).http, 202) + const a = submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}}) + await eventually(async () => (await rows(store.trace)).some(e => e.op === 'injected'), 'A log completion gate') + assert.equal((await submit(store, server.port, {id: 'B', name: 'addPerson', value: {name: 'B'}})).http, 202) + await fs.writeFile(store.release, '') + assert.equal((await a).http, 202) + const accepted = (await rows(store.journal)).filter(e => e.event === 'response' && e.id !== 'block').map(e => e.id) + assert.deepEqual(accepted, ['B', 'A'], 'external observer records acceptance independently') + assert.deepEqual((await readCommandLogRecords(store)).map(c => c.id), ['block', 'A', 'B']) + assert.deepEqual((await readCommandStatusRecords(store)).filter(c => c.status === 'accepted' && c.command !== 'block').map(c => c.command), accepted) + await crash(server.child) + const image = await copyImage(t, store) + const restarted = await ready(t, image, {server: {maxCommandCrashAttempts: 1}}) + assert.equal((await completion(image, restarted.port, 'B')).status, 'done') + assert.equal((await completion(image, restarted.port, 'A')).status, 'done') + const actual = (await queryPersons(restarted.port)).map(p => p.name) + await requirement(t, 'P4/P6', actual, accepted, 'restart must preserve established waiting-queue order') +}) + +test('PL17 failed startup reclassifies active work before discovering missing committed data', async t => { + const store = await fixture(t) + await fs.writeFile(store.commandStatus, [ + {command: 'A', status: 'active', attempt: 1}, + {command: 'C', status: 'done', code: 200} + ].map(c => JSONTag.stringify(c)).join('\n') + '\n') + const before = await fs.readFile(store.commandStatus, 'utf8') + assert.match(await restartedFailure(t, store), /Missing changeset|missing changeset/) + assert.ok((await fs.readFile(store.commandStatus, 'utf8')).startsWith(before), 'original records remain as an append-only prefix') + await requirement(t, 'P7', loadCommandStatus(store.commandStatus).get('A').status, 'active', + 'uncertain active work must not become replayable before administrator review') +}) + +for (const [id, file, op] of [ + ['PL18', 'command-log.jsontag', 'datasync'], + ['PL19', 'command-status.jsontag', 'datasync'], + ['PL20', 'command-log.jsontag', 'close:after'] +]) { + test(`${id} ${file} ${op} failure prevents acceptance`, async t => { + const store = await fixture(t) + const server = await ready(t, store, {fault: {op, file, action: 'error'}}) + const response = await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}}) + assert.equal(response.http, 500) + assert.ok((await rows(store.trace)).some(e => e.op === 'injected' && e.operation === op)) + assert.deepEqual(await queryPersons(server.port), []) + assert.ok(!(await rows(store.journal)).some(e => e.event === 'response' && e.http === 202)) + }) +} + +test('PL21 model control: an explicit directory sync prevents modeled filename loss', async t => { + const store = await fixture(t, {indexSource: `import fs from 'node:fs/promises' +import index from ${JSON.stringify(new URL('../src/index.mjs', import.meta.url).href)} +export default { update() {}, async finalize(bytes, meta, id) { + await index.finalize(bytes, meta, id) + const directory = await fs.open(meta.data, 'r') + try { await directory.sync() } finally { await directory.close() } +} }`}) + const server = await ready(t, store) + await commit(store, server) + await crash(server.child) + const image = await copyImage(t, store) + assert.equal(await loseUnpublishedName(store, image, getChangesetPath(store.datafile, 'A')), false) + assert.equal(await loseUnpublishedName(store, image, store.commandStatus), false) + const restarted = await ready(t, image) + assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), await expectedNames(store)) +}) diff --git a/test/power-loss-helpers.mjs b/test/power-loss-helpers.mjs new file mode 100644 index 0000000..1d60d13 --- /dev/null +++ b/test/power-loss-helpers.mjs @@ -0,0 +1,118 @@ +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import process from 'node:process' +import {spawn} from 'node:child_process' +import {fileURLToPath} from 'node:url' +import JSONTag from '@muze-nl/jsontag' +import {makeServerFixture, rootDir, stopServer, waitForExit, getCommandStatus} from './durability-helpers.mjs' + +export async function fixture(t, options = {}) { + const store = await makeServerFixture(t, options) + const audit = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-observer-')) + t.after(() => fs.rm(audit, {recursive: true, force: true})) + const trace = path.join(audit, 'io.jsonl') + const journal = path.join(audit, 'client.jsonl') + const release = path.join(audit, 'release') + await fs.writeFile(trace, '') + await fs.writeFile(journal, '') + return {...store, audit, trace, journal, release} +} + +export async function rows(file) { + return (await fs.readFile(file, 'utf8')).split('\n').filter(Boolean).map(line => JSON.parse(line)) +} + +export async function observe(store, event) { + await fs.appendFile(store.journal, JSON.stringify(event) + '\n') +} + +export async function submit(store, port, command, query = '') { + await observe(store, {event: 'issued', command, query}) + const response = await fetch(`http://127.0.0.1:${port}/command${query}`, { + method: 'POST', headers: {accept: 'application/json', 'content-type': 'application/jsontag'}, + body: JSONTag.stringify(command), signal: AbortSignal.timeout(5000) + }) + const text = await response.text() + let body + try { body = JSON.parse(text) } catch { body = {raw: text} } + await observe(store, {event: 'response', id: command.id, http: response.status, body}) + return {http: response.status, body} +} + +export async function eventually(fn, description, timeout = 6000) { + const end = Date.now() + timeout + while (Date.now() < end) { + const result = await fn() + if (result) return result + await new Promise(resolve => setTimeout(resolve, 10)) + } + assert.fail(`Timed out: ${description}`) +} + +export async function completion(store, port, id) { + const result = await eventually(async () => { + const status = await getCommandStatus(port, id) + return ['done', 'failed', 'unsafe'].includes(status.status) && status + }, `completion of ${id}`) + await observe(store, {event: 'completion', id, body: result}) + return result +} + +export async function launch(t, store, port, options = {}) { + const config = path.join(store.audit, `config-${Math.random().toString(16).slice(2)}.json`) + await fs.writeFile(config, JSON.stringify({root: store.dir, trace: store.trace, + release: store.release, fault: options.fault})) + const child = spawn(process.execPath, ['--import', fileURLToPath(new URL('./power-loss-preload.mjs', import.meta.url)), store.runner], { + cwd: store.dir, + env: {...process.env, SIMPLYSTORE_ENV: 'test', SIMPLYSTORE_FAULT_POINT: '', + SIMPLYSTORE_BASELINE_IO: config, + SIMPLYSTORE_TEST_OPTIONS: JSON.stringify({ + port, datafile: store.datafile, commandsFile: store.commandsFile, + indexFile: store.indexFile, commandLog: store.commandLog, + commandStatus: store.commandStatus, wwwroot: path.join(rootDir, 'www'), + maxWorkers: 1, commandTimeout: 10000, ...options.server + })}, + stdio: ['ignore', 'pipe', 'pipe'] + }) + let output = '' + child.stdout.on('data', data => { output += data }) + child.stderr.on('data', data => { output += data }) + t.after(() => stopServer(child)) + return {child, getOutput: () => output} +} + +export async function crash(child) { + child.kill('SIGKILL') + const exit = await waitForExit(child) + assert.equal(exit.signal, 'SIGKILL') +} + +export async function copyImage(t, store) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-crash-image-')) + t.after(() => fs.rm(dir, {recursive: true, force: true})) + await fs.cp(store.dir, dir, {recursive: true}) + const image = {...store, dir} + for (const key of ['datafile', 'commandsFile', 'indexFile', 'commandLog', 'commandStatus', 'runner']) { + image[key] = path.join(dir, path.basename(store[key])) + } + return image +} + +export async function loseUnpublishedName(store, image, original) { + const trace = await rows(store.trace) + const publication = trace.findLastIndex(e => e.file === original && + (e.op === 'rename' || (e.op === 'open' && !e.existed))) + assert.ok(publication >= 0, `trace contains publication of ${original}`) + const durable = trace.slice(publication + 1).some(e => + e.file === path.dirname(original) && e.op === 'sync') + if (!durable) await fs.rm(path.join(image.dir, path.basename(original))) + return !durable +} + +export async function requirement(t, id, actual, expected, message) { + await t.test(`${id}: ${message}`, { + todo: process.env.SIMPLYSTORE_BASELINE_STRICT === '1' ? false : 'Known baseline violation; run test:power-loss:strict' + }, () => assert.deepEqual(actual, expected, message)) +} diff --git a/test/power-loss-preload.mjs b/test/power-loss-preload.mjs new file mode 100644 index 0000000..6062119 --- /dev/null +++ b/test/power-loss-preload.mjs @@ -0,0 +1,129 @@ +// Test-only I/O observation/injection. Loaded explicitly in disposable servers. +import fs from 'node:fs' +import fsp from 'node:fs/promises' +import path from 'node:path' +import process from 'node:process' +import {threadId} from 'node:worker_threads' +import {syncBuiltinESMExports} from 'node:module' + +const configFile = process.env.SIMPLYSTORE_BASELINE_IO +if (configFile) { + const config = JSON.parse(fs.readFileSync(configFile, 'utf8')) + const root = path.resolve(config.root) + const raw = Object.fromEntries(['open', 'write', 'fsync', 'close', 'rename', + 'appendFileSync', 'readFileSync', 'existsSync', 'statSync', 'writeFileSync'] + .map(key => [key, fs[key].bind(fs)])) + const open = fsp.open.bind(fsp) + const handles = new Map() + let injected = false + const scoped = file => typeof file === 'string' && + (file === root || file.startsWith(root + path.sep)) + function record(op, file, extra = {}) { + if (scoped(file)) raw.appendFileSync(config.trace, + JSON.stringify({op, file, threadId, ...extra}) + '\n') + } + function snapshot(file) { + return raw.statSync(file).isDirectory() ? {} : {bytes: raw.readFileSync(file).toString('base64')} + } + async function fault(op, file, content = '') { + const rule = config.fault + if (injected || !rule || rule.op !== op || !scoped(file) || + !path.basename(file).startsWith(rule.file) || + (rule.contains && !content.includes(rule.contains))) return false + injected = true + record('injected', file, {operation: op, action: rule.action}) + if (rule.action === 'pause') { + const deadline = Date.now() + 15000 + while (!raw.existsSync(config.release)) { + if (Date.now() > deadline) throw new Error('Baseline gate was not released') + await new Promise(resolve => setTimeout(resolve, 5)) + } + } else if (rule.action === 'short-write') { + return true + } else { + throw Object.assign(new Error(`Injected ${rule.code || 'EIO'} at ${op}`), {code: rule.code || 'EIO'}) + } + return false + } + + fsp.open = async function(file, ...args) { + const absolute = path.resolve(String(file)) + const existed = raw.existsSync(absolute) + const handle = await open(file, ...args) + if (!scoped(absolute)) return handle + record('open', absolute, {existed, flags: args[0]}) + let content = '' + for (const method of ['appendFile', 'datasync', 'sync', 'close']) { + const original = handle[method].bind(handle) + handle[method] = async (...params) => { + if (method === 'appendFile') content = String(params[0]) + await fault(method, absolute, content) + const result = await original(...params) + record(method, absolute, method === 'datasync' || method === 'sync' + ? snapshot(absolute) : {}) + await fault(method + ':after', absolute, content) + return result + } + } + return handle + } + fs.open = function(file, ...args) { + const callback = args.pop() + const absolute = path.resolve(String(file)) + const existed = raw.existsSync(absolute) + return raw.open(file, ...args, (error, fd) => { + if (!error && scoped(absolute)) { + handles.set(fd, absolute) + record('open', absolute, {existed, flags: args[0]}) + } + callback(error, fd) + }) + } + fs.write = function(fd, ...args) { + const callback = args.pop() + const file = handles.get(fd) + void fault('write', file).then(short => { + if (short) { + // The fixture targets write-file-atomic's Uint8Array overload. + if (!ArrayBuffer.isView(args[0]) || typeof args[2] !== 'number') { + throw new Error('Unsupported short-write fixture') + } + args[2] = Math.max(1, Math.floor(args[2] / 2)) + } + raw.write(fd, ...args, (error, bytes, buffer) => { + if (!error) record('write', file, {bytesWritten: bytes}) + callback(error, bytes, buffer) + }) + }).catch(callback) + } + fs.fsync = function(fd, callback) { + const file = handles.get(fd) + void fault('fsync', file).then(() => raw.fsync(fd, error => { + if (!error && scoped(file)) record('sync', file, snapshot(file)) + callback(error) + })).catch(callback) + } + fs.close = function(fd, callback) { + const file = handles.get(fd) + return raw.close(fd, error => { + if (!error) { + record('close', file) + handles.delete(fd) + } + callback(error) + }) + } + fs.rename = function(from, to, callback) { + const file = path.resolve(String(to)) + void fault('rename', file).then(() => raw.rename(from, to, error => { + if (!error) record('rename', file, {from: path.resolve(String(from))}) + callback(error) + })).catch(callback) + } + fs.writeFileSync = function(file, ...args) { + const result = raw.writeFileSync(file, ...args) + record('writeFileSync', path.resolve(String(file))) + return result + } + syncBuiltinESMExports() +} From 40f87397774e08b6549fc95dcd966922ddbaa7e9 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 12:30:41 +0200 Subject: [PATCH 07/37] Record power-loss baseline failures and recovery evidence --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 27 +++++- .spiral/evidence/EVD-20260919-TTZ7C-23.md | 107 +++++++++++++++++++++ .spiral/evidence/EVD-20260919-TTZ7C-23.ttl | 11 +++ .spiral/project-context.md | 2 + 4 files changed, 143 insertions(+), 4 deletions(-) create mode 100644 .spiral/evidence/EVD-20260919-TTZ7C-23.md create mode 100644 .spiral/evidence/EVD-20260919-TTZ7C-23.ttl diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index 317ed50..5b828b3 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -8,8 +8,8 @@ untracked `package.json~` is unrelated and remains untouched. Status: **Active**. The maintainer committed to this cycle on 2026-09-19: "Lets make a cycle to do all this work." This accepts the proposed work boundary, -not the eventual runtime result. Creation records the plan; implementation and -acceptance evidence remain outstanding. +not the eventual runtime result. The first baseline test slice is implemented +and evidenced; runtime hardening and cycle acceptance remain outstanding. ## Analyze @@ -141,8 +141,27 @@ Cycle opening captured source, understanding, and request as separate immutable causal commits, with companion Turtle references to prior versions. Project context and repository instructions now point to this refinement. -Next executable work: the failure matrix and baseline tests in step 1. -No new runtime implementation or power-loss test results exist at opening. +### Baseline slice β€” 2026-09-19 + +DES-20260919-TTZ7C-21 defines an independent observer and test-only I/O fault +harness. IMP-20260919-TTZ7C-22 implements 21 scenarios plus the matrix in +`docs/power-loss-baseline.md`; EVD-20260919-TTZ7C-23 records results and limits. + +`npm run test:power-loss` passes its 21 scenario/control tests and explicitly +reports 10 violated invariant assertions as TODO. Strict mode exposes those +failures with exit 1. Existing regression suites pass 59/59. Runtime behavior +is unchanged; this is evidence of gaps, not a power-loss safety claim. + +The failures cover lost log/status/changeset names, acknowledged partial writes, +repeated external effects, absent later-dataset rerun guards, configured-worker +request input loss, changed waiting-queue order after restart, and automatic +reclassification of uncertain work before failed startup. Positive directory-sync +and normal-restart controls distinguish the modeled barrier conditions. + +Next executable work: step 2's durable-publication/error and administrative +recovery design, guided by this matrix. Additional initial-file/integrity cases, +writer ownership, administration, backup/restore, and real filesystem evidence +remain open; the matrix explicitly identifies untested rows. ## Evaluate diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-23.md b/.spiral/evidence/EVD-20260919-TTZ7C-23.md new file mode 100644 index 0000000..fa820da --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-23.md @@ -0,0 +1,107 @@ +# Evidence: Power-Loss Baseline And Recovery Gaps + +## Claim and implementation + +IMP-20260919-TTZ7C-22 at `0766186d24e973c47bc78865c3b71a6272effde0` +implements the independent baseline harness and fault matrix. This verifies the +first work item of CYC-20260919-TTZ7C-17, not completion of the durability cycle. +The unchanged runtime baseline remains +`44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. + +## Method and environment + +Executed on 2026-09-19 using Node v24.7.0 on Linux with the installed dependencies. +Tests bind localhost on ephemeral ports and use disposable store/observer +directories. The server preloader traces actual current file API calls in both +main and worker threads. Client observations and side-effect witnesses remain +outside the copied crash images. No production data was changed. + +The tests model selected permitted cache/namespace-loss outcomes after SIGKILL, +inject actual API failures/partial writes, and exercise explicit adversarial and +policy fixtures. They do not cut physical power or test the deployment filesystem. +`docs/power-loss-baseline.md` records each distinction and uncovered matrix rows. + +## Commands and results + +| Check | Result | +|---|---| +| `npm run test:power-loss` | Exit 0; 21 scenario tests pass, 10 nested invariant assertions fail as explicit TODO; 31 total, no unexpected failures | +| `npm run test:power-loss:strict` | Expected exit 1; the same 10 invariant failures become ordinary failures; 9 enclosing scenarios also fail, giving Node totals of 12 pass / 19 fail / 31 tests | +| Existing startup/offset, ACID, crash, and handler suites | 59/59 pass | +| ESLint on all three new test files | Pass | +| JavaScript syntax checks and `git diff --check` | Pass | +| Staged provenance validation before design and implementation commits | Pass | + +The regression command was: + +```sh +node --test --test-isolation=none --test-reporter=tap \ + test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs \ + test/acid-baseline.mjs test/durability-crash.mjs test/after-change-handlers.mjs +``` + +Strict-mode failures were inspected: all top-level failures are `subtestsFailed` +from the named requirement assertions, not fixture/instrumentation failures. +The baseline and strict final runs took about 5.9 seconds each; this is test +duration, not a production commit-latency measurement. + +## Reproduced gaps + +1. **PL02 / P2:** after external observation of `done`, dropping the unsynced new + status filename makes restart serve `[]` instead of independently expected + `['A']`. The surviving log alone does not detect this silent rollback. +2. **PL03 / P1:** dropping the unsynced new command-log filename leaves accepted + work without a reconstructible task; server still starts. +3. **PL04 / P2:** losing an unsynced changeset rename after `done` produces an + explicit missing-changeset startup failure rather than recovery of the commit. +4. **PL10 / P3:** a callback write that writes only half the changeset succeeds; + the installed atomic-write dependency does not complete the remaining bytes. + The live server receives intact memory bytes and reports `done`, but startup + subsequently rejects the truncated disk file. This is actual short-write + injection, not post-commit tampering with a complete file. +5. **PL12 / P7:** after an external effect and a pause before done-status sync, + restore the last synced status bytes and lose the unpublished changeset name. + Restart automatically repeats the effect: witness `['A', 'A']` while the + dataset contains one transition. No administrator authorized the second call. +6. **PL13 / P6 (two assertions):** synthetic B-accepted/missing and C-with-dataset + policy fixtures still queue B whether C's current status is accepted or done. +7. **PL15 / P8:** a configured worker forwards request query input to its handler; + live execution uses `name=from-request`, while reconstructed task request is + null. The ordinary JSONTag command body itself passes the PL14 round-trip + control. This qualifies the earlier FIXME speculation with actual evidence. +8. **PL16 / P4/P6:** controlled concurrent acceptance produces external acceptance + and waiting-queue order B/A, but log order A/B. After a crash, recovery executes + A/B. The observer's order is independent of the log being checked. +9. **PL17 / P7:** startup reclassifies uncertain active A as accepted before + failing on missing committed C. Original records remain intact as an append + prefix; the finding is unauthorized replay eligibility, not erased history. + +## Passing controls and useful bounds + +- PL01 traces real append/data-sync and worker temp-file-sync/rename order and + verifies normal restart against independent expected values. +- PL21 supplies an explicit directory sync in a test-only finalizer. The trace + observes it, the model preserves the published filenames, and restart matches + the external expectation. The loss model distinguishes presence of a barrier. +- PL05 shows current default startup reconstructs without ID/offset sidecars; + this does not classify custom consumers or future lazy-index loading. +- PL06 rejects a torn status tail while preserving the untouched original store. +- PL07/08/09 reject changeset fsync, rename, and write errors before completion. +- PL11 exits on done-status sync failure while written done bytes remain visible: + an I/O failure is not proof that the command rolled back. +- PL18/19/20 return HTTP 500 rather than acceptance after log/status sync or + post-close errors. + +## Consequences and remaining work + +Baseline investigation is complete for the named matrix rows. Design the durable +publication/error-handling changes and ordered administrator recovery against +these failures before altering runtime behavior. Retain immutable failure +evidence and convert TODO invariants into ordinary passing tests as fixes land. + +Initial conversion publication, integrity combinations, additional I/O cases, +single-writer enforcement, administrator preview/apply and stale-plan guards, +allowed trailing recovery, backup/restore, latency, and the disposable block-level +filesystem exercise remain open. Full rebuild remains the recorded follow-up. +Neither passing controls nor expected failures establish actual power-loss +resilience or cycle acceptance. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-23.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-23.ttl new file mode 100644 index 0000000..b6dd11a --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-23.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260919-TTZ7C-23 a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260919-TTZ7C-23" ; + sd:repositoryPath ".spiral/evidence/EVD-20260919-TTZ7C-23.md" ; + sd:status sd:Active ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-22 ; + sd:gitCommit "0766186d24e973c47bc78865c3b71a6272effde0" ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index a5f4597..f1ad97d 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -22,6 +22,8 @@ Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-L Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. +Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. No runtime fixes or filesystem power-loss guarantee are claimed. Next is the durable-publication/error and administrative recovery design; remaining test rows and the rest of the cycle stay open. + ## Intake State Status: **Complete** From f48e92e2d1c08807b44c4998384317f3c7570eae Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:09:54 +0200 Subject: [PATCH 08/37] Design durable publication and ordered input-complete command records --- .spiral/designs/DES-20260919-TTZ7C-24.md | 220 ++++++++++++++++++++++ .spiral/designs/DES-20260919-TTZ7C-24.ttl | 21 +++ 2 files changed, 241 insertions(+) create mode 100644 .spiral/designs/DES-20260919-TTZ7C-24.md create mode 100644 .spiral/designs/DES-20260919-TTZ7C-24.ttl diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md new file mode 100644 index 0000000..af8ec31 --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -0,0 +1,220 @@ +# Design: Durable Publication And Ordered Command Records + +## Status, scope, and evidence + +Design step of CYC-20260919-TTZ7C-17, following REQ-20260919-TTZ7C-20 and the +failures in EVD-20260919-TTZ7C-23. This document specifies intended behavior; +it does not claim implementation or cycle acceptance. It continues the active +cycle's agreed refinement of the durability roadmap. + +The storage primitives can be implemented without changing OD-JSONTag bytes. +The versioned command-record proposal below is an explicit compatibility +decision for review before that format is implemented or deployed. There is no +authorization here to migrate existing user data silently. + +## Durable file operations + +Introduce a small storage module, shared by runtime, conversion, and recovery: + +| Operation | Completion condition | +|---|---| +| `writeAll(handle, bytes)` | Loop over bytesWritten until complete; reject zero progress, invalid counts, and errors; never equate one successful write callback with a complete file | +| `publishFile(path, bytes)` | Create a unique temporary file in the same directory, write all bytes, sync file, close, rename to destination, sync parent directory, close directory | +| `appendRecord(path, bytes)` | Serialize writes per file; append a complete newline-terminated record, sync file data, close, sync parent directory, close directory | +| `syncDirectory(path)` | Open and sync the directory, await completion and close; propagate unsupported-operation and I/O errors | + +The initial implementation syncs the append parent on every completed append. +This avoids relying on a fragile existence check or process-local cache to infer +that a filename was previously durable. Optimize only after measurement and tests. +For nested new directories, make each containing entry durable up to an already +established directory. File permissions should follow existing behavior. + +File sync does not itself persist a directory entry: see the +[Linux fsync contract](https://man7.org/linux/man-pages/man2/fsync.2.html). +Use [Node FileHandle writes](https://nodejs.org/docs/latest-v24.x/api/fs.html#filehandlewritebuffer-offset-length-position) +and check their reported byte count. The design does not assume `writeFileSync` +or an atomic rename alone constitutes a persistence barrier. + +Temporary-file cleanup is best effort and must not obscure the first I/O error. +A failed open must not be masked by closing an undefined handle. Propagate close +failures as well as write/sync/rename failures. Preserve uncertain final artifacts +for inspection; do not delete a published file in an attempt to prove rollback. + +`publishFile` alone is not a multi-file transaction. Commit sequencing below +establishes which complete files belong to canonical state. Initial conversion +should publish into a new directory, with a completion descriptor written last; +do not overwrite an operating store's base independently of its other artifacts. + +## Ownership and serialization + +All mutation runs under exclusive store ownership. For the first dependency-free +implementation use atomically created lock directories, one per canonical mutable +directory, acquired in sorted path order. This deliberately treats stores sharing +the default index/log directory as conflicting. Resolve existing parent paths; +do not allow symlink aliases to create independent locks on the same directory. + +Write owner diagnostics (random token, PID, host, start time) and sync the lock +entry before any store mutation. The token must match on release; clean shutdown +removes the lock and syncs its parent. Crash leftovers require an explicit offline +administrator release after confirming no writer remains. PID age, a timeout, or +a failed signal probe alone must never authorize automatic stealing. A crash +between mkdir and metadata publication is still a lock, not an empty slot. + +This is cooperative exclusion, not protection against arbitrary file writers. +Read-only inspection does not create source lock files. Backup and recovery apply +require a stopped/quiescent source plus ownership; verify source fingerprints +after acquiring it. Tests must demonstrate competing startup/apply exclusion and +crash-leftover handling. Kernel advisory locks are an alternative, but would need +a proven Node integration; no new native dependency is chosen in this step. + +Within the owner process use one acceptance FIFO and one serialized status writer. +Worker messages do not release the command-runner slot until the parent finishes +persisting the outcome. Only one command handler runs at a time. Acceptance may +continue during a healthy running command, but storage uncertainty closes both +acceptance and execution queues. + +## Input-complete, versioned command log proposal + +Keep the HTTP command body and handler command object unchanged. Capture the +serializable execution request before logging, and deliver exactly that captured +record to both initial execution and recovery. The default fields are the current +method, original URL, and parsed query; configured capture adapters must declare +any additional required inputs. Do not serialize live request/socket objects or +indiscriminately add credentials. Missing required input rejects acceptance. + +Proposed v2 record (one JSONTag record per line; values below are illustrative): + +```json +{ + "recordVersion": 2, + "sequence": 1, + "command": {"id": "A", "name": "addPerson", "value": {"name": "Ada"}}, + "request": {"method": "POST", "url": "/command", "query": {}}, + "executionProfile": "deployment-revision-and-context-contract" +} +``` + +`command` is the original complete JSONTag value, not a merge with transport +metadata. Preserve tagged values. Workers retain their existing task.command, +task.request, data, and meta interface: the dispatcher unwraps the log record and +uses the same serialized command/request snapshot for initial and recovered work. +The default worker forwards request to runCommand; existing handler argument +positions stay unchanged. Persisted profile identity names the chosen code and +input-capture contract; recovery verifies the operator-selected deployment against +it. A hash of one entry-point file is not proof of dependency/code equivalence. + +Sequence is allocated under the acceptance FIFO and grows monotonically (safe +integer, reject overflow). Allocation, log append, accepted-status append, +in-memory status, and queue insertion share that FIFO. Only after both appends +are durable may HTTP 202 be sent. Response delivery order across clients is not +the authoritative order; durable sequence is. Append accepted/active/terminal +statuses with matching command ID and sequence. Never reorder through Map keys. +Reconstruction orders committed changesets by this sequence; queued work uses +the same order. Failed/unsafe commands can have no canonical transition without +forcing later sequences to be contiguous committed numbers. + +Duplicate IDs are checked inside the FIFO. Matching command/context retries +return the known status without another transition; conflicting retries return +an explicit conflict in v2. A log entry without accepted status is an orphan, +not executed work. An I/O error during acceptance halts admission; restart must +classify the orphan rather than silently reuse its ID/sequence. A valid durable +accepted record with a lost HTTP response is still accepted. + +## Explicit compatibility and initialization + +Versioned log records are a format extension, not a backward-compatible claim. +The choice is an envelope rather than reserved command fields or a second +request-input sidecar: it avoids field collisions and keeps execution inputs in +the command log. The tradeoff is explicit reader/writer version coordination. + +Proposed managed-store initialization writes a small versioned descriptor last, +after base, empty or imported logs/status, and required integrity metadata are +durable. It identifies format, base identity/digest, configured artifact paths, +and execution profile. Required declared files must exist at startup; do not +recreate missing logs merely because the base parses. A missing descriptor for +an explicitly configured v2 store is an error, never automatic fallback to v1. +The descriptor is static layout/initialization evidence, not another per-command +commit log or a signature against deliberate whole-store replacement. + +Reader/writer behavior for the proposed rollout: + +- Existing v1 stores remain inspectable and can serve validated reads. Do not + implicitly rewrite their logs or synthesize request context. +- V2 mutation requires explicit initialization/migration into a separate + destination and explicit activation. Enabling the v2 writer against a legacy + store fails with a migration diagnostic. Legacy mode must not advertise the + complete new durability/replay guarantee. +- The byte/directory fixes can apply to v1 writes without format change. V1 + remains a compatibility path during rollout, with the known input/replay limits + disclosed; the complete P1-P12 claim requires v2 adoption. Removing that legacy + write path or making migration mandatory is a separate public-behavior decision. +- Migration copies retained original evidence, verifies committed data, derives + order only when legacy status/execution evidence supports it, and explicitly + records unknown historical inputs. Missing request inputs must never become + invented empty objects. Incomplete historical commands cannot be rerun until + the administrator supplies independently justified inputs in the recovery plan. +- Downgrade requires restoring the untouched original or a compatible backup. + Old binaries are not claimed to understand or safely reject a v2 store; pin + the binary/configuration at activation. Do not mix v1 and v2 writes in one log. + +The migration/default-mode tradeoff needs explicit maintainer review before the +log-format rollout. It does not block implementation of byte-complete publication +and directory barriers against the existing format. + +## Command execution and completion + +1. Select the earliest eligible accepted sequence. Append/sync active before + invoking a worker, so accepted-only v2 work can be distinguished from uncertain + started work when evidence is intact. +2. Execute command and current pre-serialization index mutations. Await required + hooks, including returned promises, before final serialization; test ordering + and rejection behavior rather than silently changing handler permissions. +3. Publish the complete changeset through the durable primitive. Preserve sparse + OD-JSONTag numbering and zero-byte no-op changesets. +4. Await index finalization; publish required built-in sidecars durably. The + current default loader can rebuild ID/offset indexes, as PL05 shows, but that + is not a reason to leave promised output files half-written. Custom finalizers + that promise required external files must use equivalent barriers; awaiting + a hook alone cannot certify arbitrary external I/O. +5. If integrity is enabled, append/sync its changeset digest, including the + manifest directory barrier. Validate manifest/base requirements at startup. +6. Append/sync done. Only then update live status, publish buffers to queries, + and release the runner slot. A query-publication failure cannot undo done; + stop/reload query service from committed state instead of appending failed. + +Any required persistence failure enters a storage-failed state: close admission, +terminate active execution if necessary, stop queue advancement, and report an +uncertain outcome. Do not append a misleading failed record after a possibly +durable done. The error path must not try to keep operating through the same +failed status writer. Safe reads require separately established committed state; +the simplest first implementation may exit after surfacing the diagnostic. + +Handler errors and timeouts remain distinct from storage faults. If terminal +failed/unsafe status can be durably recorded, no canonical transition is published +and normal subsequent work may continue under the existing timeout contract. +Such commands are never automatically rerun; existing uncommitted files and +external effects still require diagnosis before administrative action. + +## Startup and implementation order + +Parse and cross-check descriptor, input log, status transitions/order, required +files, framing, and enabled integrity without changing command state. Reconstruct +only committed transitions. Missing or inconsistent committed data fails before +any recovery status append. Active or artifact-bearing uncommitted commands are +uncertain; route them to the administrative procedure, never active-to-accepted +rewrites. V2 accepted-only work can run normally only when complete validated +evidence proves it never started and no unresolved earlier work is bypassed. +Legacy uncertainty is never resolved by assuming the new protocol held then. + +Implement byte-complete publish/append and directory barriers first (PL02/03/04/ +10 plus conversion/integrity variants), then parent error handling and queue +ownership. Implement v2 logging/order only after its compatibility decision is +settled. Follow with administrative recovery under DES-20260919-TTZ7C-25. + +New tests must cover initial creation, zero-byte/short writes, close/open error +masking, every barrier, optional integrity, concurrent acceptance/duplicates, +terminal-status persistence failures, query-update failure after done, lock +exclusion, and explicit migration/downgrade behavior. The baseline harness must +trace any newly used filesystem API before it can claim barrier coverage. +Measure fsync latency and run the disposable filesystem exercise; documentation +alone establishes none of those runtime guarantees. diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.ttl b/.spiral/designs/DES-20260919-TTZ7C-24.ttl new file mode 100644 index 0000000..1939f0a --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-24.ttl @@ -0,0 +1,21 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . +@prefix culture: . + +project:DES-20260919-TTZ7C-24 a sd:Design ; + dcterms:identifier "DES-20260919-TTZ7C-24" ; + sd:repositoryPath ".spiral/designs/DES-20260919-TTZ7C-24.md" ; + sd:status sd:Active ; + sd:satisfies [ a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] ; + sd:derivedFrom [ a sd:ArtifactReference ; + sd:artifact project:EVD-20260919-TTZ7C-23 ; + sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; + sd:shapedBy [ a sd:ArtifactReference ; + sd:artifact culture:CUL-MUZE-001 ; + sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ], + [ a sd:ArtifactReference ; + sd:artifact culture:CUL-MUZE-LIB-001 ; + sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ] . From 065cc523d12c0dfb57bcff550b9fc689ee9b2291 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:12:24 +0200 Subject: [PATCH 09/37] Design guarded administrative recovery and consistent backup restoration --- .spiral/designs/DES-20260919-TTZ7C-25.md | 208 ++++++++++++++++++++++ .spiral/designs/DES-20260919-TTZ7C-25.ttl | 17 ++ 2 files changed, 225 insertions(+) create mode 100644 .spiral/designs/DES-20260919-TTZ7C-25.md create mode 100644 .spiral/designs/DES-20260919-TTZ7C-25.ttl diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md new file mode 100644 index 0000000..dc0327d --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -0,0 +1,208 @@ +# Design: Administrative Recovery And Consistent Restore + +## Status and scope + +Step 2 of CYC-20260919-TTZ7C-17. Implements the recovery intent in +REQ-20260919-TTZ7C-20 at design level, using the durable operations and ordered +records of DES-20260919-TTZ7C-24. EVD-20260919-TTZ7C-23 supplies the baseline gaps. +No recovery tool or new runtime guarantee is implemented by this document. + +Ordinary recovery preserves the meaning/order of the existing history. Rebuild +is the separately retained option to execute selected history using changed code +into a new dataset. It must not become a `--force` bypass in ordinary recovery. +Legacy-format and default-mode rollout remain the explicit compatibility review +in DES-20260919-TTZ7C-24. + +## Read-only inspection + +Proposed entry point: `node scripts/recover.mjs inspect --store --out `. +The store config identifies the base, log/status paths, integrity settings and +code profile, so non-default paths are not silently replaced by CLI defaults. +This command parses data and metadata without invoking command handlers, custom +workers, custom index modules, or their top-level imports. Existing load-worker +behavior that imports a custom index is not an acceptable inspection mechanism. + +Source must be quiescent or an immutable copy for an actionable plan. Inspection +may diagnose a live store but must label that output non-actionable. It records +source identity, file inventory, sizes and cryptographic hashes, expected missing +paths, enabled integrity state, and code/profile identity. Check inventory and +hashes before and after inspection; that check detects change but is not a +substitute for quiescence. Write plans outside the source tree; do not create +lock files, rewrite statuses, truncate logs, or generate indexes during inspection. + +For each command report ID, authoritative order/evidence, original logged inputs, +acceptance and execution attempts, final status, expected dataset path, artifact +condition, and external-effect uncertainty. Report the last verified committed +prefix and commands whose effects cannot currently be reconstructed. Failed or +unsafe commands with a valid terminal outcome have no committed delta and do not +by themselves break the canonical prefix. Missing done data does break it. + +Validate complete newline-terminated records, required fields, uniqueness and +legal transitions, sequence/ID consistency, base identity, OD-JSONTag framing, +and enabled integrity. A malformed tail remains evidence requiring intervention; +do not discard it merely because it is last. A syntactically valid log is not +proof that missing history never existed. If coverage cannot be established, +report unknown coverage and block assertions about rerun eligibility. + +## Classification and decisions + +| Evidence | Default recovery decision | +|---|---| +| Done with verified dataset and prerequisites | Reconstruct from bytes; never execute the handler | +| Done with missing dataset | Needs recovery; assess later-dataset rule and external effects | +| Accepted only, complete v2 evidence, no active attempt or dataset | Known waiting work; retain original order; include explicitly if part of an administrative recovery plan | +| Active, any uncertain attempt, or uncommitted dataset present | Administrator diagnosis; no automatic retry or automatic promotion to done | +| Failed/unsafe terminal outcome | No automatic rerun; report attempts and any surviving uncommitted artifacts | +| Existing corrupt/partial dataset | Diagnose or restore verified artifacts; do not pretend the file is missing | +| Missing/inconsistent command inputs or order | Block rerun until independent evidence resolves it; do not fabricate context | +| Valid data with missing default derived indexes | Rebuild default indexes from committed bytes on the destination; do not run mutation-capable legacy hooks | + +Restoring a missing file from a verified matching backup is not execution. It can +repair an interior gap when its identity, bytes, and corresponding commit evidence +match. A matching command ID alone is insufficient. Existing data without a +trustworthy commit outcome is not automatically committed just because it parses. + +## Exact ordinary-rerun predicate + +Use the preserved source's accepted history and complete artifact inventory, +not a target directory after files have been removed: + +1. Establish the command's position and the later accepted commands, retaining + acceptance history even if their latest status is failed/unsafe/active. +2. For an execution candidate with a missing dataset, **every later accepted + command must also have no dataset**. Presence includes zero-byte no-op files, + corrupt files, files belonging to accepted-but-not-done commands, and files + whose validity is uncertain. Unknown presence/order fails the predicate. +3. Establish a valid preceding committed state. Keep original order across the + selected trailing missing/waiting commands. Do not skip an unresolved earlier + command and execute a later command against a different starting state. +4. Require explicit approval identifying commands and assessed secondary effects. + The predicate establishes eligibility, not evidence that execution is safe. + +Thus A:data / B:missing / C:data blocks B regardless of C's current status. +A:data / B:missing / C:waiting / D:missing can admit B/C/D in order after approval. +Known terminal no-transition commands can remain terminal; an administrator must +explicitly select a rerun if reconsidering them. If an earlier failed command is +selected for rerun, the same later-dataset restriction applies. + +There is no ordinary option to delete/quarantine a later source dataset to make +an interior command eligible. Selecting an earlier backup is an explicit rollback +with its own known-loss report, not a claim to have recovered all existing history. +The full changed-code rebuild exception is described separately below. + +## Preview and apply + +The plan contains the source snapshot fingerprint, source/base and backup +identities, captured order and inputs, code/profile fingerprints, intended target, +commands to reconstruct/restore/rerun, preserved terminal commands, expected +artifacts, unresolved issues, and known/unknown loss. It states which decisions +require administrator judgment. It contains no assertion that side effects are +safe merely because a checkbox can be selected. + +Proposed application: + +```text +node scripts/recover.mjs apply --plan plan.json --to recovered-store \ + --approve-rerun B,C,D --operator --reason +``` + +1. Require a quiescent source and acquire the ownership locks specified by the + storage design. No live server queue may advance concurrently. Acquire target + ownership too; target must be new and outside the source. Reject symlink/path + overlap and destination overwrite. Plan paths do not grant arbitrary writes. +2. Rescan the full inventory, including absent expected paths and newly appearing + datasets, and verify source/backup/profile hashes. Any mismatch invalidates the + plan before command code is loaded. Recompute the later-dataset predicate; + an administrator approval cannot override it. +3. Copy/preserve source evidence and construct the verified preceding state in + the target. Never hard-link mutable source logs/files into the target. Preserve + both original records and the plan rather than editing history to look clean. +4. Publish a durable recovery authorization record in the target before invoking + any selected command. It binds plan fingerprint, operator/reason, ordered IDs, + supplied-and-justified missing legacy inputs, source evidence, and code profile. +5. Execute sequentially with the same worker/handler inputs as normal execution. + Mark each recovery attempt durably active before invoking it, and commit using + DES-20260919-TTZ7C-24. A special recovery transition must reference the durable + authorization; a normal client cannot turn done/unsafe into accepted. Preserve + logical command IDs and original sequence; attempts and recovery IDs distinguish + outcomes. No HTTP server or ordinary queue runs on the candidate during apply. +6. Stop on uncertain execution, I/O error, handler failure, or timeout. Leave the + candidate and audit record for inspection; never automatically resume an active + recovery attempt. A new plan and fresh external-effect decision are required + before a retry. Completed attempts can be reconstructed without executing them. +7. Validate the candidate's complete committed state and required metadata, and + publish a completion report durably. Mark unfinished candidates as not ready + so normal startup cannot mistake a partial recovery directory for a store. + +Ordinary recovery uses the original execution profile. Changed or unverifiable +code identity is an explicit unresolved input; it must not silently turn a rerun +into a changed-code rebuild. The tool cannot verify arbitrary transitive runtime +dependencies from one source-file hash; deployment packaging/profile identity is +part of the administrator's evidence. + +## Backup, restore, and promotion + +First implement offline consistent backup rather than live incremental snapshots: +stop the writer, acquire ownership, inventory configured base/log/status/integrity +and required artifacts, and copy them to a new destination. Record configuration +and code-profile identity without copying unrelated secrets. Durable-publish all +files, validate their hashes and committed reconstruction, and write a backup +completion manifest last, syncing its directory. Reject incomplete backups for +automatic restore. A backup that merely parses may already contain data loss; +record its source coverage and integrity limits honestly. + +Restore accepts a verified completed backup, writes a separate candidate, validates +it, and reports its recovered commands and differences from retained source +history. Unknown source history yields explicit unknown loss. Additional surviving +committed changesets may be applied only when order, base lineage, commit evidence, +and required predecessors are verified; never guess that two backups share a base. + +Promotion remains a separate explicit administrator operation. Stop the existing +service, revalidate the completed candidate/report, and switch the deployment's +store configuration to the new location while preserving the old directory and +configuration. Document a reversible service-specific sequence; do not claim +that swapping several directories is one atomic operation. The new server must +validate the chosen completed store before listening. After new commands are +accepted, rolling back to the old directory can lose those commands and requires +another explicit coverage/loss assessment. + +Lock release after a crash is offline and explicit; stale ownership does not +authorize removing data or re-executing commands. The administrator guide must +cover lock diagnostics alongside missing/corrupt artifacts and interrupted apply. + +## Rebuild exception retained for follow-up + +A future rebuild operation takes an explicit starting dataset, chosen history, +and chosen code profile, producing a new store and regenerated derived/integrity +artifacts. Its execution context communicates `mode: rebuild` and +`sideEffectsAllowed: false` to commands and relevant hooks. Ordinary recovery +does not silently set this flag, because skipping an effect that never happened +can be just as incorrect as repeating one that did. + +The flag is cooperative; audit commands/hooks and test effect suppression before +calling a profile rebuild-safe. External reads, clock/randomness, and omitted +inputs still affect reproducibility. Rebuild can deliberately differ under new +code. Full rebuild CLI, execution, and activation are outside this cycle. + +## Verification mapping and implementation sequence + +| Design boundary | Required evidence | +|---|---| +| Read-only inspection | Byte-identical source before/after, no module/handler effects, malformed-tail and missing-history diagnostics | +| Rerun predicate | PL13 both status variants, zero-byte/corrupt later files, failed/unsafe acceptance history, missing order, permitted B/C/D suffix | +| Apply binding | Changed source file, newly appearing dataset, changed code/backup, path alias/overlap, stale plan and competing writer all rejected before execution | +| Execution | PL12 effect witness unchanged until approval; approved sequence preserves inputs/order; no live queue overtake; every active/done barrier fault exercised | +| Interrupted apply | Originals intact, incomplete candidate blocked, completed attempts not repeated, active attempt requires renewed decision | +| Inputs/order | PL14/15/16 plus v1 unknown-input/order migration cases and code-profile mismatch | +| Backup/restore | Consistent backup under exclusion, incomplete backup rejection, verified reconstruction, coverage/loss report, interrupted copying, restore without rerunning handlers | +| Promotion | Candidate validation, explicit configuration switch, retained original, failures before/after activation and honest rollback limits | + +Implement inspection/classification first, then plan validation and reconstruction +on a copy, then guarded execution and interruption records, then offline backup/ +restore and operator documentation. Reuse the same ordered record decoder and +durable operations as normal execution. Keep the reference oracle independent. +Do not convert expected failures to passing tests by weakening these decisions. + +Still required for acceptance: implement and execute the above tests, record +latency and the Linux/storage envelope, run the disposable block-level power-loss +exercise, and obtain human evaluation of the entire active cycle. diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.ttl b/.spiral/designs/DES-20260919-TTZ7C-25.ttl new file mode 100644 index 0000000..9969ea3 --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-25.ttl @@ -0,0 +1,17 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260919-TTZ7C-25 a sd:Design ; + dcterms:identifier "DES-20260919-TTZ7C-25" ; + sd:repositoryPath ".spiral/designs/DES-20260919-TTZ7C-25.md" ; + sd:status sd:Active ; + sd:satisfies [ a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] ; + sd:derivedFrom [ a sd:ArtifactReference ; + sd:artifact project:EVD-20260919-TTZ7C-23 ; + sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; + sd:constrainedBy [ a sd:ArtifactReference ; + sd:artifact project:DES-20260919-TTZ7C-24 ; + sd:gitCommit "f48e92e2d1c08807b44c4998384317f3c7570eae" ] . From 6a8cd61acef708791b7d01763bd556d19475cf00 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:13:49 +0200 Subject: [PATCH 10/37] Record completed design step and explicit compatibility rollout --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 25 ++++++++++++++++++++---- .spiral/designs/DES-20260919-TTZ7C-24.md | 18 +++++++++-------- .spiral/designs/DES-20260919-TTZ7C-25.md | 4 ++-- .spiral/project-context.md | 2 +- 4 files changed, 34 insertions(+), 15 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index 5b828b3..ad97f89 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -158,10 +158,27 @@ request input loss, changed waiting-queue order after restart, and automatic reclassification of uncertain work before failed startup. Positive directory-sync and normal-restart controls distinguish the modeled barrier conditions. -Next executable work: step 2's durable-publication/error and administrative -recovery design, guided by this matrix. Additional initial-file/integrity cases, -writer ownership, administration, backup/restore, and real filesystem evidence -remain open; the matrix explicitly identifies untested rows. +### Durable-write and recovery design β€” 2026-09-19 + +DES-20260919-TTZ7C-24 specifies complete-write loops, file and directory barriers, +initial publication, ownership, serialized acceptance/status/outcome handling, +versioned input-complete command records, and a staged compatibility path. +OD-JSONTag datasets and public command payloads remain unchanged. The v2 log +format uses explicit initialization/migration into a separate store; legacy +behavior is not silently converted or advertised as satisfying the new guarantee. + +DES-20260919-TTZ7C-25 specifies read-only inspection, source-bound recovery plans, +the exact later-accepted-dataset guard, explicit execution approval, preserved +source evidence, interruption handling, offline backup/restore, and separate +promotion. Rebuild remains a follow-up contract. Both designs map baseline gaps +and remaining acceptance tests to concrete implementation responsibilities. + +Step 2 is recorded. Next executable work is step 3: implement byte-complete +durable publication/append primitives and directory barriers, including short +writes, errors, conversion, and integrity cases. Then implement ownership, +ordered records, and guarded recovery as specified. Runtime fixes, new passing +invariants, backup/restore tooling, latency, and real filesystem evidence remain +outstanding. No new test run or runtime guarantee is claimed by the design step. ## Evaluate diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md index af8ec31..6a1a5b8 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.md +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -8,9 +8,9 @@ it does not claim implementation or cycle acceptance. It continues the active cycle's agreed refinement of the durability roadmap. The storage primitives can be implemented without changing OD-JSONTag bytes. -The versioned command-record proposal below is an explicit compatibility -decision for review before that format is implemented or deployed. There is no -authorization here to migrate existing user data silently. +The versioned command-record proposal below makes the compatibility path explicit +within the authorized cycle. Existing user data must not be migrated silently; +initialization/migration and activation are explicit administrator operations. ## Durable file operations @@ -157,9 +157,10 @@ Reader/writer behavior for the proposed rollout: Old binaries are not claimed to understand or safely reject a v2 store; pin the binary/configuration at activation. Do not mix v1 and v2 writes in one log. -The migration/default-mode tradeoff needs explicit maintainer review before the -log-format rollout. It does not block implementation of byte-complete publication -and directory barriers against the existing format. +This staged compatibility path permits implementation within the committed +cycle. Changing the public default to require migration would be a different +compatibility decision and is not included here. Byte-complete publication and +directory barriers can be implemented first against the existing format. ## Command execution and completion @@ -208,8 +209,9 @@ Legacy uncertainty is never resolved by assuming the new protocol held then. Implement byte-complete publish/append and directory barriers first (PL02/03/04/ 10 plus conversion/integrity variants), then parent error handling and queue -ownership. Implement v2 logging/order only after its compatibility decision is -settled. Follow with administrative recovery under DES-20260919-TTZ7C-25. +ownership. Implement v2 logging/order with the explicit initialization and +compatibility path above. Follow with administrative recovery under +DES-20260919-TTZ7C-25. New tests must cover initial creation, zero-byte/short writes, close/open error masking, every barrier, optional integrity, concurrent acceptance/duplicates, diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md index dc0327d..980f9a0 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.md +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -10,8 +10,8 @@ No recovery tool or new runtime guarantee is implemented by this document. Ordinary recovery preserves the meaning/order of the existing history. Rebuild is the separately retained option to execute selected history using changed code into a new dataset. It must not become a `--force` bypass in ordinary recovery. -Legacy-format and default-mode rollout remain the explicit compatibility review -in DES-20260919-TTZ7C-24. +Legacy-format and default-mode behavior follow the explicit staged compatibility +path in DES-20260919-TTZ7C-24; no implicit migration or default change is included. ## Read-only inspection diff --git a/.spiral/project-context.md b/.spiral/project-context.md index f1ad97d..6ccdc32 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -22,7 +22,7 @@ Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-L Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. -Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. No runtime fixes or filesystem power-loss guarantee are claimed. Next is the durable-publication/error and administrative recovery design; remaining test rows and the rest of the cycle stay open. +Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. The design step is now recorded in `DES-20260919-TTZ7C-24` (durable publication and ordered records) and `DES-20260919-TTZ7C-25` (administrative recovery and consistent restore). No runtime fixes or filesystem power-loss guarantee are claimed. Next is implementation of complete-write loops, file/directory barriers, and error handling, followed by ordered records and recovery; remaining test rows and the rest of the cycle stay open. ## Intake State From d70a93593536236c3f318233e623872bc03bdf60 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:16:47 +0200 Subject: [PATCH 11/37] Reconcile recovery design with explicit compatibility rollout --- .spiral/designs/DES-20260919-TTZ7C-25.ttl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.ttl b/.spiral/designs/DES-20260919-TTZ7C-25.ttl index 9969ea3..7b00340 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-25.ttl @@ -14,4 +14,4 @@ project:DES-20260919-TTZ7C-25 a sd:Design ; sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; - sd:gitCommit "f48e92e2d1c08807b44c4998384317f3c7570eae" ] . + sd:gitCommit "6a8cd61acef708791b7d01763bd556d19475cf00" ] . From 67be05c0d648461666f577d808ea2747f5e324f6 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:29:10 +0200 Subject: [PATCH 12/37] Record design-only constraint to retain existing store formats --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index 6aa647d..e77b63f 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -49,6 +49,28 @@ cooperative command/hook contract, not proof that arbitrary code is sandboxed. ## Limits +## Subsequent design correction β€” 2026-09-19 + +After the first design proposed a versioned log envelope and migration, the +maintainer clarified the intended input boundary: + +> The original design goal was that commands would be the complete set of information. I think we might fix the issue by removing extra information in the command worker, instead of adding it to the log. + +The maintainer then authorized a design revision: + +> Lets see how far we can get in this cycle without a new dataformat and migration. Design only. + +This authorizes revising the cycle design, not runtime implementation in this +step. The interpretation to explore is complete command values as the execution +inputs beyond the existing dataset, removal of transport request context from +command execution, and retention of the current store file/record formats. +The assistant's earlier envelope, sequence fields, and mandatory v2 activation +were design choices, not explicit maintainer requirements. New recovery-tool +plans/reports are administrative output; they must not become a disguised +required store format or migration. Remaining limits must be stated explicitly. + +## Remaining limits + Exact recovery CLI syntax, durable command ordering, log-format implications, filesystem test configuration, and handling of ambiguous/corrupt evidence need design and executable evidence. No runtime outcome or cycle acceptance is From 682d405fd35c433565ea103c126d45f4c5156ffa Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:32:08 +0200 Subject: [PATCH 13/37] Design durable writes and complete commands within existing formats --- .spiral/designs/DES-20260919-TTZ7C-24.md | 290 +++++++++++++--------- .spiral/designs/DES-20260919-TTZ7C-24.ttl | 5 +- .spiral/sources/SRC-20260919-TTZ7C-18.md | 2 - 3 files changed, 175 insertions(+), 122 deletions(-) diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md index 6a1a5b8..17ae4e5 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.md +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -7,10 +7,18 @@ failures in EVD-20260919-TTZ7C-23. This document specifies intended behavior; it does not claim implementation or cycle acceptance. It continues the active cycle's agreed refinement of the durability roadmap. -The storage primitives can be implemented without changing OD-JSONTag bytes. -The versioned command-record proposal below makes the compatibility path explicit -within the authorized cycle. Existing user data must not be migrated silently; -initialization/migration and activation are explicit administrator operations. +The maintainer's subsequent correction in SRC-20260919-TTZ7C-18 constrains this +revision to design only, using the existing store formats without migration. +It supersedes this document's earlier v2 envelope/descriptor proposal. Neither +that proposal nor this revision has changed runtime behavior. + +Retain OD-JSONTag base/changeset bytes, filenames, command records, status records, +and the optional integrity-record schema. No envelope, sequence field, execution +profile field, required store descriptor, log conversion, or mixed-format decoder +is introduced. Existing stores receive prospective write hardening after validated +opening; historical loss or uncertainty cannot be repaired by upgrading software. +Operational locks and external recovery plans/reports are described explicitly +below and in DES-20260919-TTZ7C-25; they are not a new canonical store format. ## Durable file operations @@ -42,8 +50,9 @@ for inspection; do not delete a published file in an attempt to prove rollback. `publishFile` alone is not a multi-file transaction. Commit sequencing below establishes which complete files belong to canonical state. Initial conversion -should publish into a new directory, with a completion descriptor written last; -do not overwrite an operating store's base independently of its other artifacts. +should publish into a new directory under ownership and validate all output before +releasing ownership; do not overwrite an operating store's base independently of +its other artifacts. No completion descriptor is added to the store. ## Ownership and serialization @@ -73,100 +82,115 @@ persisting the outcome. Only one command handler runs at a time. Acceptance may continue during a healthy running command, but storage uncertainty closes both acceptance and execution queues. -## Input-complete, versioned command log proposal - -Keep the HTTP command body and handler command object unchanged. Capture the -serializable execution request before logging, and deliver exactly that captured -record to both initial execution and recovery. The default fields are the current -method, original URL, and parsed query; configured capture adapters must declare -any additional required inputs. Do not serialize live request/socket objects or -indiscriminately add credentials. Missing required input rejects acceptance. - -Proposed v2 record (one JSONTag record per line; values below are illustrative): - -```json -{ - "recordVersion": 2, - "sequence": 1, - "command": {"id": "A", "name": "addPerson", "value": {"name": "Ada"}}, - "request": {"method": "POST", "url": "/command", "query": {}}, - "executionProfile": "deployment-revision-and-context-contract" -} -``` - -`command` is the original complete JSONTag value, not a merge with transport -metadata. Preserve tagged values. Workers retain their existing task.command, -task.request, data, and meta interface: the dispatcher unwraps the log record and -uses the same serialized command/request snapshot for initial and recovered work. -The default worker forwards request to runCommand; existing handler argument -positions stay unchanged. Persisted profile identity names the chosen code and -input-capture contract; recovery verifies the operator-selected deployment against -it. A hash of one entry-point file is not proof of dependency/code equivalence. - -Sequence is allocated under the acceptance FIFO and grows monotonically (safe -integer, reject overflow). Allocation, log append, accepted-status append, -in-memory status, and queue insertion share that FIFO. Only after both appends -are durable may HTTP 202 be sent. Response delivery order across clients is not -the authoritative order; durable sequence is. Append accepted/active/terminal -statuses with matching command ID and sequence. Never reorder through Map keys. -Reconstruction orders committed changesets by this sequence; queued work uses -the same order. Failed/unsafe commands can have no canonical transition without -forcing later sequences to be contiguous committed numbers. - -Duplicate IDs are checked inside the FIFO. Matching command/context retries -return the known status without another transition; conflicting retries return -an explicit conflict in v2. A log entry without accepted status is an orphan, -not executed work. An I/O error during acceptance halts admission; restart must -classify the orphan rather than silently reuse its ID/sequence. A valid durable -accepted record with a lost HTTP response is still accepted. - -## Explicit compatibility and initialization - -Versioned log records are a format extension, not a backward-compatible claim. -The choice is an envelope rather than reserved command fields or a second -request-input sidecar: it avoids field collisions and keeps execution inputs in -the command log. The tradeoff is explicit reader/writer version coordination. - -Proposed managed-store initialization writes a small versioned descriptor last, -after base, empty or imported logs/status, and required integrity metadata are -durable. It identifies format, base identity/digest, configured artifact paths, -and execution profile. Required declared files must exist at startup; do not -recreate missing logs merely because the base parses. A missing descriptor for -an explicitly configured v2 store is an error, never automatic fallback to v1. -The descriptor is static layout/initialization evidence, not another per-command -commit log or a signature against deliberate whole-store replacement. - -Reader/writer behavior for the proposed rollout: - -- Existing v1 stores remain inspectable and can serve validated reads. Do not - implicitly rewrite their logs or synthesize request context. -- V2 mutation requires explicit initialization/migration into a separate - destination and explicit activation. Enabling the v2 writer against a legacy - store fails with a migration diagnostic. Legacy mode must not advertise the - complete new durability/replay guarantee. -- The byte/directory fixes can apply to v1 writes without format change. V1 - remains a compatibility path during rollout, with the known input/replay limits - disclosed; the complete P1-P12 claim requires v2 adoption. Removing that legacy - write path or making migration mandatory is a separate public-behavior decision. -- Migration copies retained original evidence, verifies committed data, derives - order only when legacy status/execution evidence supports it, and explicitly - records unknown historical inputs. Missing request inputs must never become - invented empty objects. Incomplete historical commands cannot be rerun until - the administrator supplies independently justified inputs in the recovery plan. -- Downgrade requires restoring the untouched original or a compatible backup. - Old binaries are not claimed to understand or safely reject a v2 store; pin - the binary/configuration at activation. Do not mix v1 and v2 writes in one log. - -This staged compatibility path permits implementation within the committed -cycle. Changing the public default to require migration would be a different -compatibility decision and is not included here. Byte-complete publication and -directory barriers can be implemented first against the existing format. +## Complete commands; no transport execution inputs + +The logged command value is the complete invocation input beyond the preceding +dataset. Keep one JSONTag command per line, with existing `id`, `name`, and +application fields. Serialize once after validation; use that exact command +snapshot for logging and initial execution, and its retained value for recovery. +Preserve tagged values; do not add server metadata or HTTP fields to the command. + +Remove request context from the server's command-worker messages and remove the +request parameter from `runCommand`. The handler call becomes +`handler(dataspace, command, undefined, metaProxy)`: keep the fourth argument's +position for compatibility, leaving the former request slot empty. Dataset and +index access remain. Queries and HTTP admission/access checks can still use the +request; command execution cannot. Applications needing a value must supply it +as part of the complete command before acceptance/logging, not through a worker +request side channel. Never infer or silently inject omitted application inputs. + +The default worker already omits request when calling `runCommand`; PL15 exposes +a configured-worker escape route, not a requirement to preserve that route. +The revised P8 test must prove that HTTP query/method/URL context is unavailable +even through a configured worker, and that command fields and dataset/index +access are equivalent during initial execution and authorized recovery. Keep the +original baseline as historical evidence; its proposed request-retention assertion +is superseded by this corrected input contract, not accepted as a product need. + +Removing transport context does not sandbox arbitrary configured code. Required +timestamps, random choices, or external lookup results must also be command data +if they affect the intended repeatable transition; application validation and +tests establish that contract. The engine cannot detect every omitted semantic +input or certify arbitrary hooks. Historical request-dependent commands remain +defective and may be unrecoverable by rerun; this cycle will diagnose them, not +invent replacement inputs or rewrite their logs. Restoring verified data can still +avoid execution. A future rebuild-mode control is separate from business inputs. + +## Ordering with existing records + +The acceptance FIFO includes duplicate-ID checking, command-log append, +accepted-status append, in-memory status, and queue insertion. Both appends and +their required directory entries must be durable before HTTP 202. The resulting +log order is also first-acceptance order and handler execution order for new work. +An in-memory ordinal can aid checks; no sequence number is stored. Response +delivery order across clients is not the order of accepted commands. + +Retain existing status fields and values: `command`, `code`, `status`, and existing +attempt/message fields where applicable. Serialize all status appends, including +active and terminal outcomes. Reconstruct and queue using an explicit ordered +history, not incidental latest-status Map iteration. Failed/unsafe terminal +commands with no committed delta remain in that history without applying data. + +For existing history, compare command-log order, first-acceptance order, and +active/done execution evidence before choosing an order. New records should agree; +older records may not (PL16). Keep repeated status/attempt records for inspection. +Identical duplicate command-log entries can reference the same logical ID once; +conflicting values for one ID require diagnosis. A verified committed prefix may +be reconstructed when its order is established even if a pending suffix is +ambiguous; ambiguity must block suffix execution and admission that would bypass +it. Do not reorder an old history to make it obey the new FIFO. If order cannot be +established from retained evidence, ordinary recovery is blocked, not migrated. + +Keep the existing client retry contract: an existing ID returns its known status +without another append or execution; fixing the concurrent duplicate race is in +scope. A new payload-conflict response policy is unnecessary for this cycle. +A log record without acceptance is an orphan; a status without its command is +inconsistent evidence. Neither may be silently executed, discarded, or reused. +Storage uncertainty halts admission even if acceptance might already be durable. + +## Opening and initialization without a descriptor + +Separate opening an existing store from explicit new-store initialization. +Opening requires the configured base, command log, and status log, plus the +manifest when integrity is configured as required. Missing files are errors, +never an invitation to create empty logs. Empty log/status pairs remain valid +for a fresh store when consistent with its other evidence; one empty log with +nonempty accepted history or unexplained changesets requires diagnosis. +Inventory configured artifact locations, including unexpected changesets. + +Initialization/conversion targets new output, acquires ownership before writing, +and writes the base, empty logs/status, required indexes and any configured +integrity manifest in their current formats. Sync and validate all output before +releasing ownership or serving requests. Crash leftovers retain the ownership +lock and require offline inspection. Release never makes an incomplete output +ready merely because some files parse. A pre-existing base with absent logs needs +an explicit administrator assessment; normal open cannot prove it has no history. + +For an existing validated store, establish file/directory barriers for the retained +base and required committed artifacts before admitting new writes, so subsequent +completion does not depend on old cached prerequisites. This syncs existing bytes; +it does not rewrite formats or prove no earlier data was lost. Custom required +artifacts must be covered. Startup cost is measurable work in P11. + +There is no on-disk flag proving a store was written by the hardened implementation. +The guarantee is prospective for operations performed by that implementation, +from a validated starting state, under its documented filesystem/ownership +envelope. Keep the deployed binary/configuration as operational evidence. An old +binary can still read these formats, but it does not provide the new ordering, +sync, or recovery safety policy; downgrading does not preserve those guarantees. + +Integrity must be explicitly configured if its continued presence is required; +the existing existence-based autodetection alone cannot detect a lost manifest. +Current digests detect covered byte mismatches, not loss of all history or a +coherent older snapshot. Without an independent retained witness, syntactically +valid truncation or replacement of a whole history may be undetectable. Neither +new software nor a static descriptor could reconstruct that missing information. ## Command execution and completion -1. Select the earliest eligible accepted sequence. Append/sync active before - invoking a worker, so accepted-only v2 work can be distinguished from uncertain - started work when evidence is intact. +1. Select the earliest eligible accepted command in established order. Append/sync + active before invoking a worker. Eligibility on restart is conservative as + specified below; absence of active is not retrospective proof of never started. 2. Execute command and current pre-serialization index mutations. Await required hooks, including returned promises, before final serialization; test ordering and rejection behavior rather than silently changing handler permissions. @@ -198,25 +222,53 @@ external effects still require diagnosis before administrative action. ## Startup and implementation order -Parse and cross-check descriptor, input log, status transitions/order, required -files, framing, and enabled integrity without changing command state. Reconstruct -only committed transitions. Missing or inconsistent committed data fails before -any recovery status append. Active or artifact-bearing uncommitted commands are -uncertain; route them to the administrative procedure, never active-to-accepted -rewrites. V2 accepted-only work can run normally only when complete validated -evidence proves it never started and no unresolved earlier work is bypassed. -Legacy uncertainty is never resolved by assuming the new protocol held then. - -Implement byte-complete publish/append and directory barriers first (PL02/03/04/ -10 plus conversion/integrity variants), then parent error handling and queue -ownership. Implement v2 logging/order with the explicit initialization and -compatibility path above. Follow with administrative recovery under -DES-20260919-TTZ7C-25. - -New tests must cover initial creation, zero-byte/short writes, close/open error -masking, every barrier, optional integrity, concurrent acceptance/duplicates, -terminal-status persistence failures, query-update failure after done, lock -exclusion, and explicit migration/downgrade behavior. The baseline harness must -trace any newly used filesystem API before it can claim barrier coverage. -Measure fsync latency and run the disposable filesystem exercise; documentation -alone establishes none of those runtime guarantees. +Parse and cross-check the input log, status transitions/order, required files, +framing, and configured integrity without changing command state or importing +mutation-capable modules. Reconstruct only established committed transitions. +Missing or inconsistent committed data fails before any recovery status append. +Active or artifact-bearing uncommitted commands are uncertain; no automatic +active-to-accepted rewrites, even below the former crash-attempt threshold. + +Because old and new records are indistinguishable, this design also requires +administrator assessment for pending accepted-only commands found on restart. +They may have run under the old retry protocol or lost old unsynced evidence. +They cannot be labeled certainly never-started solely from their current status. +The administrative plan includes waiting work explicitly and enforces the same +later-dataset predicate. Fully terminal, consistent stores can restart normally; +freshly accepted work in a healthy uninterrupted process runs normally. Recovery +of a pending suffix blocks new mutations that would overtake it. Automatic restart +resumption with durable proof of the protocol is deferred, not smuggled into a +new checkpoint format. This conservative availability tradeoff needs tests and +operator documentation; it does not require data migration. + +Implementation sequence when runtime work resumes: byte-complete publish/append +and directory barriers first (PL02/03/04/10 plus conversion/integrity), then parent +error handling/ownership, the acceptance FIFO and command-only worker boundary, +read-only startup validation, and administrative recovery under DES-20260919-TTZ7C-25. +The current user request ends at this design revision. + +New tests must cover initial creation versus open, unchanged stored schemas, +existing-store operation without conversion, short/zero writes, close/open error +masking, every barrier, optional integrity loss, concurrent acceptance/duplicates, +terminal-status persistence failures, query-update failure after done, locks, +ambiguous historical ordering, and conservative pending-command startup. PL15's +replacement must test removal of unlogged inputs, not their retention. The harness +must trace newly used filesystem APIs before claiming coverage. Measure write and +startup sync latency and run the disposable filesystem exercise. None of these +future passing results is claimed by the design. + +## Reach within this cycle + +| Acceptance rows | Design achievable with existing store formats | Boundary | +|---|---|---| +| P1-P3 | Complete writes, ordered durable acceptance/completion, explicit uncertain errors | Prospective protocol guarantee; hardware must honor the tested barriers | +| P4, P11 | Independent witness, fault matrix, named filesystem exercise, latency | Implementation and actual filesystem evidence still required | +| P5-P7, P10 | Read-only inspection, source-bound suffix recovery, preserved evidence and deliberate promotion | Unknown history/order blocks rerun; restart pending work requires administrator assessment | +| P8 | Commands are the complete invocation input; remove worker request context | Arbitrary code needs conformance testing; omitted historical inputs are not repairable by assumption | +| P9 | Offline consistent backup and restore using unchanged store files | Report backup coverage and any unknown/lost history; no guarantee of lossless repair | +| P12 | Retain separate rebuild/effect-suppression contract | Full rebuild tooling remains follow-up | + +The P1-P12 goals need no new canonical data format on this design. Acceptance +remains contingent on implementation/evidence within these stated bounds. If a +required behavior cannot be achieved within them, record the gap and revisit the +cycle with the maintainer; do not quietly introduce a format extension/migration. diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.ttl b/.spiral/designs/DES-20260919-TTZ7C-24.ttl index 1939f0a..889c933 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-24.ttl @@ -12,7 +12,10 @@ project:DES-20260919-TTZ7C-24 a sd:Design ; sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] ; sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:EVD-20260919-TTZ7C-23 ; - sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; + sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ], + [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "67be05c0d648461666f577d808ea2747f5e324f6" ] ; sd:shapedBy [ a sd:ArtifactReference ; sd:artifact culture:CUL-MUZE-001 ; sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ], diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index e77b63f..ff0e495 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -47,8 +47,6 @@ This boundary is the assistant's proposal confirmed by the final human message; it is not a quotation authored by the maintainer. The side-effect flag is a cooperative command/hook contract, not proof that arbitrary code is sandboxed. -## Limits - ## Subsequent design correction β€” 2026-09-19 After the first design proposed a versioned log envelope and migration, the From d27a8d30e987adfaf918ee1df8dfbdefdba2947f Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:37:24 +0200 Subject: [PATCH 14/37] Keep administrative recovery and cycle scope within existing store formats --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 50 +++++--- .spiral/cycles/CYC-20260919-TTZ7C-17.ttl | 5 + .spiral/designs/DES-20260919-TTZ7C-25.md | 147 +++++++++++++++++----- .spiral/designs/DES-20260919-TTZ7C-25.ttl | 2 +- .spiral/project-context.md | 2 +- 5 files changed, 158 insertions(+), 48 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index ad97f89..9dce5eb 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -61,6 +61,10 @@ REQ-20260919-TTZ7C-20 is the acceptance matrix (P1-P12), interpreted from retain human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. - Complete command inputs must be logged; demonstrated omissions are bugs. +- The subsequent maintainer clarification makes the command itself the complete + input beyond the dataset: remove transport context from command execution. + Explore this cycle using current store formats without migration. The current + step is design only; source wording is retained in SRC-20260919-TTZ7C-18. - Ordinary rerun requires administrator approval and no later accepted command with a dataset. Recover the eligible trailing sequence in original order. - Missing data is not proof that external effects did not happen. Existing @@ -89,7 +93,7 @@ human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. effects, revalidate stale plans, and record outcomes without erasing history. Cover interruption of recovery itself and normal/automatic queue interactions. 5. **Inputs and backup/restore:** test normal versus recovered invocation inputs, - correct demonstrated log omissions, diagnose incomplete legacy records, and + remove unlogged request inputs, diagnose incomplete historical commands, and exercise consistent backup creation and restoration with reported coverage. 6. **Filesystem validation and evaluation:** exercise a disposable block-level power-loss setup on a documented Linux/local filesystem, measure commit @@ -120,7 +124,10 @@ intentional retry-policy changes reflected by justified new tests. Full rebuild tooling, broad randomized/soak campaigns, arbitrary external-effect rollback or exactly-once delivery, sandbox replacement, general database/message frameworks, remote-filesystem promises, package release, and production recovery -or deployment. The focused block-level fault exercise remains in scope. +or deployment. New canonical file/record formats and migration are excluded by +the current design constraint. External administrative plans/reports are tool +outputs, not required store metadata. The focused block-level fault exercise +remains in scope. This revision does not authorize runtime changes in this step. ### Material assumptions and replan conditions @@ -130,8 +137,8 @@ Destructive tests use disposable storage only. Missing test infrastructure leave the required exercise outstanding; it does not authorize a weaker claim. Replan affected work if authoritative command order cannot be established, -necessary legacy inputs cannot be recovered, a breaking format/API change is -required without an accepted compatibility path, or the smallest safe solution +necessary historical inputs cannot be recovered, a required behavior cannot be +achieved without a format change/migration, or the smallest safe solution contradicts the stated architecture/administrator rules. Storage irrecoverability must be reported honestly, not hidden by an invented replay outcome. @@ -162,23 +169,36 @@ and normal-restart controls distinguish the modeled barrier conditions. DES-20260919-TTZ7C-24 specifies complete-write loops, file and directory barriers, initial publication, ownership, serialized acceptance/status/outcome handling, -versioned input-complete command records, and a staged compatibility path. -OD-JSONTag datasets and public command payloads remain unchanged. The v2 log -format uses explicit initialization/migration into a separate store; legacy -behavior is not silently converted or advertised as satisfying the new guarantee. +and command-only execution. After the maintainer clarified the original command +contract and requested design without a new data format/migration, the design +removed its v2 envelope, stored sequence/profile fields, and store descriptor. +Use the existing command/status/integrity records, ordered by established log +and execution evidence; new acceptance is serialized so those orders agree. DES-20260919-TTZ7C-25 specifies read-only inspection, source-bound recovery plans, the exact later-accepted-dataset guard, explicit execution approval, preserved source evidence, interruption handling, offline backup/restore, and separate -promotion. Rebuild remains a follow-up contract. Both designs map baseline gaps +promotion. Its revised authorization/audit and backup reports live outside the +canonical store, while recovery appends existing status types/fields. It no longer +requires special recovery transitions or a migration. Rebuild remains a follow-up +contract. Both designs map baseline gaps and remaining acceptance tests to concrete implementation responsibilities. -Step 2 is recorded. Next executable work is step 3: implement byte-complete -durable publication/append primitives and directory barriers, including short -writes, errors, conversion, and integrity cases. Then implement ownership, -ordered records, and guarded recovery as specified. Runtime fixes, new passing -invariants, backup/restore tooling, latency, and real filesystem evidence remain -outstanding. No new test run or runtime guarantee is claimed by the design step. +Step 2 is revised under the design-only instruction. P1-P12 remain design targets +within the existing formats. The material limits are explicit: prospective +acknowledgment guarantees cannot prove lost historical data; old ordering or +inputs may be unresolved; pending commands on restart need administrator +assessment; recovery cannot promise lossless repair without adequate surviving +evidence or a backup. PL15's requested-context-retention assertion is a superseded +interpretation, while the baseline observation remains valid evidence. Its future +replacement must verify removal of request context and command-input equivalence. + +When implementation resumes, step 3 begins with byte-complete durable publication/ +append primitives and directory barriers, including short writes, errors, +conversion, and integrity cases. Then ownership, acceptance ordering, the worker +input boundary, and guarded recovery follow. Runtime fixes, new passing invariants, +backup/restore tooling, latency, and real filesystem evidence remain outstanding. +No new test run or runtime guarantee is claimed by this design-only revision. ## Evaluate diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl index 4661c22..87004d9 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl @@ -11,4 +11,9 @@ project:CYC-20260919-TTZ7C-17 a sd:ArtifactReference ; sd:artifact project:REQ-20260919-TTZ7C-20 ; sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" + ] ; + sd:constrainedBy [ + a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "67be05c0d648461666f577d808ea2747f5e324f6" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md index 980f9a0..dbf2c84 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.md +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -4,20 +4,30 @@ Step 2 of CYC-20260919-TTZ7C-17. Implements the recovery intent in REQ-20260919-TTZ7C-20 at design level, using the durable operations and ordered -records of DES-20260919-TTZ7C-24. EVD-20260919-TTZ7C-23 supplies the baseline gaps. +history of DES-20260919-TTZ7C-24. EVD-20260919-TTZ7C-23 supplies the baseline gaps. No recovery tool or new runtime guarantee is implemented by this document. Ordinary recovery preserves the meaning/order of the existing history. Rebuild is the separately retained option to execute selected history using changed code into a new dataset. It must not become a `--force` bypass in ordinary recovery. -Legacy-format and default-mode behavior follow the explicit staged compatibility -path in DES-20260919-TTZ7C-24; no implicit migration or default change is included. +The maintainer's design-only correction retains current store files and schemas, +with no migration. This revision removes the earlier dependence on v2 records, +stored sequence/profile fields, and special recovery status types. + +Recovery plans, audit journals, and backup/completion reports are new outputs of +the administrative tools, kept outside the canonical store. Normal loading does +not need to parse them. They bind administrator actions and interrupted-tool +resumption, not ordinary dataset reconstruction. Temporary ownership locks are +operational exclusion, not a format-version marker. No required store descriptor, +new commit log, or runtime dependency on an administrative manifest is introduced. ## Read-only inspection Proposed entry point: `node scripts/recover.mjs inspect --store --out `. The store config identifies the base, log/status paths, integrity settings and -code profile, so non-default paths are not silently replaced by CLI defaults. +deployment information, so non-default paths are not silently replaced by CLI +defaults. Accept declarative configuration; do not execute server configuration +modules merely to discover paths. This command parses data and metadata without invoking command handlers, custom workers, custom index modules, or their top-level imports. Existing load-worker behavior that imports a custom index is not an acceptable inspection mechanism. @@ -37,24 +47,39 @@ prefix and commands whose effects cannot currently be reconstructed. Failed or unsafe commands with a valid terminal outcome have no committed delta and do not by themselves break the canonical prefix. Missing done data does break it. -Validate complete newline-terminated records, required fields, uniqueness and -legal transitions, sequence/ID consistency, base identity, OD-JSONTag framing, +Validate complete newline-terminated records, required fields, duplicate-ID +consistency, acceptance/attempt history, established order, base identity, +OD-JSONTag framing, and enabled integrity. A malformed tail remains evidence requiring intervention; do not discard it merely because it is last. A syntactically valid log is not proof that missing history never existed. If coverage cannot be established, report unknown coverage and block assertions about rerun eligibility. +Use command-log position, first-acceptance order, and active/done records as the +ordering evidence described in DES-20260919-TTZ7C-24. There is no stored ordinal +to fill in retroactively. An old concurrent-acceptance disagreement must be +reported with the relevant records; never sort IDs or filenames to invent order. +Independent retained logs/backup evidence may resolve the disagreement. If they +cannot, retain the diagnosis and offer evidence-qualified backup restoration. +Allow consistent committed reads only where the preceding state is established. + +Reading a command cannot establish that its implementation used no hidden input. +The plan records the administrator's evidence about the chosen code and complete +command contract. Historical request-dependent handlers cannot be repaired by +supplying a transient request or modifying the logged command during ordinary +recovery. If verified saved data is unavailable, report that rerun is blocked. + ## Classification and decisions | Evidence | Default recovery decision | |---|---| | Done with verified dataset and prerequisites | Reconstruct from bytes; never execute the handler | | Done with missing dataset | Needs recovery; assess later-dataset rule and external effects | -| Accepted only, complete v2 evidence, no active attempt or dataset | Known waiting work; retain original order; include explicitly if part of an administrative recovery plan | +| Accepted only, no active attempt or dataset | Apparently waiting work; on restart require administrator assessment because the format cannot prove the write protocol or absence of prior execution | | Active, any uncertain attempt, or uncommitted dataset present | Administrator diagnosis; no automatic retry or automatic promotion to done | | Failed/unsafe terminal outcome | No automatic rerun; report attempts and any surviving uncommitted artifacts | | Existing corrupt/partial dataset | Diagnose or restore verified artifacts; do not pretend the file is missing | -| Missing/inconsistent command inputs or order | Block rerun until independent evidence resolves it; do not fabricate context | +| Missing/inconsistent command inputs or order | Block rerun; independent evidence may establish order or confirm existing inputs, but this cycle does not inject missing inputs or rewrite a command | | Valid data with missing default derived indexes | Rebuild default indexes from committed bytes on the destination; do not run mutation-capable legacy hooks | Restoring a missing file from a verified matching backup is not execution. It can @@ -102,7 +127,7 @@ safe merely because a checkbox can be selected. Proposed application: ```text -node scripts/recover.mjs apply --plan plan.json --to recovered-store \ +node scripts/recover.mjs apply --plan plan.json --to recovered-store --audit-dir recovery-audit \ --approve-rerun B,C,D --operator --reason ``` @@ -114,31 +139,70 @@ node scripts/recover.mjs apply --plan plan.json --to recovered-store \ datasets, and verify source/backup/profile hashes. Any mismatch invalidates the plan before command code is loaded. Recompute the later-dataset predicate; an administrator approval cannot override it. + Recognized ownership locks are excluded from content fingerprints so acquiring + the tool's own lock is not a false mismatch; no data/log files are excluded. 3. Copy/preserve source evidence and construct the verified preceding state in the target. Never hard-link mutable source logs/files into the target. Preserve both original records and the plan rather than editing history to look clean. -4. Publish a durable recovery authorization record in the target before invoking - any selected command. It binds plan fingerprint, operator/reason, ordered IDs, - supplied-and-justified missing legacy inputs, source evidence, and code profile. +4. Publish a durable recovery authorization in the external audit directory before + invoking any selected command. It binds plan fingerprint, operator/reason, + ordered IDs and command hashes, source/target locations, and deployment + evidence. No missing business inputs are supplied through the tool. Keep the + immutable source copy, plan, and audit together as the recovery evidence bundle; + make all these writes durable before execution, outside canonical store paths. 5. Execute sequentially with the same worker/handler inputs as normal execution. Mark each recovery attempt durably active before invoking it, and commit using - DES-20260919-TTZ7C-24. A special recovery transition must reference the durable - authorization; a normal client cannot turn done/unsafe into accepted. Preserve - logical command IDs and original sequence; attempts and recovery IDs distinguish - outcomes. No HTTP server or ordinary queue runs on the candidate during apply. + DES-20260919-TTZ7C-24. Use the existing active/done/failed/unsafe status values, + command IDs and attempt fields; append outcomes without erasing earlier records. + Only the offline approved tool can reopen a terminal command. Link each attempt + to authorization and original order in the external audit, not through new + status fields/types. Record the planned attempt durably there before the active + append; if either write is uncertain, stop without invoking the command. No HTTP + server or ordinary queue runs on the candidate during apply. Since reopening + status is not a new schema, older readers may parse it but must not be used to + operate an unfinished candidate; they lack the new exclusion/recovery policy. 6. Stop on uncertain execution, I/O error, handler failure, or timeout. Leave the candidate and audit record for inspection; never automatically resume an active recovery attempt. A new plan and fresh external-effect decision are required before a retry. Completed attempts can be reconstructed without executing them. -7. Validate the candidate's complete committed state and required metadata, and - publish a completion report durably. Mark unfinished candidates as not ready - so normal startup cannot mistake a partial recovery directory for a store. - -Ordinary recovery uses the original execution profile. Changed or unverifiable -code identity is an explicit unresolved input; it must not silently turn a rerun -into a changed-code rebuild. The tool cannot verify arbitrary transitive runtime -dependencies from one source-file hash; deployment packaging/profile identity is -part of the administrator's evidence. +7. Verify that every selected restore/rerun has completed as approved, with no + unresolved commands or unexplained artifacts. Validate the candidate's complete + committed state and required files, sync them, + and publish the external completion report durably before releasing candidate + ownership. Unfinished candidates retain the ordinary ownership lock; normal + startup refuses that lock. Lock removal is never the generic stale-PID action: + it requires an offline assessment, and for a recovery candidate that includes + its bound plan/audit, remaining commands, and completion report. If that bundle + is lost, the tool cannot certify or automatically resume the attempt. Preserve + the candidate for diagnosis or start a new approved recovery from the source. + +The source inventory remains the guard throughout a plan. As B, C, D are recovered +in order, B's newly created target data is a predecessor, not a later source +dataset that would invalidate C. On interruption, revalidate the preserved source +and audit plus current candidate before any renewed approval. A durable target +done record with valid data can be reconstructed without rerun even if writing +its external audit outcome failed; an audit claim alone cannot establish commit. +Do not mark failed after a possibly durable done. Retrying an uncertain attempt +with an existing partial dataset remains blocked pending separate diagnosis; +automatic cleanup must not turn that artifact into apparent absence. + +Where a retained digest identifies the missing committed changeset, compare the +recreated bytes to that expectation before recording recovery completion. A +mismatch requires diagnosis; do not erase the old expectation by appending a new +digest and then claim the original state was recovered. Preserve the candidate +bytes and uncertainty. Without such a witness, structural reconstruction and a +complete-command contract cannot independently prove byte-identical historical +output; report that verification limit. + +Ordinary recovery uses the original command meaning and compatible deployment. +There is no execution-profile field in the command log. The administrator supplies +retained deployment/source/package evidence in the external plan, and the tool +binds the selected code to prevent changes after preview. A current hash proves +only what is selected now, not what executed originally. Unresolved code identity +blocks a confident ordinary rerun; a materially changed handler must not silently +become a changed-code rebuild. The engine's request-removal fix is compatible with +handlers that already obeyed the complete-command contract. It is not proof that +historical handlers which used request context can be rerun correctly. ## Backup, restore, and promotion @@ -147,8 +211,11 @@ stop the writer, acquire ownership, inventory configured base/log/status/integri and required artifacts, and copy them to a new destination. Record configuration and code-profile identity without copying unrelated secrets. Durable-publish all files, validate their hashes and committed reconstruction, and write a backup -completion manifest last, syncing its directory. Reject incomplete backups for -automatic restore. A backup that merely parses may already contain data loss; +completion manifest last in the external backup evidence bundle, syncing its +directory. The copied store retains its original format and does not require +that manifest for ordinary loading; the restore tool requires it to verify this +backup procedure completed. Reject incomplete backups for automatic restore. +A backup that merely parses may already contain data loss; record its source coverage and integrity limits honestly. Restore accepts a verified completed backup, writes a separate candidate, validates @@ -157,13 +224,23 @@ history. Unknown source history yields explicit unknown loss. Additional survivi committed changesets may be applied only when order, base lineage, commit evidence, and required predecessors are verified; never guess that two backups share a base. +Keep original command/status records as a byte-preserved prefix of a candidate +when they are complete and valid; recovery appends use existing schemas. Preserve +all originals separately in every case. Malformed logs, unresolved order, or +missing command inputs are diagnostic/backup-restore cases in this cycle, not an +automatic log-repair or migration facility. Selecting a completed older backup +can be a useful deliberate rollback, but cannot be called recovery of all later +accepted work. Missing or corrupt saved data need matching backup evidence or +an eligible rerun; the design offers no lossless solution when neither exists. + Promotion remains a separate explicit administrator operation. Stop the existing service, revalidate the completed candidate/report, and switch the deployment's store configuration to the new location while preserving the old directory and configuration. Document a reversible service-specific sequence; do not claim that swapping several directories is one atomic operation. The new server must -validate the chosen completed store before listening. After new commands are -accepted, rolling back to the old directory can lose those commands and requires +validate the chosen store before listening; the promotion tool verifies the +external completion report. Normal loading does not need that report. After new +commands are accepted, rolling back to the old directory can lose those commands and requires another explicit coverage/loss assessment. Lock release after a crash is offline and explicit; stale ownership does not @@ -193,7 +270,10 @@ code. Full rebuild CLI, execution, and activation are outside this cycle. | Apply binding | Changed source file, newly appearing dataset, changed code/backup, path alias/overlap, stale plan and competing writer all rejected before execution | | Execution | PL12 effect witness unchanged until approval; approved sequence preserves inputs/order; no live queue overtake; every active/done barrier fault exercised | | Interrupted apply | Originals intact, incomplete candidate blocked, completed attempts not repeated, active attempt requires renewed decision | -| Inputs/order | PL14/15/16 plus v1 unknown-input/order migration cases and code-profile mismatch | +| Inputs/order | PL14/16, replacement for PL15 proving request context is unavailable, ambiguous old order, incomplete historical commands, and changed/unknown deployment identity | +| Existing format | Unchanged command/status/integrity schemas, no descriptor or conversion, prior command bytes retained, accepted-only restart requires assessment | +| Audit separation | Final store loads without tool reports; apply/promotion requires its external evidence; faults between audit/active/done/report writes stop safely | +| Recreated state | Existing expected digest matches or recovery stops; absent digest reports verification limits; incomplete approved plans cannot be promoted | | Backup/restore | Consistent backup under exclusion, incomplete backup rejection, verified reconstruction, coverage/loss report, interrupted copying, restore without rerunning handlers | | Promotion | Candidate validation, explicit configuration switch, retained original, failures before/after activation and honest rollback limits | @@ -202,7 +282,12 @@ on a copy, then guarded execution and interruption records, then offline backup/ restore and operator documentation. Reuse the same ordered record decoder and durable operations as normal execution. Keep the reference oracle independent. Do not convert expected failures to passing tests by weakening these decisions. +The exception is PL15's input-retention assertion: its historical failure remains +evidence, but its interpretation is corrected by the maintainer's complete-command +boundary. Document that change and add the discriminating removal/equivalence test. Still required for acceptance: implement and execute the above tests, record latency and the Linux/storage envelope, run the disposable block-level power-loss -exercise, and obtain human evaluation of the entire active cycle. +exercise, and obtain human evaluation of the entire active cycle. This step is +design only. Unknown historical inputs/order, unavailable backup evidence, and +irrecoverable history remain explicit limits, not reasons to introduce migration. diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.ttl b/.spiral/designs/DES-20260919-TTZ7C-25.ttl index 7b00340..b30cc89 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-25.ttl @@ -14,4 +14,4 @@ project:DES-20260919-TTZ7C-25 a sd:Design ; sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; - sd:gitCommit "6a8cd61acef708791b7d01763bd556d19475cf00" ] . + sd:gitCommit "682d405fd35c433565ea103c126d45f4c5156ffa" ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 6ccdc32..a5e9bc1 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -22,7 +22,7 @@ Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-L Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. -Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. The design step is now recorded in `DES-20260919-TTZ7C-24` (durable publication and ordered records) and `DES-20260919-TTZ7C-25` (administrative recovery and consistent restore). No runtime fixes or filesystem power-loss guarantee are claimed. Next is implementation of complete-write loops, file/directory barriers, and error handling, followed by ordered records and recovery; remaining test rows and the rest of the cycle stay open. +Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. The design step is now recorded in `DES-20260919-TTZ7C-24` (durable publication and ordered records) and `DES-20260919-TTZ7C-25` (administrative recovery and consistent restore). No runtime fixes or filesystem power-loss guarantee are claimed. The maintainer subsequently requested a design-only revision retaining current store formats without migration. DES-24/25 now remove the v2 envelope, stored sequence/profile fields, and store descriptor. Commands themselves supply all execution inputs beyond the dataset; HTTP request context is removed from the designed worker contract. Acceptance serialization establishes log/execution order without stored sequence fields. Pending work on restart requires administrator assessment because existing records do not prove which protocol wrote them. Recovery audit/backup reports remain external tool outputs, not required store metadata. Historical missing inputs/order and unobservable history loss remain explicit limits. PL15's request-retention assertion is superseded by the corrected input contract; its observed baseline behavior remains evidence. This step is design only. Proposed implementation starts with complete-write loops, file/directory barriers, and error handling, then ownership/ordering and recovery; remaining test rows and the rest of the cycle stay open. ## Intake State From b064824b696868d817959473f9a8adbc492b7abe Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:38:07 +0200 Subject: [PATCH 15/37] Clarify recovery digest verification precedes commit --- .spiral/designs/DES-20260919-TTZ7C-25.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md index dbf2c84..464bc14 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.md +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -187,8 +187,8 @@ with an existing partial dataset remains blocked pending separate diagnosis; automatic cleanup must not turn that artifact into apparent absence. Where a retained digest identifies the missing committed changeset, compare the -recreated bytes to that expectation before recording recovery completion. A -mismatch requires diagnosis; do not erase the old expectation by appending a new +recreated bytes to that expectation before appending a new integrity record or +done status. A mismatch requires diagnosis; do not erase the old expectation by appending a new digest and then claim the original state was recovered. Preserve the candidate bytes and uncertainty. Without such a witness, structural reconstruction and a complete-command contract cannot independently prove byte-identical historical From 8a42ed15242c174a296eb90ed4a35b3354d79e8d Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:44:10 +0200 Subject: [PATCH 16/37] Record command log as the authoritative command order --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index ff0e495..bae5a48 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -69,6 +69,24 @@ required store format or migration. Remaining limits must be stated explicitly. ## Remaining limits +## Command-log authority clarification β€” 2026-09-19 + +The maintainer requested verification of the intended ordering rule: + +> The current system should have a single sequence of commands being processed according to the order in the command log. You should be able to rely on that order. Unless I have missed something. Please verify. + +After verification exposed a concurrency gap between append and queue insertion, +the maintainer confirmed: + +> That is a problem, the command log should be authoritive + +The command log defines the order. Status records, client responses, and queue +timing are not alternative authorities. Their disagreement is evidence of a bug +or damaged history. The previous design-only and no-migration boundary remains; +the immediate work is to correct the designs and future verification expectations. + +## Outstanding design and evidence + Exact recovery CLI syntax, durable command ordering, log-format implications, filesystem test configuration, and handling of ambiguous/corrupt evidence need design and executable evidence. No runtime outcome or cycle acceptance is From 088e389bfb2e2fa33c77c3c8d6c6b3f8af110e9d Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:44:57 +0200 Subject: [PATCH 17/37] Require execution and reconstruction to follow command-log order --- .spiral/designs/DES-20260919-TTZ7C-24.md | 43 +++++++++++++++++------ .spiral/designs/DES-20260919-TTZ7C-24.ttl | 2 +- .spiral/sources/SRC-20260919-TTZ7C-18.md | 2 -- 3 files changed, 33 insertions(+), 14 deletions(-) diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md index 17ae4e5..cc0af76 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.md +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -118,6 +118,12 @@ avoid execution. A future rebuild-mode control is separate from business inputs. ## Ordering with existing records +**Command-log position is the sole authoritative command order**, as confirmed +by the maintainer in SRC-20260919-TTZ7C-18. Acceptance, live execution, committed +reconstruction, restart queues, and the administrative later-dataset predicate +must all follow that order. Status records describe outcomes and attempts; their +insertion order and client response timing cannot redefine the command sequence. + The acceptance FIFO includes duplicate-ID checking, command-log append, accepted-status append, in-memory status, and queue insertion. Both appends and their required directory entries must be durable before HTTP 202. The resulting @@ -127,19 +133,34 @@ delivery order across clients is not the order of accepted commands. Retain existing status fields and values: `command`, `code`, `status`, and existing attempt/message fields where applicable. Serialize all status appends, including -active and terminal outcomes. Reconstruct and queue using an explicit ordered -history, not incidental latest-status Map iteration. Failed/unsafe terminal +active and terminal outcomes. Reconstruct and queue by command-log position, +not incidental latest-status Map iteration. Failed/unsafe terminal commands with no committed delta remain in that history without applying data. -For existing history, compare command-log order, first-acceptance order, and -active/done execution evidence before choosing an order. New records should agree; -older records may not (PL16). Keep repeated status/attempt records for inspection. -Identical duplicate command-log entries can reference the same logical ID once; -conflicting values for one ID require diagnosis. A verified committed prefix may -be reconstructed when its order is established even if a pending suffix is -ambiguous; ambiguity must block suffix execution and admission that would bypass -it. Do not reorder an old history to make it obey the new FIFO. If order cannot be -established from retained evidence, ordinary recovery is blocked, not migrated. +For existing history, compare first-acceptance and active/done execution evidence +against the command log to detect violations, never to choose another authority. +Older records may disagree (PL16). Keep repeated status/attempt records for +inspection. Identical duplicate command-log entries reference one logical ID at +its first occurrence; conflicting values for one ID require diagnosis. + +If only waiting-queue or acceptance-status order differs, the command order is +still known: use the complete log. That disagreement alone does not make the log +order ambiguous. Pending work still follows the administrator-assessment policy +below. If commands actually executed out of log order, their positional changesets +may encode an incompatible preceding state. Do not simply reorder those files +and claim a correct reconstruction. Diagnose the affected history, serve only a +verified unaffected prefix where appropriate, and use the guarded recovery or +backup path. Status evidence can reveal the defect; it cannot legitimize a +different command order. A missing/corrupt/incomplete command log is a separate +loss of authoritative evidence: other records cannot silently replace it. + +The revised ordering test must compare actual handler starts, completed state, +and restarted pending execution with command-log order. Exercise an A append +whose asynchronous completion is delayed while B is submitted; B must not overtake +A. Include a sequential control and the waiting-at-crash case. PL16's historical +assertion requiring restart to preserve the old B/A queue is superseded: with an +A/B log, restarting A/B is correct and the live B/A queue is the defect. Retain +that observation as evidence; do not fix the test by making restart follow B/A. Keep the existing client retry contract: an existing ID returns its known status without another append or execution; fixing the concurrent duplicate race is in diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.ttl b/.spiral/designs/DES-20260919-TTZ7C-24.ttl index 889c933..5373b82 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-24.ttl @@ -15,7 +15,7 @@ project:DES-20260919-TTZ7C-24 a sd:Design ; sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ], [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; - sd:gitCommit "67be05c0d648461666f577d808ea2747f5e324f6" ] ; + sd:gitCommit "8a42ed15242c174a296eb90ed4a35b3354d79e8d" ] ; sd:shapedBy [ a sd:ArtifactReference ; sd:artifact culture:CUL-MUZE-001 ; sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ], diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index bae5a48..1568d31 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -67,8 +67,6 @@ were design choices, not explicit maintainer requirements. New recovery-tool plans/reports are administrative output; they must not become a disguised required store format or migration. Remaining limits must be stated explicitly. -## Remaining limits - ## Command-log authority clarification β€” 2026-09-19 The maintainer requested verification of the intended ordering rule: From dc7f50c34149316c174d1b8615dc61a4ac44ed29 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:46:03 +0200 Subject: [PATCH 18/37] Align recovery guards and cycle expectations with command-log authority --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 14 ++++++++++-- .spiral/cycles/CYC-20260919-TTZ7C-17.ttl | 2 +- .spiral/designs/DES-20260919-TTZ7C-25.md | 28 +++++++++++++++-------- .spiral/designs/DES-20260919-TTZ7C-25.ttl | 2 +- .spiral/project-context.md | 2 +- 5 files changed, 33 insertions(+), 15 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index 9dce5eb..d5b5744 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -61,6 +61,8 @@ REQ-20260919-TTZ7C-20 is the acceptance matrix (P1-P12), interpreted from retain human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. - Complete command inputs must be logged; demonstrated omissions are bugs. +- Command-log order is authoritative for live execution, reconstruction, restart, + and the later-dataset recovery guard. Status/queue timing cannot redefine it. - The subsequent maintainer clarification makes the command itself the complete input beyond the dataset: remove transport context from command execution. Explore this cycle using current store formats without migration. The current @@ -172,8 +174,9 @@ initial publication, ownership, serialized acceptance/status/outcome handling, and command-only execution. After the maintainer clarified the original command contract and requested design without a new data format/migration, the design removed its v2 envelope, stored sequence/profile fields, and store descriptor. -Use the existing command/status/integrity records, ordered by established log -and execution evidence; new acceptance is serialized so those orders agree. +Use the existing command/status/integrity records, with command-log position as +the sole authoritative order. New acceptance is serialized so queue/execution +order agrees; contrary historical execution is diagnosed as a defect. DES-20260919-TTZ7C-25 specifies read-only inspection, source-bound recovery plans, the exact later-accepted-dataset guard, explicit execution approval, preserved @@ -192,6 +195,13 @@ assessment; recovery cannot promise lossless repair without adequate surviving evidence or a backup. PL15's requested-context-retention assertion is a superseded interpretation, while the baseline observation remains valid evidence. Its future replacement must verify removal of request context and command-input equivalence. +The maintainer subsequently confirmed command-log authority after verification +showed that sequential execution can still disagree with log order during +concurrent acceptance. PL16's queue-preservation assertion must likewise be +corrected: an A/B log requires A/B execution and restart, even if the old queue +was B/A. A complete log defines order; historical out-of-order changesets may +still require diagnosis because their preceding state was wrong. Runtime/tests +remain unchanged in this design-only correction. When implementation resumes, step 3 begins with byte-complete durable publication/ append primitives and directory barriers, including short writes, errors, diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl index 87004d9..e44b8ec 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl @@ -15,5 +15,5 @@ project:CYC-20260919-TTZ7C-17 sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; - sd:gitCommit "67be05c0d648461666f577d808ea2747f5e324f6" + sd:gitCommit "8a42ed15242c174a296eb90ed4a35b3354d79e8d" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md index 464bc14..88235bf 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.md +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -55,13 +55,15 @@ do not discard it merely because it is last. A syntactically valid log is not proof that missing history never existed. If coverage cannot be established, report unknown coverage and block assertions about rerun eligibility. -Use command-log position, first-acceptance order, and active/done records as the -ordering evidence described in DES-20260919-TTZ7C-24. There is no stored ordinal -to fill in retroactively. An old concurrent-acceptance disagreement must be -reported with the relevant records; never sort IDs or filenames to invent order. -Independent retained logs/backup evidence may resolve the disagreement. If they -cannot, retain the diagnosis and offer evidence-qualified backup restoration. -Allow consistent committed reads only where the preceding state is established. +Command-log position is authoritative, as specified in DES-20260919-TTZ7C-24. +Use acceptance and active/done records to detect deviations, not to select another +order. A complete log settles order even if an old waiting queue or acceptance +status sequence disagrees. If actual execution differed, diagnose whether retained +changesets depend on an incorrect preceding state; do not reorder positional +changesets and claim a valid repair. Allow committed reads only for verified +unaffected state. Independent backups can restore missing authoritative evidence +or valid data; they cannot make a conflicting status order authoritative. Missing +or damaged command-log evidence remains a separate recovery limitation. Reading a command cannot establish that its implementation used no hidden input. The plan records the administrator's evidence about the chosen code and complete @@ -92,8 +94,10 @@ trustworthy commit outcome is not automatically committed just because it parses Use the preserved source's accepted history and complete artifact inventory, not a target directory after files have been removed: -1. Establish the command's position and the later accepted commands, retaining - acceptance history even if their latest status is failed/unsafe/active. +1. Locate the command and every later accepted command by command-log position, + retaining acceptance history even if the latest status is failed/unsafe/active. + Never use acceptance-status insertion order or actual execution order to make + a later dataset appear earlier and thereby permit a rerun. 2. For an execution candidate with a missing dataset, **every later accepted command must also have no dataset**. Presence includes zero-byte no-op files, corrupt files, files belonging to accepted-but-not-done commands, and files @@ -270,7 +274,7 @@ code. Full rebuild CLI, execution, and activation are outside this cycle. | Apply binding | Changed source file, newly appearing dataset, changed code/backup, path alias/overlap, stale plan and competing writer all rejected before execution | | Execution | PL12 effect witness unchanged until approval; approved sequence preserves inputs/order; no live queue overtake; every active/done barrier fault exercised | | Interrupted apply | Originals intact, incomplete candidate blocked, completed attempts not repeated, active attempt requires renewed decision | -| Inputs/order | PL14/16, replacement for PL15 proving request context is unavailable, ambiguous old order, incomplete historical commands, and changed/unknown deployment identity | +| Inputs/order | PL14; corrected PL16 expectations comparing live/restarted execution to the log; replacement for PL15 proving request context is unavailable; missing log evidence, out-of-order historical changesets, incomplete commands, and changed/unknown deployment identity | | Existing format | Unchanged command/status/integrity schemas, no descriptor or conversion, prior command bytes retained, accepted-only restart requires assessment | | Audit separation | Final store loads without tool reports; apply/promotion requires its external evidence; faults between audit/active/done/report writes stop safely | | Recreated state | Existing expected digest matches or recovery stops; absent digest reports verification limits; incomplete approved plans cannot be promoted | @@ -285,6 +289,10 @@ Do not convert expected failures to passing tests by weakening these decisions. The exception is PL15's input-retention assertion: its historical failure remains evidence, but its interpretation is corrected by the maintainer's complete-command boundary. Document that change and add the discriminating removal/equivalence test. +PL16's assertion is also corrected: restart must follow the command log, not the +old in-memory queue. Add a predicate test with log A/B/C and conflicting status +order so C's dataset still blocks rerunning B. The live acceptance race remains +a defect to fix; the correction does not erase the reproduced ordering mismatch. Still required for acceptance: implement and execute the above tests, record latency and the Linux/storage envelope, run the disposable block-level power-loss diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.ttl b/.spiral/designs/DES-20260919-TTZ7C-25.ttl index b30cc89..477ba3b 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-25.ttl @@ -14,4 +14,4 @@ project:DES-20260919-TTZ7C-25 a sd:Design ; sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; - sd:gitCommit "682d405fd35c433565ea103c126d45f4c5156ffa" ] . + sd:gitCommit "088e389bfb2e2fa33c77c3c8d6c6b3f8af110e9d" ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index a5e9bc1..ad37f92 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -22,7 +22,7 @@ Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-L Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. -Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. The design step is now recorded in `DES-20260919-TTZ7C-24` (durable publication and ordered records) and `DES-20260919-TTZ7C-25` (administrative recovery and consistent restore). No runtime fixes or filesystem power-loss guarantee are claimed. The maintainer subsequently requested a design-only revision retaining current store formats without migration. DES-24/25 now remove the v2 envelope, stored sequence/profile fields, and store descriptor. Commands themselves supply all execution inputs beyond the dataset; HTTP request context is removed from the designed worker contract. Acceptance serialization establishes log/execution order without stored sequence fields. Pending work on restart requires administrator assessment because existing records do not prove which protocol wrote them. Recovery audit/backup reports remain external tool outputs, not required store metadata. Historical missing inputs/order and unobservable history loss remain explicit limits. PL15's request-retention assertion is superseded by the corrected input contract; its observed baseline behavior remains evidence. This step is design only. Proposed implementation starts with complete-write loops, file/directory barriers, and error handling, then ownership/ordering and recovery; remaining test rows and the rest of the cycle stay open. +Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. The design step is now recorded in `DES-20260919-TTZ7C-24` (durable publication and ordered records) and `DES-20260919-TTZ7C-25` (administrative recovery and consistent restore). No runtime fixes or filesystem power-loss guarantee are claimed. The maintainer subsequently requested a design-only revision retaining current store formats without migration. DES-24/25 now remove the v2 envelope, stored sequence/profile fields, and store descriptor. Commands themselves supply all execution inputs beyond the dataset; HTTP request context is removed from the designed worker contract. Command-log position is the sole authoritative order for live execution, reconstruction, restart, and the later-dataset guard; acceptance serialization must enforce it without stored sequence fields. Contrary status/queue order is a bug, not an alternative authority. PL16's old queue-preservation assertion must be corrected to compare execution against the log. A complete log settles order; historical out-of-order changesets can still require diagnosis because they may encode the wrong preceding state. Pending work on restart requires administrator assessment because existing records do not prove which protocol wrote them. Recovery audit/backup reports remain external tool outputs, not required store metadata. Historical missing inputs/order and unobservable history loss remain explicit limits. PL15's request-retention assertion is superseded by the corrected input contract; its observed baseline behavior remains evidence. This step is design only. Proposed implementation starts with complete-write loops, file/directory barriers, and error handling, then ownership/ordering and recovery; remaining test rows and the rest of the cycle stay open. ## Intake State From 9fecb10ba1b58c79779394130e1c5cf500206ea9 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 13:52:21 +0200 Subject: [PATCH 19/37] Record authorization to complete the power-loss recovery cycle --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index 1568d31..b6f1073 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -85,6 +85,20 @@ the immediate work is to correct the designs and future verification expectation ## Outstanding design and evidence +## Authorization to complete implementation β€” 2026-09-19 + +Following explanation of serialized acceptance through queue insertion, the +maintainer authorized continuing the entire cycle: + +> Ok, that sounds like a good fix, I'll let you continue the whole cycle with this design. Only stop when I need to accept the cycle, or you cannot resolve a question in a sensible direction yourself. + +This supersedes the earlier design-only execution limit. Continue implementation, +tests, recovery tooling, documentation and evaluation under the current designs, +with unchanged store formats and command-log authority. Cycle acceptance remains +the maintainer's decision. Routine implementation choices require no new approval. + +## Remaining acceptance evidence + Exact recovery CLI syntax, durable command ordering, log-format implications, filesystem test configuration, and handling of ambiguous/corrupt evidence need design and executable evidence. No runtime outcome or cycle acceptance is From 59179b6c93b7e21c3b2c69db46c53fbe706927c3 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 14:34:46 +0200 Subject: [PATCH 20/37] Carry corrected durability design into authorized implementation --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 10 ++++++---- .spiral/cycles/CYC-20260919-TTZ7C-17.ttl | 5 ++++- .spiral/designs/DES-20260919-TTZ7C-24.md | 8 ++++---- .spiral/designs/DES-20260919-TTZ7C-24.ttl | 5 ++++- .spiral/designs/DES-20260919-TTZ7C-25.md | 21 +++++++++++++++++++-- .spiral/designs/DES-20260919-TTZ7C-25.ttl | 5 ++++- .spiral/sources/SRC-20260919-TTZ7C-18.md | 2 -- 7 files changed, 41 insertions(+), 15 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index d5b5744..23bbac1 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -9,7 +9,8 @@ untracked `package.json~` is unrelated and remains untouched. Status: **Active**. The maintainer committed to this cycle on 2026-09-19: "Lets make a cycle to do all this work." This accepts the proposed work boundary, not the eventual runtime result. The first baseline test slice is implemented -and evidenced; runtime hardening and cycle acceptance remain outstanding. +and evidenced. The maintainer subsequently authorized the whole implementation +and evaluation cycle; only human acceptance may close it. ## Analyze @@ -65,8 +66,9 @@ human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. and the later-dataset recovery guard. Status/queue timing cannot redefine it. - The subsequent maintainer clarification makes the command itself the complete input beyond the dataset: remove transport context from command execution. - Explore this cycle using current store formats without migration. The current - step is design only; source wording is retained in SRC-20260919-TTZ7C-18. + Explore this cycle using current store formats without migration. The earlier + design-only phase is superseded by authorization to complete the cycle; + source wording is retained in SRC-20260919-TTZ7C-18. - Ordinary rerun requires administrator approval and no later accepted command with a dataset. Recover the eligible trailing sequence in original order. - Missing data is not proof that external effects did not happen. Existing @@ -129,7 +131,7 @@ frameworks, remote-filesystem promises, package release, and production recovery or deployment. New canonical file/record formats and migration are excluded by the current design constraint. External administrative plans/reports are tool outputs, not required store metadata. The focused block-level fault exercise -remains in scope. This revision does not authorize runtime changes in this step. +remains in scope. The later maintainer instruction authorizes runtime changes within this boundary. ### Material assumptions and replan conditions diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl index e44b8ec..9cd3555 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl @@ -16,4 +16,7 @@ project:CYC-20260919-TTZ7C-17 a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "8a42ed15242c174a296eb90ed4a35b3354d79e8d" - ] . + ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md index cc0af76..287f165 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.md +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -8,9 +8,9 @@ it does not claim implementation or cycle acceptance. It continues the active cycle's agreed refinement of the durability roadmap. The maintainer's subsequent correction in SRC-20260919-TTZ7C-18 constrains this -revision to design only, using the existing store formats without migration. -It supersedes this document's earlier v2 envelope/descriptor proposal. Neither -that proposal nor this revision has changed runtime behavior. +earlier revision to design only, using the existing store formats without migration. +It supersedes this document's earlier v2 envelope/descriptor proposal. The subsequent authorization in SRC-20260919-TTZ7C-18 permits completing the +whole cycle with this corrected design; the no-format-change constraint remains. Retain OD-JSONTag base/changeset bytes, filenames, command records, status records, and the optional integrity-record schema. No envelope, sequence field, execution @@ -266,7 +266,7 @@ Implementation sequence when runtime work resumes: byte-complete publish/append and directory barriers first (PL02/03/04/10 plus conversion/integrity), then parent error handling/ownership, the acceptance FIFO and command-only worker boundary, read-only startup validation, and administrative recovery under DES-20260919-TTZ7C-25. -The current user request ends at this design revision. +The maintainer has now authorized that implementation sequence through evaluation. New tests must cover initial creation versus open, unchanged stored schemas, existing-store operation without conversion, short/zero writes, close/open error diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.ttl b/.spiral/designs/DES-20260919-TTZ7C-24.ttl index 5373b82..efe6ab7 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-24.ttl @@ -21,4 +21,7 @@ project:DES-20260919-TTZ7C-24 a sd:Design ; sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ], [ a sd:ArtifactReference ; sd:artifact culture:CUL-MUZE-LIB-001 ; - sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ] . + sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md index 88235bf..2a733ca 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.md +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -296,6 +296,23 @@ a defect to fix; the correction does not erase the reproduced ordering mismatch. Still required for acceptance: implement and execute the above tests, record latency and the Linux/storage envelope, run the disposable block-level power-loss -exercise, and obtain human evaluation of the entire active cycle. This step is -design only. Unknown historical inputs/order, unavailable backup evidence, and +exercise, and obtain human evaluation of the entire active cycle. The subsequent maintainer authorization permits implementation and evaluation +of this design. Unknown historical inputs/order, unavailable backup evidence, and irrecoverable history remain explicit limits, not reasons to introduce migration. + +## Concrete operational choices + +Copy canonical files and explicitly declared required artifacts, preserving their +relative directory layout and permissions so existing integrity paths retain their +meaning. Do not sweep unrelated application files or secrets into a backup. Code +is separately selected and hashed; deployments and undeclared dependencies still +need administrator assessment. Backups can be transferred before restoration; +validate and rebase their external inventory without rewriting canonical bytes. + +Recovery protects incomplete destination layouts with a root ownership lock. +An interrupted final report can be finished from validated retained authorization +and completed data without executing commands. A remaining active suffix requires +fresh assessment and a new candidate. Backup coverage comparisons require a +quiescent retained source and compare base, command, status and surviving data. +Promotion remains an explicit deployment configuration operation after validation; +this cycle supplies a verified configuration preview, not a deployment integration. diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.ttl b/.spiral/designs/DES-20260919-TTZ7C-25.ttl index 477ba3b..7104b43 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-25.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-25.ttl @@ -14,4 +14,7 @@ project:DES-20260919-TTZ7C-25 a sd:Design ; sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; - sd:gitCommit "088e389bfb2e2fa33c77c3c8d6c6b3f8af110e9d" ] . + sd:gitCommit "088e389bfb2e2fa33c77c3c8d6c6b3f8af110e9d" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index b6f1073..b5a95db 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -83,8 +83,6 @@ timing are not alternative authorities. Their disagreement is evidence of a bug or damaged history. The previous design-only and no-migration boundary remains; the immediate work is to correct the designs and future verification expectations. -## Outstanding design and evidence - ## Authorization to complete implementation β€” 2026-09-19 Following explanation of serialized acceptance through queue insertion, the From 049c45642fe60146bd511bb87abb4b032f8df36a Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 14:38:22 +0200 Subject: [PATCH 21/37] Implement durable log-ordered commands and guarded administrator recovery --- .../implementations/IMP-20260919-TTZ7C-26.md | 58 +++ .../implementations/IMP-20260919-TTZ7C-26.ttl | 37 ++ DURABILITY.md | 47 ++ README.md | 14 +- docs/power-loss-baseline.md | 62 ++- docs/power-loss-filesystem.md | 47 ++ docs/recovery.md | 195 ++++++++ package.json | 7 +- scripts/convert.mjs | 36 +- scripts/power-loss-vm/exercise.py | 132 ++++++ scripts/power-loss-vm/guest.mjs | 41 ++ scripts/recover.mjs | 58 +++ src/admin-recovery.mjs | 290 ++++++++++++ src/command-worker-module.mjs | 12 +- src/command-worker.mjs | 7 +- src/execute-worker.mjs | 23 + src/index.id.mjs | 5 +- src/index.offset.mjs | 7 +- src/recovery.mjs | 38 +- src/server.mjs | 320 +++++-------- src/storage.mjs | 152 ++++++ src/store-inspection.mjs | 179 +++++++ src/store-ownership.mjs | 51 ++ src/util.mjs | 15 +- test/admin-recovery.mjs | 211 +++++++++ test/after-change-handlers.mjs | 21 +- test/conversion-offsets.mjs | 13 + test/durability-crash.mjs | 186 ++------ test/durability-helpers.mjs | 18 + test/durability-latency.mjs | 22 + test/durability-startup.mjs | 54 +-- test/power-loss-baseline.mjs | 441 +++++------------- test/power-loss-preload.mjs | 17 +- test/storage.mjs | 34 ++ 34 files changed, 2011 insertions(+), 839 deletions(-) create mode 100644 .spiral/implementations/IMP-20260919-TTZ7C-26.md create mode 100644 .spiral/implementations/IMP-20260919-TTZ7C-26.ttl create mode 100644 DURABILITY.md create mode 100644 docs/power-loss-filesystem.md create mode 100644 docs/recovery.md create mode 100644 scripts/power-loss-vm/exercise.py create mode 100644 scripts/power-loss-vm/guest.mjs create mode 100644 scripts/recover.mjs create mode 100644 src/admin-recovery.mjs create mode 100644 src/execute-worker.mjs create mode 100644 src/storage.mjs create mode 100644 src/store-inspection.mjs create mode 100644 src/store-ownership.mjs create mode 100644 test/admin-recovery.mjs create mode 100644 test/durability-latency.mjs create mode 100644 test/storage.mjs diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md new file mode 100644 index 0000000..f7f4876 --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -0,0 +1,58 @@ +# Implementation: Durable Ordered Writes And Administrative Recovery + +Implements the corrected DES-24/25 within existing canonical formats, following +the maintainer's authorization to complete CYC-17. It transforms the effective +server/worker, converter/index and baseline-test implementations identified in +the companion Turtle. Existing timeout containment, awaited hooks, final offsets, +framing and optional integrity semantics remain; automatic uncertain replay and +transport-derived command inputs are deliberately removed. + +`src/storage.mjs` centralizes complete writes, durable publication/appends, file +and directory barriers, permission preservation, and first-error propagation. +Default indexes retain synchronous hook compatibility with equivalent barriers. +Conversion finishes a new base, indexes, empty logs and optional integrity under +ownership, refusing overwrite. `requiredFiles` explicitly includes custom +recovery dependencies in startup checks, completion syncing and copied inventory. + +`src/server.mjs` serializes acceptance through durable command/status writes and +queue insertion, then executes sequentially in log order. It passes the exact +logged command to workers, with no request object. Awaited handler/index work, +required-file barriers and done persistence precede query publication. Uncertain +storage failures stop mutation. `src/execute-worker.mjs` retains worker timeout +containment. Healthy shutdown drains accepted work and releases ownership. + +`src/store-inspection.mjs` validates canonical history and reconstructs its verified +prefix without importing user modules. It distinguishes missing, corrupt and +uncommitted data, records ever-accepted history, and checks later datasets in log +order. `src/store-ownership.mjs` excludes cooperating writers using canonical +paths and durable owner records; crash locks are never automatically stolen. +Normal startup refuses missing canonical files and pending/uncertain work without +rewriting statuses or rerunning handlers. + +`src/admin-recovery.mjs` and `scripts/recover.mjs` implement inspect, preview, +explicitly approved suffix recovery into a new candidate, preserved original +copies and external audit, fresh source/code validation, completion without +reexecution after an interrupted report, consistent backup/restore, offline +unlock and promotion verification. Existing status schemas/IDs are retained. +A later ever-accepted dataset blocks earlier rerun even when empty/corrupt/unsafe. +Backup copy preserves relative artifact paths and permissions, excludes unrelated +secrets, and remains portable. Restore reports known coverage and uncertainty. +Code and undeclared dependencies retain the documented administrator obligations. + +Tests cover injected short writes/errors/barrier ordering, authoritative concurrent +acceptance, JSONTag command inputs without request context, process-crash policy, +read-only inspection, guards, staleness, interruption/effect witnesses, retained +digests, portable backups and code-aware promotion verification. PL15 and PL16 +expectations change only as explicitly corrected by the maintainer; historical +baseline evidence remains at EVD-23. New tests have no TODO exceptions. + +`scripts/power-loss-vm/` packages a disposable Linux/ext4/QEMU exercise with a +volatile NBD cache, flush/FUA persistence, independent serial observer, device +and filesystem negative controls, accepted/active/done cuts and commit timings. +It never touches a host block device. Results belong in EVD-27 after this +implementation has an exact committed revision. + +Operational documentation is in `DURABILITY.md`, `docs/recovery.md`, and +`docs/power-loss-filesystem.md`. Full rebuild remains contract-only. Physical +hardware certification, exhaustive faults, arbitrary effect rollback, deployment, +release and integration are outside this implementation. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl new file mode 100644 index 0000000..a0eaa4e --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -0,0 +1,37 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260919-TTZ7C-26 a sd:Implementation ; + dcterms:identifier "IMP-20260919-TTZ7C-26" ; + sd:repositoryPath ".spiral/implementations/IMP-20260919-TTZ7C-26.md" ; + sd:status sd:Active ; + sd:implementationLocation "src/storage.mjs" ; + sd:implementationLocation "src/server.mjs" ; + sd:implementationLocation "src/command-worker-module.mjs" ; + sd:implementationLocation "src/execute-worker.mjs" ; + sd:implementationLocation "src/store-inspection.mjs" ; + sd:implementationLocation "src/store-ownership.mjs" ; + sd:implementationLocation "src/admin-recovery.mjs" ; + sd:implementationLocation "src/index.offset.mjs" ; + sd:implementationLocation "src/index.id.mjs" ; + sd:implementationLocation "src/recovery.mjs" ; + sd:implementationLocation "scripts/convert.mjs" ; + sd:implementationLocation "scripts/recover.mjs" ; + sd:implementationLocation "scripts/power-loss-vm/exercise.py" ; + sd:implementationLocation "test/power-loss-baseline.mjs" ; + sd:implementationLocation "test/admin-recovery.mjs" ; + sd:implementationLocation "test/storage.mjs" ; + sd:implementationLocation "docs/recovery.md" ; + sd:implementationLocation "DURABILITY.md" ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-25 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-011 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260917-TTZ7C-11 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260917-TTZ7C-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-22 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] ; + sd:implementationChangeKind sd:MixedChange . diff --git a/DURABILITY.md b/DURABILITY.md new file mode 100644 index 0000000..cd8366e --- /dev/null +++ b/DURABILITY.md @@ -0,0 +1,47 @@ +# Durability contract + +SimplyStore retains its existing base, changeset, command/status and optional +integrity formats. There is no data migration. The command log is authoritative: +acceptance serialization keeps execution in log order, including concurrent +requests and duplicate IDs. A command supplies all inputs beyond its preceding +dataset; handlers receive `undefined` as the third argument and metadata as the +fourth. Audit application commands that previously used request context. + +Before returning accepted, SimplyStore completely writes and syncs the command +and acceptance records and their parent directories. Before publishing done, it +finishes the changeset, index hooks and required artifacts, syncs file contents +and published directory entries, and persists terminal status. Short writes are +retried; write, sync, rename and close failures propagate. Uncertain persistence +halts mutation and leaves ownership evidence for inspection. + +The tested envelope is one cooperating writer on Linux/ext4, `data=ordered`, +`barrier=1`, with storage honoring flushes. A QEMU guest over a volatile NBD device +model exercises full guest/cache loss and recovery. This is bounded filesystem +and block-model evidence, not physical hardware certification. See the +[reproducible exercise](docs/power-loss-filesystem.md) and +[cycle evidence](.spiral/evidence/EVD-20260919-TTZ7C-27.md). + +Startup validates existing canonical files before executing user modules or +changing command status. Complete consistent stores reopen normally. Missing +logs, malformed records, missing/corrupt committed data, or pending/uncertain +commands stop startup for administrator assessment. Upgrading cannot prove that +older writers preserved the whole history. Explicit integrity configuration is +needed to detect a lost manifest; existence-based detection alone cannot do so. + +The [administrator guide](docs/recovery.md) provides read-only inspection, +source-bound preview, explicitly approved suffix recovery on a copy, interrupted +recovery handling, consistent backup/restore and deliberate promotion. No later +ever-accepted command may have a dataset when an earlier command is rerun, +including corrupt or unsafe later datasets. Missing files never establish that +external effects did not happen. Recovery cannot invent missing inputs or promise +exactly-once external effects. Changed-code rebuild with cooperative effect +suppression remains a separate follow-up contract. + +Ownership locks exclude cooperating server/admin writers across configured +paths. They are not protection against arbitrary filesystem writers or older +binaries. Stale locks require explicit offline assessment, never PID-age stealing. +Custom hooks must await their work; declare additional required artifacts with +`requiredFiles`. Default indexes are reconstructible; custom dependencies need +their own conformance checks. Unsupported directory-sync operations fail rather +than silently weakening the guarantee. Broader filesystems, random/soak testing, +and deployment-specific power-loss testing remain future validation work. diff --git a/README.md b/README.md index f822972..1210d77 100644 --- a/README.md +++ b/README.md @@ -38,18 +38,20 @@ Import the server in your main file like this: import simplystore from '@muze-nl/simplystore' ``` -Then configure and start the server, like this: +Initialize the store first using the [conversion procedure](docs/recovery.md#configure-and-initialize); opening an existing store requires its base and logs. Then configure and start the server: ```javascript simplystore.run({ - datafile: process.cwd().'data.json' + datafile: './store/data.jsontag', + commandLog: './store/command-log.jsontag', + commandStatus: './store/command-status.jsontag' }) -```` +``` simplystore is an [express application](https://expressjs.com/), with all the usual options. Other options are: - port: The port number to use, defaults to 3000 -- dataspace: an object or array with all the data that SimplyStore will serve. Optional, replaces the datafile. +- commandsFile: the module implementing commands; every invocation input must be in the logged command. HTTP request context is not passed to handlers. If you start your server: @@ -61,6 +63,10 @@ You should be able to go http://localhost:3000/query/ and see something like thi ![image](https://github.com/SimplyEdit/SimplyStore/assets/1006453/3bec6b97-ffa1-4114-9ed4-51a68f73476e) +## Durability and recovery + +See the [durability contract](DURABILITY.md) and [administrator recovery guide](docs/recovery.md). Command-log order governs execution. Acceptance and completion await file and directory barriers. Uncertain or pending work found at startup requires administrator assessment; missing data does not prove that external effects did not happen. Existing store formats are retained without migration. + ## Custom Index Modules Use `indexFile` to configure a module whose default export provides the existing diff --git a/docs/power-loss-baseline.md b/docs/power-loss-baseline.md index b67f57c..39a563e 100644 --- a/docs/power-loss-baseline.md +++ b/docs/power-loss-baseline.md @@ -1,29 +1,36 @@ # Power-loss failure matrix and baseline -This is the first evidence slice of CYC-20260919-TTZ7C-17, against runtime -baseline `44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. It does not change production -persistence or recovery. The cycle's administrative tooling, sync fixes, and -filesystem acceptance exercise remain outstanding. +The historical matrix below records the first evidence slice against runtime +`44ea91717a328ad7b5e049c6c0845b5e3bcc1718`, preserved by +[EVD-23](../.spiral/evidence/EVD-20260919-TTZ7C-23.md). That baseline had ten +violated invariant assertions. It is not a description of the hardened runtime. ## Run and interpret ```sh npm run test:power-loss npm run test:power-loss:strict +npm run test:recovery ``` -The tests need permission to bind localhost ports and use disposable directories. -The first command characterizes current behavior: fixture checks and controls -must pass, while known violated requirements are explicit nested TODO tests. -The strict command removes those annotations and must currently exit nonzero. -Neither a green baseline command nor its TODO count means power-loss safety. -Remove the corresponding TODO when later implementation satisfies an invariant; -also revise any characterization assumptions that intentionally describe the -old defect. Do not suppress a new harness/setup failure with TODO. +The current power-loss suite checks corrected invariants with no TODO allowances; +both power-loss commands must pass. Tests need localhost ports and use disposable +storage. Administrative policy cases now live in `test/admin-recovery.mjs`. +PL15's old request-retention expectation was superseded by the maintainer's +command-only contract: the replacement proves the request is unavailable and +logged JSONTag values survive. PL16 now compares both live execution and restart +with the authoritative command log, fixing the old queue order instead of +preserving it. Historical observations remain evidence of the original defects. -## Persistence boundaries +Current coverage includes full-write loops, file/directory barriers and errors, +missing canonical files, no automatic effect repetition, concurrent acceptance +and duplicates, and recovery/backup policy. See [the current durability +contract](../DURABILITY.md), [administrator guide](recovery.md), and [filesystem +exercise](power-loss-filesystem.md) for the implemented behavior and its bounds. -| Artifact | Current writer / barrier | Required role | +## Historical persistence boundaries + +| Artifact | Baseline writer / barrier | Required role | |---|---|---| | Base dataset | Converter uses `writeFileSync` | Durable starting state before operating or backing up a store; untested conversion publication remains open | | Command log | `appendFile` then `datasync` | Complete inputs before accepted; persist first-created filename | @@ -37,7 +44,7 @@ File sync and directory sync are distinct barriers. The namespace-loss model uses this distinction from the [Linux fsync contract](https://man7.org/linux/man-pages/man2/fsync.2.html). It does not assert that every tested filesystem would actually lose an entry. -## Executable matrix +## Historical baseline matrix | Test | Fault or question | Observed baseline | Requirement disposition | |---|---|---|---| @@ -93,20 +100,11 @@ hardware lies about flushes, real filesystem mount behavior, or external systems The external-effect witness proves repeated invocation only, not exactly-once behavior of any real email/payment/network service. -## Still required in this cycle - -- Initial base/log publication and integrity-enabled power-loss combinations. -- Append partial-write/ENOSPC details, other close failures, interrupted rename - completion, and sync-error recovery beyond the named cases above. -- Explicit authoritative ordering, failed/unsafe histories, duplicated IDs, - writer ownership, and exclusion of concurrent administrative/normal writes. -- Administrative inspect/preview/apply, stale-plan rejection, allowed trailing - recovery, interrupted recovery, verified promotion, and loss reporting. -- Complete log inputs across relevant configured integrations and compatibility - for legacy records; a FIXME alone is not the input-loss evidence. -- Consistent backup creation and restore, including cases where rerun is unsafe. -- Measured latency, a named Linux filesystem/storage configuration, and an actual - disposable block-level exercise before claiming filesystem power-loss safety. - -Full changed-code rebuild tooling remains follow-up work under the cycle's -recorded side-effect-suppression contract. +## Current evaluation + +Implementation and current test/VM results are recorded separately in +[EVD-27](../.spiral/evidence/EVD-20260919-TTZ7C-27.md). The bounded preload model +and a real ext4 guest over a volatile block-device model provide complementary +evidence. Neither is physical hardware certification or an exhaustive campaign. +Full changed-code rebuild tooling remains follow-up work under the recorded +side-effect-suppression contract. diff --git a/docs/power-loss-filesystem.md b/docs/power-loss-filesystem.md new file mode 100644 index 0000000..17ac1ef --- /dev/null +++ b/docs/power-loss-filesystem.md @@ -0,0 +1,47 @@ +# Disposable filesystem power-loss exercise + +`scripts/power-loss-vm/exercise.py` runs the actual server and storage code in a +Linux VM with an ext4 data volume (`data=ordered,barrier=1`). It creates only +regular image files beneath a new temporary workspace; it never mounts or writes +a host block device. A statically linked BusyBox, Node, Python, cpio, gzip, mkfs.ext4, +a QEMU x86 system binary/firmware, and a Linux kernel with built-in ext4 and virtio +block support are required. Installed packages are not modified by the script. + +```sh +python3 scripts/power-loss-vm/exercise.py \ + --qemu /usr/bin/qemu-system-x86_64 --kernel /path/to/readable/vmlinuz \ + --firmware-dir /usr/share/qemu --bios /usr/share/seabios/bios-256k.bin +``` + +`--library-path` can point at libraries unpacked with QEMU into a temporary +workspace. `--work` selects a new, nonexistent output directory; otherwise the +script allocates one under `/tmp`. No host root privileges or KVM access are +required: it uses TCG emulation with two CPUs and 1024 MiB RAM. No external guest +network is attached; client/server traffic stays on guest loopback. + +The host observes acknowledgment markers over the VM's serial output and kills +the entire VM, losing guest RAM and kernel caches. The test NBD device buffers +writes in memory and persists only completed FLUSH/FUA operations (it may persist +other bytes in the same page on FUA). At each cut the device also discards its +remaining dirty pages. Reboot attaches the surviving image and lets ext4 recover; +read-only engine inspection reports committed data and uncertain commands. + +The device has a separate negative control: flushed A survives while unflushed B +is lost. A filesystem negative control also verifies that an ordinary unsynced +write disappears. Other scenarios cut after accepted, during an active handler, +and after observed completion. The completion scenario also records guest ext4 +acceptance/completion latency for five commands. The output includes exact tool/ +kernel identification, mount/cache settings, source revision/diff digest, serial +logs, block write/flush counts, images and a results report. + +This tests a named Linux/filesystem/storage model, not physical power removal or +all hardware/cache implementations. It is a focused boundary exercise, not an +exhaustive randomized campaign. The device model honors flushes; devices that lie +about them are outside the guarantee. It cannot prove absence of historical loss +from a different writer/protocol. Local `npm run bench:durability` timings report +their filesystem separately; tmpfs timings are not disk performance measurements. + +The model uses QEMU's documented [block device cache settings](https://www.qemu.org/docs/master/system/qemu-manpage.html). +See also QEMU's [block I/O fault testing documentation](https://www.qemu.org/docs/master/devel/testing/blkdebug.html) +for the distinction between guest writes and flush events. The test backend is +purpose-built for this bounded exercise, not a production NBD server. diff --git a/docs/recovery.md b/docs/recovery.md new file mode 100644 index 0000000..2cc9d94 --- /dev/null +++ b/docs/recovery.md @@ -0,0 +1,195 @@ +# Administrator recovery + +SimplyStore preserves the existing OD-JSONTag data, command-log, status-log and +optional integrity formats. The command log defines command order. Commands must +contain every invocation input beyond the dataset; HTTP request context is no +longer supplied to command workers. The fourth handler argument still provides +metadata/index access; the former third request argument is `undefined`. + +Use this procedure with the hardened binary. Older binaries understand the file +formats but do not implement the new ownership, ordering, or recovery policy. +Do not run competing versions/writers on one store. + +## Configure and initialize + +Tools accept a declarative JSON file, not executable server configuration: + +```json +{ + "datafile": "/srv/simplystore/data.jsontag", + "commandLog": "/srv/simplystore/command-log.jsontag", + "commandStatus": "/srv/simplystore/command-status.jsontag", + "commandsFile": "/srv/application/commands.mjs", + "indexFile": "/srv/application/index.mjs", + "integrityFile": "/srv/simplystore/data.integrity.jsontag" +} +``` + +Omit `integrityFile` if integrity is unused. Set it explicitly when integrity is +required: existence-based detection cannot detect a lost manifest. Relative paths +resolve against the command's working directory, as server paths do. `requiredFiles` +can list additional required files; these are checked at startup and synced before +done. Keep custom artifact paths inside the declared directories or list them +explicitly. Custom hooks must finish all required work before resolving. + +For a new store, create an empty destination directory and run: + +```sh +node scripts/convert.mjs input.jsontag /srv/new-store/data.jsontag --integrity +``` + +The converter creates empty command/status logs alongside the base, finishes +indexes and optional integrity, and releases ownership after durable completion. +It refuses to overwrite an existing base/log/manifest. Opening an existing store +never creates missing logs. A bare historical base is not proof of an empty +history: preserve it and its surrounding evidence before deciding on initialization. +There is no migration or format-conversion step for an existing complete store. + +## Inspect before executing anything + +Stop the writer using the deployment's normal service procedure. Then: + +```sh +node scripts/recover.mjs inspect --store store.json --quiescent --out plan.json +``` + +Use `--quiescent` only for a stopped store or immutable copy. Without it, inspection +is diagnostic and the plan is not actionable. Plans must be new files outside the +source directories. Inspection does not import handlers, custom workers, or index +modules, and does not modify the source. + +Review `blocks`, the commands in log order, statuses and attempts, existing/missing +files, and the proposed `rerun` list. A complete command log fixes the order even +when an old queue/status sequence disagrees. Actually committed out-of-order +changesets require diagnosis; reordering positional data is not a valid repair. + +The tool cannot prove history completeness, original deployment identity, hidden +application inputs, or external effects from the surviving files alone. Use retained +backups, deployment records, application knowledge, and external-system evidence. +A current code hash binds the selected code against changes after preview; it +cannot prove which code ran historically. Missing business inputs are not invented +or injected by the tool. If these matters cannot be established, use a verified +backup or retain the diagnosis; do not claim a safe rerun. + +## Deal with ownership after a crash + +Each mutable directory has a `.simplystore-lock` during operation. Normal shutdown +drains accepted work before releasing ownership. A crash or storage fault leaves +locks. Neither PID age nor an apparently dead process authorizes automatic stealing. +A hung shutdown can be terminated, but then requires this offline procedure. + +After confirming that no writer remains, inspect and explicitly release locks: + +```sh +node scripts/recover.mjs unlock --store store.json --confirmed-stopped \ + --operator 'administrator name' --reason 'Writer stopped; reviewed retained evidence' \ + --out unlock-audit.json +``` + +This preserves lock diagnostics in a new external audit file before removal. It +does not authorize reruns or repair data. Normal startup still rejects unresolved +accepted/active work, missing committed data, and other inconsistent evidence. + +An interrupted recovery has an external audit bundle and candidate ownership, +including a parent lock while the candidate is being copied. Review that bundle +before releasing its locks. A candidate that looks complete but lacks its recovery +completion report must be finished with `finish` below, not merely unlocked. + +## Apply an approved trailing recovery + +For log order A/B/C, B cannot be rerun if C has a datasetβ€”even if C is accepted, +failed, unsafe, or its dataset is corrupt or zero bytes. Status insertion order +cannot make C appear earlier. If the trailing commands have no datasets, the +administrator can approve their execution in log order from a verified prefix. +Absence of a dataset does not prove absence of external effects. + +```sh +node scripts/recover.mjs apply --plan plan.json --to /srv/recovered-store \ + --audit-dir /srv/recovery-audit --approve-rerun B,C,D \ + --operator 'administrator name' \ + --reason 'Verified complete history and inputs, original command meaning, and assessed external effects' +``` + +The approval must exactly match the complete ordered `rerun` list. Source, target, +and audit locations must not overlap; destination directories must be new. The +source must remain quiescent. The tool acquires ownership and rechecks the source +inventory, missing filenames, selected code, and rerun predicate before execution. +A newly appearing later dataset invalidates the plan. + +Original store artifacts are copied into the audit bundle. Unrelated files such as +application secrets are not swept into backups; retain deployment evidence separately. Candidate files preserve their +relative layout and permissions; mutable files are copied, never hard-linked. +Commands retain original IDs and logged inputs. Recovery appends existing status +records, preserving the original bytes as a prefix. Authorization/attempt links +live in the external audit, not a new store record format. Required persistence +errors stop work; a possibly committed done is never overwritten with failed. +A retained expected changeset digest is checked before a new digest or done is +written. Without that witness, structural correctness is not independent proof +of byte-identical historical output. + +On an interrupted or failed attempt, preserve the candidate and audit. Inspect the +candidate with its configuration from the audit authorization. Completed commands +are reconstructed, not selected again. An active attempt needs fresh effect +assessment and a new plan; existing partial data needs separate diagnosis and is +not automatically deleted to make a rerun eligible. After explicit offline lock +release, apply an eligible plan to another new destination. Do not restart recovery +against an older source while ignoring later candidate evidence. + +If all approved commands committed but only completion reporting was interrupted: + +```sh +node scripts/recover.mjs finish --audit-dir /srv/recovery-audit --confirmed-stopped \ + --operator 'administrator name' --reason 'Reviewed completed attempts and original authorization' +``` + +This validates the retained authorization, source, candidate, and approved command +coverage, writes the completion report, and releases ownership without execution. +A missing audit bundle prevents certification of that recovery attempt. + +## Consistent backup and restore + +Create backups with a stopped writer. The tool acquires ownership, copies canonical +data/log/integrity files, default index artifacts, and explicitly required files, validates reconstruction, +and writes an external completion manifest last: + +```sh +node scripts/recover.mjs backup --store store.json --quiescent --to /srv/backup-001 +node scripts/recover.mjs restore --backup /srv/backup-001 \ + --to /srv/restored-store --audit-dir /srv/restore-audit +``` + +A completed backup can be moved or transferred as one directory; restore validates +and rebases its external inventory while preserving canonical bytes. Incomplete +or changed backups are refused. Restore executes no handlers. Optional +`--store damaged-store.json --quiescent` compares backup coverage against a stopped +source under ownership. Review `sameBase`, `missingFromBackup` and the warnings; comparison includes +command bytes, status, and available completed data. Without a +source, loss relative to current history is unknown. A known older backup can be +an explicit rollback with disclosed loss; it is not recovery of all later work. +This first tool restores complete backups; it does not merge arbitrary surviving +changesets, repair malformed logs, or override unknown command inputs. + +## Validate and promote + +```sh +node scripts/recover.mjs verify --report /srv/recovery-audit/complete.json \ + --out activation-preview.json +``` + +Verification also rejects changes to the selected recovery code files. Review +the exact candidate configuration and command coverage. Undeclared dependencies +and deployment configuration still require independent review. Stop the old +service, preserve its directory and configuration, and explicitly switch the +service to the candidate configuration. The tool does not change a deployment. +The server validates the selected store before listening. Switching multiple +configured directories is not an atomic filesystem operation. After accepting +new commands, switching back can lose those commands and needs a fresh assessment. +Normal loading does not require the external administrative reports. + +## Rebuild is separate + +Full changed-code rebuild remains follow-up work. Its retained contract produces a +new dataset and supplies `mode: rebuild`, `sideEffectsAllowed: false` to commands +and hooks. This is a cooperative execution control, not hidden business input or +a sandbox. Ordinary recovery does not suppress effects that might still need to +happen, and has no `--force` escape from the later-dataset rule. diff --git a/package.json b/package.json index 61d5e11..663654d 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,9 @@ "test:durability": "node --test test/durability-crash.mjs", "test:handlers": "node --test test/after-change-handlers.mjs", "test:power-loss": "node --test --test-reporter=tap test/power-loss-baseline.mjs", - "test:power-loss:strict": "SIMPLYSTORE_BASELINE_STRICT=1 node --test --test-reporter=tap test/power-loss-baseline.mjs" + "test:power-loss:strict": "SIMPLYSTORE_BASELINE_STRICT=1 node --test --test-reporter=tap test/power-loss-baseline.mjs", + "test:recovery": "node --test test/storage.mjs test/admin-recovery.mjs", + "bench:durability": "node --test test/durability-latency.mjs" }, "author": "auke@muze.nl", "license": "MIT", @@ -37,6 +39,9 @@ }, "files": [ "README.md", + "DURABILITY.md", + "docs/recovery.md", + "docs/power-loss-filesystem.md", "LICENSE", "src/", "www/", diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 966bb52..61904f7 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -1,32 +1,46 @@ +import {Buffer} from 'node:buffer' +import process from 'node:process' import JSONTag from '@muze-nl/jsontag' import serialize, { stringify } from '@muze-nl/od-jsontag/src/serialize.mjs' import Parser from '@muze-nl/od-jsontag' import { finalizeIndex } from '../src/index.mjs' import fs from 'node:fs' import path from 'node:path' +import {publishFile, syncFile} from '../src/storage.mjs' +import {acquireOwnership} from '../src/store-ownership.mjs' +import {appendIntegrityRecord, getDefaultIntegrityFile} from '../src/integrity.mjs' const __dirname = import.meta.dirname; -if (process.argv.length<=3) { - console.log('usage: node ./convert.mjs {inputfile} {outputfile} {indexlib?}') +const integrity = process.argv.includes('--integrity') +const args = process.argv.slice(2).filter(value => value !== '--integrity') +if (args.length<2) { + console.log('usage: node ./convert.mjs {inputfile} {outputfile} {indexlib?} {schema?} [--integrity]') process.exit() } // parse command line -let inputFile = process.argv[2] -let outputFile = process.argv[3] -let indexFile = process.argv[4] +let inputFile = args[0] +let outputFile = args[1] +let indexFile = args[2] if (indexFile && indexFile[0]!='/') { indexFile = process.cwd()+'/'+indexFile } else if (!indexFile) { indexFile = __dirname+'/../src/index.mjs' } -let schemaFile = process.argv[5] +let schemaFile = args[3] if (schemaFile && schemaFile[0]!='/') { schemaFile = process.cwd()+'/'+schemaFile } async function main() { + const directory = path.resolve(path.dirname(outputFile)) + const logs = ['command-log.jsontag', 'command-status.jsontag'].map(name => path.join(directory, name)) + const ownership = await acquireOwnership([directory]) + if ([outputFile, ...logs, getDefaultIntegrityFile(outputFile)].some(file => fs.existsSync(file))) { + await ownership.release() + throw new Error('Conversion requires new output files; existing store is not overwritten') + } // now create indexes console.log('Using index library:', indexFile) @@ -68,14 +82,20 @@ async function main() { meta.index.id.set(JSONTag.getAttribute(ob, 'id'), ob) } - index.create(odData, meta) + await index.create(odData, meta) console.log('Indexes created') strData = stringify(serialize(odData)) - fs.writeFileSync(outputFile, strData) + await publishFile(outputFile, strData) // Custom indexes may change record sizes or add records after initial parsing. await finalizeIndex(index, strData, meta) + if (integrity) await appendIntegrityRecord(getDefaultIntegrityFile(outputFile), outputFile, Buffer.from(strData)) + for (const file of logs) await publishFile(file, '') + for (const entry of fs.readdirSync(directory, {withFileTypes:true})) { + if (entry.isFile()) await syncFile(path.join(directory, entry.name)) + } + await ownership.release() console.log('Converted data written to ',outputFile) } diff --git a/scripts/power-loss-vm/exercise.py b/scripts/power-loss-vm/exercise.py new file mode 100644 index 0000000..429bb8c --- /dev/null +++ b/scripts/power-loss-vm/exercise.py @@ -0,0 +1,132 @@ +import os,socket,struct,threading,subprocess,selectors,time,json,pathlib,shutil,argparse,re,tempfile,hashlib +arguments=argparse.ArgumentParser(description='Disposable ext4 VM power-loss exercise; never uses a host block device') +arguments.add_argument('--qemu',required=True) +arguments.add_argument('--kernel',required=True) +arguments.add_argument('--firmware-dir',required=True) +arguments.add_argument('--bios',required=True) +arguments.add_argument('--library-path',default='') +arguments.add_argument('--work') +args=arguments.parse_args() +ROOT=pathlib.Path(args.work) if args.work else pathlib.Path(tempfile.mkdtemp(prefix='simplystore-vm-exercise-')) +if args.work: ROOT.mkdir(parents=True,exist_ok=False) +QEMU=pathlib.Path(args.qemu).resolve() +repo=pathlib.Path(__file__).resolve().parents[2] +env={**os.environ,**({'LD_LIBRARY_PATH':args.library_path} if args.library_path else {})} +guest=ROOT/'guest' +for directory in ['bin','proc','sys','dev','store','tmp','lib64','app']:(guest/directory).mkdir(parents=True,exist_ok=True) +shutil.copy(shutil.which('busybox'),guest/'bin/busybox') +for command in ['sh','mount','mkdir','sleep','cat','poweroff','ip']:(guest/'bin'/command).symlink_to('busybox') +node=shutil.which('node');shutil.copy(node,guest/'bin/node') +for library in re.findall(r'(/[^\s]+)',subprocess.check_output(['ldd',node],text=True)): + dest=guest/library.lstrip('/');dest.parent.mkdir(parents=True,exist_ok=True);shutil.copy(library,dest) +for folder in ['src','node_modules','www']:shutil.copytree(repo/folder,guest/'app'/folder) +shutil.copy(pathlib.Path(__file__).with_name('guest.mjs'),guest/'app/guest.mjs') +(guest/'app/package.json').write_text('{"type":"module"}') +(guest/'init').write_text("#!/bin/sh\nexport PATH=/bin\nmount -t proc proc /proc\nmount -t sysfs sysfs /sys\nmount -t devtmpfs devtmpfs /dev\nip link set lo up\nmount -t ext4 -o data=ordered,barrier=1 /dev/vda /store || exec sh\ncd /app\nnode /app/guest.mjs\nwhile true; do sleep 100; done\n") +os.chmod(guest/'init',0o755) +with open(ROOT/'initrd.gz','wb') as output: + files=subprocess.Popen(['find','.','-print0'],cwd=guest,stdout=subprocess.PIPE) + cpio=subprocess.Popen(['cpio','--null','-o','--format=newc'],cwd=guest,stdin=files.stdout,stdout=subprocess.PIPE,stderr=subprocess.DEVNULL) + files.stdout.close() + subprocess.run(['gzip','-1'],stdin=cpio.stdout,stdout=output,check=True) + cpio.stdout.close();assert cpio.wait()==0 and files.wait()==0 +metadata={'qemu':subprocess.check_output([str(QEMU),'--version'],env=env,text=True),'node':subprocess.check_output([node,'--version'],text=True),'kernelSha256':hashlib.sha256(pathlib.Path(args.kernel).read_bytes()).hexdigest(),'repositoryHead':subprocess.check_output(['git','rev-parse','HEAD'],cwd=repo,text=True).strip(),'worktreeDiffSha256':hashlib.sha256(subprocess.check_output(['git','diff','HEAD'],cwd=repo)).hexdigest(),'filesystem':'ext4 data=ordered,barrier=1','device':'virtio-blk, QEMU cache=writeback, volatile NBD backend honoring FLUSH/FUA','acceleration':'TCG, 2 CPUs, 1024 MiB','physicalPowerCut':False} +(ROOT/'environment.json').write_text(json.dumps(metadata,indent=2)) +print('Artifacts: '+str(ROOT),flush=True) +class Disk: + def __init__(self,image): + self.image=image;self.data=bytearray(image.read_bytes());self.dirty=set();self.flushes=0;self.writes=0;self.lost=0 + self.sockpath=str(ROOT/'block.sock') + try:os.unlink(self.sockpath) + except FileNotFoundError:pass + self.server=socket.socket(socket.AF_UNIX);self.server.bind(self.sockpath);self.server.listen(1) + self.thread=threading.Thread(target=self.serve,daemon=True);self.thread.start() + def serve(self): + try: + self.conn,_=self.server.accept();c=self.conn + c.sendall(struct.pack('>QQQI',0x4e42444d41474943,0x0000420281861253,len(self.data),1|4|8)+bytes(124)) + def receive(n): + data=b'' + while len(data)IHHQQI',receive(28)) + assert magic==0x25609513 + response=b'' + if kind==0:response=bytes(self.data[offset:offset+length]) + elif kind==1: + self.data[offset:offset+length]=receive(length);pages=set(range(offset//4096,(offset+length+4095)//4096));self.dirty.update(pages);self.writes+=1 + if flags&1:persist(pages) + elif kind==2:break + elif kind==3:self.flushes+=1;persist(set(self.dirty)) + else:raise RuntimeError('Unsupported NBD command '+str(kind)) + c.sendall(struct.pack('>IIQ',0x67446698,0,handle)+response) + except (EOFError,ConnectionResetError,BrokenPipeError,OSError):pass + def stop(self): + self.server.close() + if hasattr(self,'conn'): + try:self.conn.shutdown(socket.SHUT_RDWR) + except OSError:pass + self.conn.close() + self.thread.join(2);self.lost=len(self.dirty) + +def boot(image,scenario,recover=False): + disk=Disk(image) + command=[str(QEMU),'-L',args.firmware_dir,'-vga','none','-machine','pc,accel=tcg','-cpu','max','-m','1024','-smp','2','-display','none','-monitor','none','-serial','stdio','-no-reboot','-net','none','-bios',args.bios,'-kernel',args.kernel,'-initrd',str(ROOT/'initrd.gz'),'-append',f'console=ttyS0 quiet panic=-1 scenario={scenario}','-drive',f'file=nbd:unix:{disk.sockpath},if=virtio,format=raw,cache=writeback'] + p=subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,env=env) + selector=selectors.DefaultSelector();selector.register(p.stdout,selectors.EVENT_READ) + lines=b'';deadline=time.time()+90;found=None + marker=b'VERIFY ' if recover else {'done':b'CUT_DONE','accepted':b'CUT_ACCEPTED','active':b'CUT_ACTIVE','unsynced':b'CUT_UNSYNCED'}[scenario] + try: + while time.time()IHHQQI',0x25609513,0,kind,1,offset,len(data))+data) + assert recv_exact(16)==struct.pack('>IIQ',0x67446698,0,1) +request(1,0,b'A'*4096);request(3);request(1,4096,b'B'*4096) +client.close();disk.stop() +assert control.read_bytes()[:4096]==b'A'*4096 and control.read_bytes()[4096:8192]==bytes(4096) +assert disk.lost==1 +print(json.dumps({'deviceControl':'flushed A survives; unflushed B lost','discardedDirtyPages':disk.lost}),flush=True) +results=[] +for scenario in ['done','accepted','active','unsynced']: + image=ROOT/f'{scenario}.img' + with open(image,'wb') as f:f.truncate(128*1024*1024) + subprocess.run(['mkfs.ext4','-q','-F','-O','^metadata_csum_seed',str(image)],check=True) + cut=boot(image,scenario) + result=json.loads(boot(image,scenario,True).removeprefix('VERIFY ')) + assert not result['errors'],result + if scenario=='done':assert result['ready'] and result['persons']==[{'name':name} for name in ['A','L0','L1','L2','L3']],result + elif scenario=='unsynced':assert result['ready'] and not result['unsyncedPresent'],result + else:assert len(result['commands'])==1 and result['commands'][0]['id']=='A' and result['commands'][0]['status'] in ['accepted','active'],result + results.append({'scenario':scenario,'cut':cut,'recovered':result}) +(ROOT/'results.json').write_text(json.dumps(results,indent=2)) diff --git a/scripts/power-loss-vm/guest.mjs b/scripts/power-loss-vm/guest.mjs new file mode 100644 index 0000000..b673c0a --- /dev/null +++ b/scripts/power-loss-vm/guest.mjs @@ -0,0 +1,41 @@ +import process from 'node:process' +import fs from 'node:fs/promises' +import JSONTag from '@muze-nl/jsontag' +import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' +import server from './src/server.mjs' +import {publishFile} from './src/storage.mjs' +import {inspectStore} from './src/store-inspection.mjs' +const mode=(await fs.readFile('/proc/cmdline','utf8')).match(/scenario=(\w+)/)?.[1] || 'done' +const config={datafile:'/store/data.jsontag',commandLog:'/store/command-log.jsontag',commandStatus:'/store/command-status.jsontag',commandsFile:'/app/commands.mjs',indexFile:'/app/index.mjs'} +console.log('ENVELOPE '+JSON.stringify({kernel:await fs.readFile('/proc/version','utf8'),mounts:await fs.readFile('/proc/mounts','utf8'),writeCache:await fs.readFile('/sys/block/vda/queue/write_cache','utf8')})) +const exists=await fs.access(config.datafile).then(()=>true,()=>false) +if(exists){ + const report=await inspectStore(config) + console.log('VERIFY '+JSON.stringify({ready:report.ready,errors:report.errors,commands:report.commands.map(c=>({id:c.id,status:c.status,present:c.present})),persons:report.data?.persons,unsyncedPresent:await fs.access('/store/unsynced').then(()=>true,()=>false)})) + process.exit(0) +} +await publishFile(config.datafile,serialize(JSONTag.parse('{"persons":[]}'))) +await publishFile(config.commandLog,'');await publishFile(config.commandStatus,'') +if(mode==='unsynced'){await fs.writeFile('/store/unsynced','not guaranteed durable');console.log('CUT_UNSYNCED');await new Promise(()=>{})} +await fs.writeFile('/app/commands.mjs',`export default {addPerson(data,command){${mode==='active'?"console.log('CUT_ACTIVE');while(true){}":mode==='accepted'?"while(true){}":"data.persons.push(command.value)"}}}`) +await fs.writeFile('/app/index.mjs','export default {update(){},create(){},load(){return {}}}') +await server.run({...config,port:3000,maxWorkers:1,commandTimeout:0,wwwroot:'/app/www'}) +await new Promise(r=>setTimeout(r,100)) +const before=performance.now() +const response=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id:'A',name:'addPerson',value:{name:'A'}})}) +console.log('ACCEPTED '+JSON.stringify({http:response.status,ms:performance.now()-before})) +if(mode==='accepted')console.log('CUT_ACCEPTED') +if(mode==='done'){ + for(;;){const status=await fetch('http://127.0.0.1:3000/command/A').then(r=>r.json());if(status.status==='done')break;await new Promise(r=>setTimeout(r,10))} + const samples=[{id:'A',doneMs:performance.now()-before}] + for(let i=0;i<4;i++){ + const id='L'+i,start=performance.now() + const accepted=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id,name:'addPerson',value:{name:id}})}) + if(accepted.status!==202)throw new Error('Acceptance failed') + const acceptMs=performance.now()-start + for(;;){const status=await fetch('http://127.0.0.1:3000/command/'+id).then(r=>r.json());if(status.status==='done')break;await new Promise(r=>setTimeout(r,10))} + samples.push({id,acceptMs,doneMs:performance.now()-start}) + } + console.log('CUT_DONE '+JSON.stringify({samples})) +} +await new Promise(()=>{}) diff --git a/scripts/recover.mjs b/scripts/recover.mjs new file mode 100644 index 0000000..07a447a --- /dev/null +++ b/scripts/recover.mjs @@ -0,0 +1,58 @@ +import process from 'node:process' +import fs from 'node:fs/promises' +import path from 'node:path' +import {planRecovery,writePlan,applyRecovery,backupStore,restoreBackup,releaseOfflineLocks,finishRecovery,verifyCandidate} from '../src/admin-recovery.mjs' +import {inspectStore} from '../src/store-inspection.mjs' + +const [action,...args] = process.argv.slice(2) +const options = {} +for (let i=0;i JSON.parse(await fs.readFile(file,'utf8')) +const required = key => { if (typeof options[key] !== 'string') throw new Error(`--${key} is required`); return options[key] } +async function main() { + switch (action) { + case 'inspect': { + const plan = await planRecovery(await read(required('store')), {quiescent:options.quiescent === true}) + await writePlan(required('out'),plan) + return plan + } + case 'apply': + return applyRecovery(await read(required('plan')), {to:required('to'),auditDir:required('audit-dir'), + approveRerun:typeof options['approve-rerun'] === 'string' ? options['approve-rerun'].split(',').filter(Boolean) : [], + operator:required('operator'),reason:required('reason')}) + case 'finish': + return finishRecovery(required('audit-dir'),{operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped']===true}) + case 'backup': + return backupStore(await read(required('store')), {to:required('to'),quiescent:options.quiescent === true}) + case 'restore': + return restoreBackup(required('backup'), {to:required('to'),auditDir:required('audit-dir'), + source:options.store ? await read(options.store) : undefined,sourceQuiescent:options.quiescent===true}) + case 'unlock': { + const store = await read(required('store')) + const release = await releaseOfflineLocks(store, {operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped'] === true}) + const audit = path.resolve(required('out')) + // writePlan enforces new output outside every source data directory. + await writePlan(audit, {...release,config:(await inspectStore(store)).config}) + await release.finish() + return {audit, released:release.removed.map(item=>item.lock)} + } + case 'verify': { + const report = await read(required('report')) + const current = await verifyCandidate(report) + const output = path.resolve(required('out')) + await writePlan(output,{config:report.config,kind:'simplystore-activation-preview',store:{...report.config,...report.code},committed:current.committed, + instruction:'Stop the old service, preserve its configuration, and explicitly select this store. This command does not deploy or switch a service.'}) + return {verified:true,activationPreview:output} + } + default: + throw new Error('Usage: recover.mjs inspect|apply|backup|restore|unlock|finish|verify --store config.json ...; see docs/recovery.md') + } +} +main().then(result=>console.log(JSON.stringify(result,null,2))).catch(error=>{ + console.error(error.message) + process.exitCode = 1 +}) diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs new file mode 100644 index 0000000..c9c47f4 --- /dev/null +++ b/src/admin-recovery.mjs @@ -0,0 +1,290 @@ +import {Buffer} from 'node:buffer' +import fs from 'node:fs/promises' +import path from 'node:path' +import {fileURLToPath} from 'node:url' +import JSONTag from '@muze-nl/jsontag' +import Parser from '@muze-nl/od-jsontag/src/parse.mjs' +import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' +import {inspectStore, inventory, hash, mutableDirectories, storePaths} from './store-inspection.mjs' +import {acquireOwnership} from './store-ownership.mjs' +import {publishFile, appendRecord, durableMkdir, syncDirectory, syncFile} from './storage.mjs' +import {executeWorker} from './execute-worker.mjs' +import {nextActiveCommandStatus} from './recovery.mjs' +import {loadIntegrityManifest, verifyIntegrity} from './integrity.mjs' + +const defaultWorker = fileURLToPath(new URL('./command-worker.mjs', import.meta.url)) +const defaultIndex = fileURLToPath(new URL('./index.mjs', import.meta.url)) +const defaultCommands = fileURLToPath(new URL('./commands.mjs', import.meta.url)) +function deployment(options) { + return {commandsFile:path.resolve(options.commandsFile || defaultCommands), + indexFile:path.resolve(options.indexFile || defaultIndex), + commandWorker:path.resolve(options.commandWorker || defaultWorker)} +} +async function codeHashes(code) { + return Object.fromEntries(await Promise.all(Object.entries(code).map(async ([key,file]) => [key,hash(await fs.readFile(file))]))) +} +async function canonicalNew(file) { + const absolute = path.resolve(file) + try { await fs.lstat(absolute); throw new Error(`Destination already exists: ${absolute}`) } + catch (error) { if (error.code !== 'ENOENT') throw error } + return path.join(await fs.realpath(path.dirname(absolute)), path.basename(absolute)) +} +function outside(target, roots) { + for (const root of roots) if (target === root || target.startsWith(root + path.sep) || root.startsWith(target + path.sep)) { + throw new Error(`Overlapping source/destination: ${target} and ${root}`) + } +} +async function assertFresh(plan) { + const files = await inventory(plan.config) + if (hash(JSON.stringify(files)) !== plan.fingerprint) throw new Error('Stale recovery plan: source inventory changed') + if (JSON.stringify(await codeHashes(plan.code)) !== JSON.stringify(plan.codeHashes)) throw new Error('Stale recovery plan: selected code changed') +} +export async function planRecovery(options, {quiescent = false} = {}) { + const report = await inspectStore(options) + const code = deployment(options) + const candidates = report.commands.filter(c => ['accepted','active'].includes(c.status) || (c.status === 'done' && !c.present)) + const blocks = [...report.errors] + for (const command of report.commands) { + if (command.condition === 'corrupt' || (command.present && command.status !== 'done')) blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) + } + for (const command of candidates) { + if (command.laterDatasets.length) blocks.push(`${command.id}: later accepted datasets exist: ${command.laterDatasets.join(',')}`) + } + return {kind:'simplystore-recovery-plan', config:report.config, code, codeHashes:await codeHashes(code), + files:report.files, fingerprint:report.fingerprint, actionable:quiescent && blocks.length === 0, + blocks, rerun:candidates.map(c => c.id), committed:report.committed, + commands:report.commands.map(record => Object.fromEntries(Object.entries(record).filter(([key])=>key!=='command'))), warnings:report.warnings, + attestation:'Approval asserts source quiescence, complete authoritative history, complete logged command inputs, original command meaning, and assessed external effects.'} +} +export async function writePlan(file, plan) { + const target = await canonicalNew(file) + const roots = await Promise.all(mutableDirectories(plan.config).map(dir => fs.realpath(dir))) + outside(target, roots) + await publishFile(target, JSON.stringify(plan, null, 2) + '\n') +} +async function copyStore(config, files, target) { + const directories = [...new Set(Object.keys(files).map(file => path.dirname(file)))].sort() + let common = directories[0] + while (!directories.every(dir => dir === common || dir.startsWith(common + path.sep) || common === path.parse(common).root)) common = path.dirname(common) + const mapFile = file => path.join(target, path.relative(common,file)) + for (const directory of directories) await durableMkdir(path.join(target,path.relative(common,directory))) + for (const [file,digest] of Object.entries(files)) { + if (digest === null) continue + const bytes = await fs.readFile(file) + if (hash(bytes) !== digest) throw new Error(`Source changed while copying: ${file}`) + await publishFile(mapFile(file), bytes, {mode:(await fs.stat(file)).mode & 0o777}) + } + const result = {...config, datafile:mapFile(config.datafile), commandLog:mapFile(config.commandLog), + commandStatus:mapFile(config.commandStatus), integrityFile:mapFile(config.integrityFile), + requiredFiles:config.requiredFiles.map(mapFile)} + if (config.schemaFile) result.schemaFile = mapFile(config.schemaFile) + return result +} +async function prepareWorkspace(target, audit, roots) { + target = await canonicalNew(target); audit = await canonicalNew(audit) + outside(target, roots); outside(audit, roots); outside(target,[audit]) + await durableMkdir(target); await durableMkdir(audit) + return {target,audit} +} + +export async function applyRecovery(plan, {to, auditDir, approveRerun = [], operator, reason} = {}) { + if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) throw new Error('Plan is not actionable; inspect a quiescent source and resolve blocking evidence') + if (!operator || !reason) throw new Error('Operator and assessment reason are required') + if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) throw new Error('Explicit approval must match the complete ordered rerun list') + const sourceOwner = await acquireOwnership(mutableDirectories(plan.config)) + let candidateOwner + try { + await assertFresh(plan) + // Do not trust editable plan fields to bypass the source predicate. + const fresh = await planRecovery({...plan.config,...plan.code}, {quiescent:true}) + if (!fresh.actionable || JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun)) throw new Error('Source no longer satisfies ordered recovery predicate') + const workspace = await prepareWorkspace(to, auditDir, sourceOwner.directories) + const {target,audit} = workspace + const rootOwner = await acquireOwnership([target],{purpose:'recovery',auditDir:audit}) + // A root lock protects partially copied candidates before layout is complete. + const original = await copyStore(plan.config, plan.files, path.join(audit,'original')) + const config = await copyStore(plan.config, plan.files, target) + candidateOwner = await acquireOwnership(mutableDirectories(config).filter(dir=>dir!==target),{ancestorToken:rootOwner.token}) + const planHash = hash(JSON.stringify(plan)) + await publishFile(path.join(audit,'plan.json'), JSON.stringify(plan,null,2)) + await publishFile(path.join(audit,'original.json'), JSON.stringify(original,null,2)) + const journal = path.join(audit,'attempts.jsonl') + await appendRecord(journal, JSON.stringify({event:'authorized',planHash,operator,reason,rerun:plan.rerun,config,attestation:plan.attestation})) + const inspection = await inspectStore(config) + if (inspection.errors.length) throw new Error(inspection.errors.join('; ')) + const parser = new Parser() + let data + for (const bytes of inspection.buffers) data = parser.parse(bytes) + // Serialize committed prefix, rebuilding parser indexes without custom hooks. + let meta = {index:{id:new Map()}, data:path.dirname(config.datafile), parts:inspection.committed.length} + const buffers = [serialize(data,{meta})] + if (config.schemaFile) meta.schema = JSONTag.parse(await fs.readFile(config.schemaFile,'utf8')) + const expectedManifest = plan.files[plan.config.integrityFile] != null ? await loadIntegrityManifest(plan.config.integrityFile) : null + for (const id of plan.rerun) { + await assertFresh(plan) + const command = fresh.commands.find(c => c.id === id) + const active = nextActiveCommandStatus(id, command.history.at(-1)) + await appendRecord(journal, JSON.stringify({event:'attempt',id,attempt:active.attempt,planHash})) + await appendRecord(config.commandStatus, JSONTag.stringify(active)) + const result = await executeWorker(plan.code.commandWorker, {id,command:command.line,meta,data:buffers, + commandsFile:plan.code.commandsFile,indexFile:plan.code.indexFile,datafile:config.datafile, + // Verify a retained digest before allowing replacement manifest/done records. + integrityFile:null, integrityRequired:false}, 30000) + if (!result || result.storageFailure || result.status === 'failed' || result.status === 'unsafe' || result.code >= 300) { + await appendRecord(journal, JSON.stringify({event:'stopped',id,result:result && {status:result.status,message:result.message}})) + throw new Error(`Recovery attempt ${id} uncertain or failed; inspect before approving another attempt`) + } + for (const file of config.requiredFiles) await syncFile(file) + if (expectedManifest && command.status === 'done') { + verifyIntegrity(expectedManifest, plan.config.integrityFile, command.file, result.data, {required:true}) + } + if (config.integrity || expectedManifest) { + const {appendIntegrityRecord} = await import('./integrity.mjs') + const targetFile = path.join(path.dirname(config.datafile),path.basename(command.file)) + await appendIntegrityRecord(config.integrityFile,targetFile,result.data) + } + await appendRecord(config.commandStatus, JSONTag.stringify({command:id,code:200,status:'done'})) + buffers.push(result.data); Object.assign(meta,result.meta) + await appendRecord(journal, JSON.stringify({event:'done',id,attempt:active.attempt})) + } + const complete = await inspectStore(config) + if (!complete.ready) throw new Error('Recovered candidate is not ready: '+JSON.stringify({errors:complete.errors,commands:complete.commands.filter(c=>c.problem)})) + const report = {kind:'simplystore-recovery-complete',planHash,config,code:plan.code,codeHashes:plan.codeHashes,files:complete.files, + fingerprint:complete.fingerprint,committed:complete.committed,operator,reason,warnings:complete.warnings} + await publishFile(path.join(audit,'complete.json'), JSON.stringify(report,null,2)) + await candidateOwner.release(); candidateOwner = null + await rootOwner.release() + return report + } finally { + // On failure retain candidate locks and its audit. The unchanged source can be released. + await sourceOwner.release() + } +} + +export async function backupStore(options, {to, quiescent = false} = {}) { + if (!quiescent) throw new Error('Backup requires a stopped/quiescent source') + const config = storePaths(options), owner = await acquireOwnership(mutableDirectories(config)) + try { + const report = await inspectStore(config) + if (!report.ready) throw new Error('Backup source is not a complete validated store') + const target = await canonicalNew(to); outside(target, owner.directories) + await durableMkdir(target) + const backupOwner = await acquireOwnership([target]) + const copied = await copyStore(config, report.files, path.join(target,'store')) + const check = await inspectStore(copied) + if (!check.ready) throw new Error('Backup reconstruction failed') + if (hash(JSON.stringify(await inventory(config))) !== report.fingerprint) throw new Error('Backup source changed') + const manifest = {kind:'simplystore-backup-complete',root:target,config:copied,files:check.files,fingerprint:check.fingerprint, + committed:check.committed,sourceFingerprint:report.fingerprint,warnings:report.warnings} + await publishFile(path.join(target,'complete.json'),JSON.stringify(manifest,null,2)) + await backupOwner.release() + return manifest + } finally { await owner.release() } +} + +export async function restoreBackup(backupDirectory, {to, auditDir, source, sourceQuiescent=false} = {}) { + const backup = await fs.realpath(backupDirectory) + const manifest = JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) + if (manifest.kind !== 'simplystore-backup-complete') throw new Error('Backup is incomplete or unrecognized') + const oldRoot=manifest.root + if(typeof oldRoot!=='string' || !path.isAbsolute(oldRoot) || manifest.fingerprint!==hash(JSON.stringify(manifest.files)))throw new Error('Invalid backup inventory') + const rebase=file=>{ + if(typeof file!=='string' || file!==path.resolve(file) || !file.startsWith(oldRoot+path.sep))throw new Error('Backup manifest points outside backup') + return path.join(backup,path.relative(oldRoot,file)) + } + manifest.files=Object.fromEntries(Object.entries(manifest.files).map(([file,digest])=>[rebase(file),digest])) + manifest.fingerprint=hash(JSON.stringify(manifest.files)) + for(const key of ['datafile','commandLog','commandStatus','integrityFile','schemaFile'])if(manifest.config[key])manifest.config[key]=rebase(manifest.config[key]) + manifest.config.requiredFiles=manifest.config.requiredFiles.map(rebase) + if (mutableDirectories(manifest.config).some(dir=>dir!==backup && !dir.startsWith(backup+path.sep))) throw new Error('Backup configuration points outside backup') + if (source && !sourceQuiescent) throw new Error('Source comparison requires a stopped/quiescent source') + const owner = await acquireOwnership([...mutableDirectories(manifest.config),...(source ? mutableDirectories(storePaths(source)) : [])]) + try { + const report = await inspectStore(manifest.config) + if (!report.ready || report.fingerprint !== manifest.fingerprint) throw new Error('Backup contents differ from completed manifest') + const {target,audit} = await prepareWorkspace(to,auditDir,[backup,...owner.directories]) + const rootOwner = await acquireOwnership([target],{purpose:'restore',auditDir:audit}) + const config = await copyStore(manifest.config,manifest.files,target) + const candidate = await acquireOwnership(mutableDirectories(config).filter(dir=>dir!==target),{ancestorToken:rootOwner.token}) + const check = await inspectStore(config) + if (!check.ready) throw new Error('Restored candidate failed validation') + let missingFromBackup = null, sameBase = null + if (source) { + const retained = await inspectStore(source) + sameBase=retained.files[retained.config.datafile]===check.files[check.config.datafile] + missingFromBackup = retained.commands.filter(c => c.accepted && (!sameBase || !check.commands.some(b => b.id === c.id && b.line === c.line && b.status===c.status && (c.status!=='done' || !c.present || retained.files[c.file]===check.files[b.file])))).map(c=>c.id) + } + const complete = {kind:'simplystore-restore-complete',config,files:check.files,fingerprint:check.fingerprint, + committed:check.committed,missingFromBackup,sameBase,warnings:check.warnings, + coverage:source ? 'Compared with supplied retained history; completeness still requires external evidence' : 'Loss relative to current source is unknown'} + await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) + await candidate.release(); await rootOwner.release() + return complete + } finally { await owner.release() } +} + +// Explicit offline operation: no PID-age or liveness heuristic can authorize it. +export async function releaseOfflineLocks(options, {operator, reason, confirmedStopped = false} = {}) { + if (!confirmedStopped || !operator || !reason) throw new Error('Offline release requires confirmed stopped writer, operator, and reason') + const config = storePaths(options), report = await inspectStore(config) + const removed = [] + const lockDirectories = new Set(await Promise.all(mutableDirectories(config).map(dir=>fs.realpath(dir)))) + for (const directory of [...lockDirectories]) { + for (let parent=path.dirname(directory); path.dirname(parent)!==parent; parent=path.dirname(parent)) { + try { await fs.access(path.join(parent,'.simplystore-lock')); lockDirectories.add(parent) } catch(error) { if(error.code!=='ENOENT')throw error } + } + } + for (const directory of [...lockDirectories].sort().reverse()) { + const lock = path.join(directory,'.simplystore-lock') + let entries + try { entries = await fs.readdir(lock) } catch (error) { if (error.code === 'ENOENT') continue; throw error } + if (entries.some(entry => entry !== 'owner.json' && !entry.endsWith('.tmp'))) throw new Error(`Unknown lock contents: ${lock}`) + const saved = [] + for (const entry of entries) saved.push({name:entry,bytes:(await fs.readFile(path.join(lock,entry))).toString('base64')}) + const ownerRecord=saved.find(item=>item.name==='owner.json') + const owner=ownerRecord ? JSON.parse(Buffer.from(ownerRecord.bytes,'base64').toString()) : null + if (owner?.purpose==='recovery' && report.ready) { + let completed + try { completed=JSON.parse(await fs.readFile(path.join(owner.auditDir,'complete.json'),'utf8')) } catch { /* Missing report is not completion evidence. */ } + if(completed?.fingerprint!==report.fingerprint) throw new Error('Completed-looking recovery candidate requires finish with its retained audit before unlocking') + } + removed.push({lock,ownerFiles:saved}) + } + // Return preview only; the CLI persists its external audit before calling finish. + return {inspection:{ready:report.ready,errors:report.errors,commands:report.commands.map(c=>({id:c.id,status:c.status,problem:c.problem}))}, + operator,reason,removed,async finish() { + for (const {lock,ownerFiles} of removed) { + for (const entry of ownerFiles) { + if ((await fs.readFile(path.join(lock,entry.name))).toString('base64') !== entry.bytes) throw new Error('Lock changed after preview') + } + for (const entry of ownerFiles) await fs.unlink(path.join(lock,entry.name)) + await fs.rmdir(lock); await syncDirectory(path.dirname(lock)) + } + }} +} + +export async function finishRecovery(auditDirectory, {operator,reason,confirmedStopped=false}={}) { + if(!confirmedStopped || !operator || !reason)throw new Error('Finishing requires confirmed stopped writer and administrator assessment') + const audit=await fs.realpath(auditDirectory) + const plan=JSON.parse(await fs.readFile(path.join(audit,'plan.json'),'utf8')) + const authorization=JSON.parse((await fs.readFile(path.join(audit,'attempts.jsonl'),'utf8')).split('\n')[0]) + if(authorization.planHash!==hash(JSON.stringify(plan)))throw new Error('Audit authorization does not match retained plan') + await assertFresh(plan) + const report=await inspectStore(authorization.config) + if(!report.ready || plan.rerun.some(id=>!report.committed.includes(id)))throw new Error('Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun') + const complete={kind:'simplystore-recovery-complete',planHash:authorization.planHash,config:report.config, + code:plan.code,codeHashes:plan.codeHashes,files:report.files,fingerprint:report.fingerprint,committed:report.committed,operator,reason,warnings:report.warnings} + await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) + const release=await releaseOfflineLocks(report.config,{operator,reason,confirmedStopped}) + await publishFile(path.join(audit,'finish-unlock.json'),JSON.stringify(release)) + await release.finish() + return complete +} + +export async function verifyCandidate(report) { + if (!['simplystore-recovery-complete','simplystore-restore-complete'].includes(report.kind)) throw new Error('Expected completed recovery/restore report') + const current = await inspectStore(report.config) + if (!current.ready || current.fingerprint !== report.fingerprint) throw new Error('Candidate changed or is incomplete') + if (report.code && JSON.stringify(await codeHashes(report.code)) !== JSON.stringify(report.codeHashes)) throw new Error('Selected code changed after recovery') + return current +} diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 79f7109..1144975 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -2,7 +2,7 @@ import JSONTag from '@muze-nl/jsontag' import {getIndex, resultSet, isChanged} from '@muze-nl/od-jsontag/src/symbols.mjs' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' -import writeFileAtomic from 'write-file-atomic' +import {publishFile as writeFileAtomic, storageError} from './storage.mjs' import { faultPoint } from './faults.mjs' import { appendIntegrityRecord } from './integrity.mjs' import { finalizeIndex } from './index.mjs' @@ -89,24 +89,25 @@ export async function initialize(task) { }) } -export default async function runCommand(commandStr, request) { +export default async function runCommand(commandStr) { let response = { jsontag: true } + let publishing = false try { let task = JSONTag.parse(commandStr, null, metaReadProxy) if (!task.id) { throw new Error('missing command id')} if (!task.name) { throw new Error('missing command name parameter')} if (commands[task.name]) { let time = Date.now() - commands[task.name](dataspace, task, request, metaProxy) + await commands[task.name](dataspace, task, undefined, metaProxy) //TODO: if command/task makes no changes, skip updating data.jsontag and writing it, skip response.data const changes = meta.resultArray.filter(e => e[isChanged]) //FIXME: new entities should also report isChanged = true if (changes.length) { changes.uuid = task.id - index.update(dataspace, meta, changes) + await index.update(dataspace, meta, changes) } const uint8sab = serialize(dataspace, {meta, changes: true}) // serialize only changes response.data = uint8sab @@ -118,6 +119,7 @@ export default async function runCommand(commandStr, request) { //TODO: write data every x commands or x minutes, in seperate thread? let newfilename = basefile + '.' + task.id + '.' + extension + publishing = true await faultPoint('before-command-changeset-write') await writeFileAtomic(newfilename, uint8sab) // Final bytes include new records and mutations made by the custom index hook. @@ -139,7 +141,7 @@ export default async function runCommand(commandStr, request) { } } catch(err) { console.error('task error', err) - throw err + throw publishing ? storageError(err) : err } return response } diff --git a/src/command-worker.mjs b/src/command-worker.mjs index 4b0deff..67248cf 100644 --- a/src/command-worker.mjs +++ b/src/command-worker.mjs @@ -4,6 +4,11 @@ import runCommand, { initialize } from '../src/command-worker-module.mjs' parentPort.on('message', async data => { let result await initialize(data) - result = await runCommand(data.command) + try { + result = await runCommand(data.command) + } catch (error) { + result = {status: 'failed', code: 500, message: error.message, + storageFailure: Boolean(error.storageFailure)} + } parentPort.postMessage(result) }) \ No newline at end of file diff --git a/src/execute-worker.mjs b/src/execute-worker.mjs new file mode 100644 index 0000000..6638155 --- /dev/null +++ b/src/execute-worker.mjs @@ -0,0 +1,23 @@ +import {Worker} from 'node:worker_threads' + +export function executeWorker(filename, task, timeout = 30000) { + return new Promise((resolve, reject) => { + const worker = new Worker(filename) + let settled = false, timer + const finish = async (settle, result) => { + if (settled) return + settled = true + clearTimeout(timer) + try { await worker.terminate() } catch (error) { reject(error); return } + settle(result) + } + worker.on('message', result => { void finish(resolve, result) }) + worker.on('error', error => { void finish(reject, error) }) + worker.on('exit', code => { + if (!settled) void finish(reject, new Error(`Command worker exited without a result (${code})`)) + }) + if (timeout) timer = setTimeout(() => { void finish(resolve, + {status: 'unsafe', code: 504, message: `command worker timed out after ${timeout}ms`}) }, timeout) + try { worker.postMessage(task) } catch (error) { void finish(reject, error) } + }) +} diff --git a/src/index.id.mjs b/src/index.id.mjs index bf20991..cc91bc9 100644 --- a/src/index.id.mjs +++ b/src/index.id.mjs @@ -1,3 +1,4 @@ +import {publishFileSync} from './storage.mjs' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' import { getIndex } from '@muze-nl/od-jsontag/src/symbols.mjs' @@ -18,7 +19,7 @@ export default { } index[key] = entity[getIndex] } - fs.writeFileSync(meta.data+'/index.id.json', JSON.stringify(index)) + publishFileSync(meta.data+'/index.id.json', JSON.stringify(index)) }, update(data, meta, changes) { if (!changes.length) { @@ -31,7 +32,7 @@ export default { index[id] = entry[getIndex] } } - fs.writeFileSync(meta.data+'/index.id.'+changes.uuid+'.json', JSON.stringify(index)) + publishFileSync(meta.data+'/index.id.'+changes.uuid+'.json', JSON.stringify(index)) }, load(uuid=null) { let filename diff --git a/src/index.offset.mjs b/src/index.offset.mjs index c2a9b2a..250fe75 100644 --- a/src/index.offset.mjs +++ b/src/index.offset.mjs @@ -1,7 +1,8 @@ +import {publishFileSync} from './storage.mjs' import fs from 'fs' import path from 'node:path' import { Buffer } from 'node:buffer' -import writeFileAtomic from 'write-file-atomic' +import {publishFile as writeFileAtomic} from './storage.mjs' import { getIndex, position } from '@muze-nl/od-jsontag/src/symbols.mjs' import { scanOdJsonTagRecords } from './recovery.mjs' @@ -24,7 +25,7 @@ export default { const entity = meta.resultArray[i] index[i] = [ entity[position].start, entity[position].end ] } - fs.writeFileSync(meta.data+'/index.offset.json', JSON.stringify(index)) + publishFileSync(meta.data+'/index.offset.json', JSON.stringify(index)) }, update(data, meta, changes) { if (!changes.length) { @@ -37,7 +38,7 @@ export default { index[entry[getIndex]] = [ pos.start, pos.end ] } } - fs.writeFileSync(meta.data+'/index.offset.'+changes.uuid+'.json', JSON.stringify(index)) + publishFileSync(meta.data+'/index.offset.'+changes.uuid+'.json', JSON.stringify(index)) }, load(uuid=null) { let filename diff --git a/src/recovery.mjs b/src/recovery.mjs index dfc1458..a3093c1 100644 --- a/src/recovery.mjs +++ b/src/recovery.mjs @@ -1,6 +1,5 @@ import fs from 'fs' import JSONTag from '@muze-nl/jsontag' -import { appendFile } from './util.mjs' export const committedCommandStatus = 'done' export const pendingCommandStatus = 'accepted' @@ -100,7 +99,6 @@ export function loadCommandLog(status, commandLog, taskDefaults = {}) { ...taskDefaults, id: command.id, command: line, - request: null }) } } @@ -116,37 +114,11 @@ function readAttempt(record, fallback = 1) { return fallback } -export async function recoverActiveCommands(status, commandStatusFile, options = {}) { - const maxCrashAttempts = options.maxCrashAttempts ?? defaultMaxCommandCrashAttempts - - for (const [commandId, command] of status.entries()) { - if (command?.status !== activeCommandStatus) { - continue - } - const attempt = readAttempt(command) - let nextStatus - if (attempt >= maxCrashAttempts) { - nextStatus = { - command: commandId, - code: 500, - status: unsafeCommandStatus, - message: `Command marked unsafe after ${attempt} crashed active attempt(s)`, - attempt - } - } else { - nextStatus = { - command: commandId, - code: 202, - status: pendingCommandStatus, - message: `Retrying command after ${attempt} crashed active attempt(s)`, - attempt - } - } - status.set(commandId, nextStatus) - await appendFile(commandStatusFile, JSONTag.stringify(nextStatus)) - } - - return status +export async function recoverActiveCommands(status) { + if ([...status.values()].some(command => command?.status === activeCommandStatus)) { + throw new RecoveryIntegrityError('Active command requires administrator assessment; automatic rerun is disabled') + } + return status } export function nextActiveCommandStatus(commandId, currentStatus) { diff --git a/src/server.mjs b/src/server.mjs index 5ce4609..30d199c 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -1,3 +1,4 @@ +import {setImmediate} from 'node:timers' import express from 'express' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' @@ -8,7 +9,11 @@ import {appendFile} from './util.mjs' import path from 'path' import httpStatusCodes from './statusCodes.mjs' import process from 'node:process' -import { getCommittedCommandIds, loadCommandLog, loadCommandStatus, nextActiveCommandStatus, pendingCommandStatus, recoverActiveCommands, unsafeCommandStatus } from './recovery.mjs' +import { nextActiveCommandStatus } from './recovery.mjs' +import {serialWriter, syncFile} from './storage.mjs' +import {inspectStore, storePaths, mutableDirectories, validCommandId} from './store-inspection.mjs' +import {acquireOwnership} from './store-ownership.mjs' +import {executeWorker} from './execute-worker.mjs' import { assertRuntimeEnvironmentConfiguration } from './runtime-environment.mjs' import { faultPoint } from './faults.mjs' import { getDefaultIntegrityFile } from './integrity.mjs' @@ -60,7 +65,6 @@ async function main(options) { const commandStatus = options.commandStatus || './command-status.jsontag' const integrityFile = options.integrityFile || getDefaultIntegrityFile(datafile) const integrityEnabled = Boolean(options.integrity || options.integrityFile || fs.existsSync(integrityFile)) - const maxCommandCrashAttempts = options.maxCommandCrashAttempts ?? 2 const access = options.access || null const timeout = options.timeout || 1000 const slowTimeout = options.slowTimeout || 10000 @@ -75,29 +79,36 @@ async function main(options) { limit: '50MB' })) - let status = loadCommandStatus(commandStatus) - status = await recoverActiveCommands(status, commandStatus, { - maxCrashAttempts: maxCommandCrashAttempts - }) - + const config = storePaths({...options, datafile, commandLog, commandStatus, integrity: integrityEnabled, + ...(integrityEnabled ? {integrityFile} : {})}) + const ownership = await acquireOwnership(mutableDirectories(config)) + let inspection try { - let data = await loadData(getCommittedCommandIds(status)) // only completed command ids are used to generate filenames of committed changes + inspection = await inspectStore(config) + if (!inspection.ready) throw new Error('Administrative recovery required: ' + + [...inspection.errors, ...inspection.commands.filter(c => c.problem).map(c => c.problem), + ...inspection.commands.filter(c => c.status === 'accepted' || c.status === 'active').map(c => `${c.id}: ${c.status}; administrator assessment required`), + ...inspection.commands.filter(c => c.present && c.status !== 'done').map(c => `${c.id}: uncommitted dataset`)].join('; ')) + for (const [file, digest] of Object.entries(inspection.files)) if (digest !== null) await syncFile(file) + const data = await loadData(inspection.committed) jsontagBuffers = [data.data] meta = data.meta - } catch(err) { - console.error('ERROR: SimplyStore cannot load '+datafile, err) - process.exit(1) + } catch (error) { + await ownership.release() + throw error + } + const status = new Map(inspection.commands.map(c => [c.id, c.history.at(-1)])) + const commandQueue = [] + const acceptSerial = serialWriter() + let storageFailed = false, closing = false + let runner = Promise.resolve() + function failStorage(error) { + if (storageFailed) return + storageFailed = true + console.error('Storage outcome uncertain; stopping mutation. Administrator recovery required:', error) + // Retain ownership evidence; a process exit must never imply a rollback. + setImmediate(() => process.exit(1)) } - - let commandQueue = loadCommandLog(status, commandLog, { - meta, - data:jsontagBuffers, - commandsFile, - indexFile, - datafile, - integrityFile: integrityEnabled ? integrityFile : null, - integrityRequired: integrityEnabled - }) const queryWorkerInitTask = () => { return { @@ -119,7 +130,6 @@ async function main(options) { let queryWorkerPool = new WorkerPool(maxWorkers, queryWorker, queryWorkerInitTask()) let slowQueryWorkerPool = new WorkerPool(1, queryWorker, slowQueryWorkerInitTask()) - let commandWorkerInstance let commandRunnerActive = false server.get('/query/', (req,next) => handleGetQuery(req,next)) @@ -135,24 +145,24 @@ async function main(options) { server.use(express.static(wwwroot)) - try { - await fetch(`http://localhost:${port}`, { - signal: AbortSignal.timeout(2000) - }) - console.error(`Port ${port} is already occupied, aborting.`) - process.exit() - } catch { + const listener = server.listen(port, () => { + console.log('SimplyStore listening on port '+port) + }) + listener.on('error', error => { failStorage(error) }) + async function shutdown() { + if (closing) return + closing = true + listener.close() try { - await drainCommandQueue() - } catch { - // console.log(err) // ignore errors here, already logged to console - } - server.listen(port, () => { - console.log('SimplyStore listening on port '+port) - let used = Math.round(process.memoryUsage().rss / 1024 / 1024); - console.log(`(${used} MB)`); - }) + await acceptSerial(async () => {}) + await runner + queryWorkerPool.close(); slowQueryWorkerPool.close() + if (!storageFailed) await ownership.release() + process.exit(storageFailed ? 1 : 0) + } catch (error) { failStorage(error) } } + process.once('SIGTERM', shutdown) + process.once('SIGINT', shutdown) /* ------ */ @@ -293,36 +303,21 @@ async function main(options) { } async function handlePostCommand(req, res) { - let commandId = await checkCommand(req, res) - if (!commandId) { - return - } + if (storageFailed || closing) return sendResponse({code:503, body: JSON.stringify({message:'Store is unavailable'})}, res) try { - let commandStr = req.body.toString() - let request = { - method: req.method, - url: req.originalUrl, - query: req.query - } - - commandQueue.push({ - id:commandId, - command:commandStr, - request, - meta, - data:jsontagBuffers, - commandsFile, - indexFile, - datafile, - integrityFile: integrityEnabled ? integrityFile : null, - integrityRequired: integrityEnabled + await acceptSerial(async () => { + if (storageFailed) throw new Error('Store is unavailable') + if (closing) return sendResponse({code:503,body:JSON.stringify({message:'Store is closing'})},res) + const accepted = await checkCommand(req, res) + if (!accepted) return + if (storageFailed) throw new Error('Store failed during acceptance') + commandQueue.push({id: accepted.id, command: accepted.line}) + sendResponse({code:202, body:JSON.stringify(status.get(accepted.id))}, res) }) - await drainCommandQueue() - } catch(err) { - let s = {code:err.code||500, status:'failed', message:err.message, details:err.details} - status.set(commandId, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:commandId}, s))) - console.error('ERROR: SimplyStore cannot run command ', commandId, err) + void drainCommandQueue() + } catch (error) { + if (!res.headersSent) sendResponse({code:500, body:JSON.stringify({message:'Storage outcome uncertain'})}, res) + failStorage(error) } } @@ -342,156 +337,49 @@ async function main(options) { } } - async function drainCommandQueue() { - if (commandRunnerActive) { - return - } + function drainCommandQueue() { + if (commandRunnerActive || storageFailed) return runner commandRunnerActive = true - try { - let result - do { - result = await runNextCommand() - } while(result) - } finally { - commandRunnerActive = false - } - } - - async function runNextCommand() { - return new Promise((mainResolve, mainReject) => { - if (commandWorkerInstance) { - mainResolve(false) - return - } - let command = commandQueue.shift() - while (command && status.get(command.id)?.status !== pendingCommandStatus) { - console.log('skipping non-pending command', command.id) - command = commandQueue.shift() - } - if (command) { - console.log('starting command',command.id) - let start = (resolve, reject) => { - const worker = new Worker(commandWorker) - commandWorkerInstance = worker - let settled = false - let timeoutId - const finish = async (settle, value) => { - if (settled) { - return - } - settled = true - if (timeoutId) { - clearTimeout(timeoutId) - } - await worker.terminate() - if (commandWorkerInstance === worker) { - commandWorkerInstance = null - } - settle(value) - } - if (commandTimeout) { - timeoutId = setTimeout(() => { - const error = new WorkerTimeoutError('command worker', commandTimeout) - void finish(resolve, { - code: error.code, - status: unsafeCommandStatus, - message: error.message, - details: { - timeout: error.timeout, - workerKind: error.workerKind - } - }) - }, commandTimeout) + runner = (async () => { + try { + while (commandQueue.length && !storageFailed) { + const command = commandQueue.shift() + console.log('starting command', command.id) + const active = nextActiveCommandStatus(command.id, status.get(command.id)) + await appendFile(commandStatus, JSONTag.stringify(active)) + if (storageFailed) throw new Error('Store failed before execution') + status.set(command.id, active) + await faultPoint('after-active-status-before-command-worker') + const result = await executeWorker(commandWorker, {...command, + meta, data: jsontagBuffers, commandsFile, indexFile, datafile, + integrityFile: integrityEnabled ? integrityFile : null, + integrityRequired: integrityEnabled}, commandTimeout) + if (storageFailed) throw new Error('Store failed during execution') + if (result?.storageFailure) throw new Error(result.message || 'Worker persistence failure') + if (!result || result.status === 'failed' || result.status === 'unsafe' || result.code >= 300) { + const terminal = {command: command.id, status: result?.status === 'unsafe' ? 'unsafe' : 'failed', + code: result?.code || 500, message: result?.message || 'Command failed', attempt: active.attempt} + await appendFile(commandStatus, JSONTag.stringify(terminal)) + status.set(command.id, terminal) + continue } - worker.on('message', result => { - void finish(resolve, result) - }) - worker.on('error', error => { - void finish(reject, error) - }) - try { - worker.postMessage(command) - } catch (error) { - void finish(reject, error) + for (const file of config.requiredFiles) await syncFile(file) + await faultPoint('before-command-done-status') + const done = {command: command.id, code:200, status:'done'} + await appendFile(commandStatus, JSONTag.stringify(done)) + await faultPoint('after-command-done-status-before-query-update') + status.set(command.id, done) + if (result.data) { + jsontagBuffers.push(result.data) + Object.assign(meta, result.meta) + const task = {name:'update', req:{body:result.data, meta}} + queryWorkerPool.update(task); slowQueryWorkerPool.update(task) } } - const activeStatus = nextActiveCommandStatus(command.id, status.get(command.id)) - status.set(command.id, activeStatus) - void (async () => { - await appendFile(commandStatus, JSONTag.stringify(activeStatus)) - await faultPoint('after-active-status-before-command-worker') - start( - // resolve() - async (data) => { - let s - if (data?.status === unsafeCommandStatus) { - s = { - code: data.code || 504, - status: unsafeCommandStatus, - message: data.message, - details: data.details, - attempt: activeStatus.attempt - } - status.set(command.id, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - mainResolve(s) - } else if (!data || (data.code>=300 && data.code<=499)) { - console.error('ERROR: SimplyStore cannot run command ', command.id, data) - if (!data?.code) { - s = {code: 500, status: "failed"} - } else { - s = {code: data.code, status: "failed", message: data.message, details: data.details} - } - status.set(command.id, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - mainReject(s) - } else { - await faultPoint('before-command-done-status') - s = {code: 200, status: "done"} - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - await faultPoint('after-command-done-status-before-query-update') - status.set(command.id, s) - if (data.data) { // data has changed, commands may do other things instead of changing data - jsontagBuffers.push(data.data) // push changeset to jsontagBuffers so that new query workers get all changes from scratch - Object.assign(meta, data.meta) - const updateTask = { - name: 'update', - req: { - body: jsontagBuffers[jsontagBuffers.length-1], // only add the last change, update tasks for earlier changes have already been sent - meta - } - } - queryWorkerPool.update(updateTask) - slowQueryWorkerPool.update(updateTask) - } - mainResolve(s) - } - }, - //reject() - async (error) => { - let s = {status: "failed", code: error.code, message: error.message, details: error.details} - status.set(command.id, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - console.log('command error', command.id, error) - mainReject(s) - } - ) - })().catch(mainReject) - return - } else { - console.log('no pending commands') - // this code can never be triggered from the post(/command/) route, since it always adds a command to the queue - // so you can only get here from commandWorkerInstance.on() route - // which means that the commandWorkerInstance has finished running the previous command - if (commandWorkerInstance) { - commandWorkerInstance.terminate().then(() => { - mainResolve(false) - }, mainReject) - return - } - mainResolve(false) - } - }) + } catch (error) { failStorage(error) } + finally { commandRunnerActive = false } + })() + return runner } async function checkCommand(req, res) { @@ -515,7 +403,7 @@ async function main(options) { sendResponse({code: 400, body: JSON.stringify(error)}, res) return false } - if (!command || !command.id) { + if (!command || !validCommandId(command.id)) { error = { code: 422, message: "Command has no id", @@ -536,13 +424,13 @@ async function main(options) { sendResponse({code:422, body: JSON.stringify(error)}, res) return false } - await appendFile(commandLog, JSONTag.stringify(command)) //FIXME: this loses request data + const line = JSONTag.stringify(command) + await appendFile(commandLog, line) await faultPoint('after-command-log-before-accepted-status') await appendFile(commandStatus, JSONTag.stringify(commandOK)) status.set(command.id, commandOK) await faultPoint('after-command-accepted-status-before-response') - sendResponse({code: 202, body: JSON.stringify(commandOK)}, res) - return command.id + return {id:command.id, line} } } diff --git a/src/storage.mjs b/src/storage.mjs new file mode 100644 index 0000000..ca679aa --- /dev/null +++ b/src/storage.mjs @@ -0,0 +1,152 @@ +import fs from 'node:fs' +import fsp from 'node:fs/promises' +import path from 'node:path' +import {randomUUID} from 'node:crypto' +import {Buffer} from 'node:buffer' + +export function storageError(error) { + if (!(error instanceof Error)) error = new Error(String(error)) + error.storageFailure = true + return error +} + +// A rejected operation poisons its writer: later operations must not pass it. +export function serialWriter() { + let tail = Promise.resolve() + return operation => { + const result = tail.then(operation) + tail = result + void result.catch(() => {}) + return result + } +} + +async function withHandle(file, flags, operation, mode) { + let handle, failure, result + try { + handle = await fsp.open(file, flags, mode) + result = await operation(handle) + } catch (error) { failure = error } + if (handle) { + try { await handle.close() } catch (error) { failure ??= error } + } + if (failure) throw storageError(failure) + return result +} + +export async function writeAll(handle, data) { + const bytes = Buffer.isBuffer(data) ? data : Buffer.from(data) + let offset = 0 + while (offset < bytes.length) { + const {bytesWritten} = await handle.write(bytes, offset, bytes.length - offset, null) + if (!Number.isInteger(bytesWritten) || bytesWritten <= 0 || bytesWritten > bytes.length - offset) { + throw storageError(new Error('Invalid or zero-progress write')) + } + offset += bytesWritten + } +} + +export async function syncDirectory(directory) { + await withHandle(directory, 'r', handle => handle.sync()) +} + +export async function syncAncestors(directory) { + const device = (await fsp.stat(directory)).dev + for (let current = directory; ; current = path.dirname(current)) { + await syncDirectory(current) + const parent = path.dirname(current) + if (parent === current || (await fsp.stat(parent)).dev !== device) break + } +} + +export async function syncFile(file) { + await withHandle(file, 'r', handle => handle.sync()) + await syncDirectory(path.dirname(file)) +} + +export async function durableMkdir(directory) { + try { await fsp.mkdir(directory) } catch (error) { + if (error.code === 'ENOENT') { + await durableMkdir(path.dirname(directory)) + await fsp.mkdir(directory) + } else if (error.code !== 'EEXIST') throw storageError(error) + } + await syncDirectory(directory) + await syncDirectory(path.dirname(directory)) +} + +export async function publishFile(file, data, {mode} = {}) { + const temporary = `${file}.${randomUUID()}.tmp` + let renamed = false + try { + try { + const stat = await fsp.lstat(file) + if (stat.isSymbolicLink()) throw new Error('Refusing to publish over a symlink') + mode ??= stat.mode & 0o777 + } catch (error) { if (error.code !== 'ENOENT') throw error } + await withHandle(temporary, 'wx', async handle => { + if (mode !== undefined) await handle.chmod(mode) + await writeAll(handle, data) + await handle.sync() + }, mode) + await fsp.rename(temporary, file) + renamed = true + await syncDirectory(path.dirname(file)) + } catch (error) { + if (!renamed) await fsp.unlink(temporary).catch(() => {}) + throw storageError(error) + } +} + +const appenders = new Map() +export function appendRecord(file, record) { + const key = path.resolve(file) + if (!appenders.has(key)) appenders.set(key, serialWriter()) + return appenders.get(key)(async () => { + await withHandle(file, 'a', async handle => { + await writeAll(handle, record + '\n') + await handle.datasync() + }) + await syncDirectory(path.dirname(file)) + }) +} + +// Existing synchronous index hooks keep their contract while gaining barriers. +export function publishFileSync(file, data) { + const temporary = `${file}.${randomUUID()}.tmp` + let fd, failure, renamed = false + try { + const bytes = Buffer.from(data) + let mode + try { + const stat = fs.lstatSync(file) + if (stat.isSymbolicLink()) throw new Error('Refusing to publish over a symlink') + mode = stat.mode & 0o777 + } catch (error) { if (error.code !== 'ENOENT') throw error } + fd = fs.openSync(temporary, 'wx', mode) + if (mode !== undefined) fs.fchmodSync(fd,mode) + let offset = 0 + while (offset < bytes.length) { + const count = fs.writeSync(fd, bytes, offset, bytes.length - offset) + if (!Number.isInteger(count) || count <= 0 || count > bytes.length - offset) throw new Error('Invalid or zero-progress write') + offset += count + } + fs.fsyncSync(fd) + } catch (error) { failure = error } + if (fd !== undefined) { + try { fs.closeSync(fd) } catch (error) { failure ??= error } + } + try { + if (failure) throw failure + fs.renameSync(temporary, file) + renamed = true + fd = fs.openSync(path.dirname(file), 'r') + failure = undefined + try { fs.fsyncSync(fd) } catch (error) { failure = error } + try { fs.closeSync(fd) } catch (error) { failure ??= error } + if (failure) throw failure + } catch (error) { + if (!renamed) { try { fs.unlinkSync(temporary) } catch { /* Preserve the primary failure. */ } } + throw storageError(error) + } +} diff --git a/src/store-inspection.mjs b/src/store-inspection.mjs new file mode 100644 index 0000000..95003f7 --- /dev/null +++ b/src/store-inspection.mjs @@ -0,0 +1,179 @@ +import fs from 'node:fs/promises' +import path from 'node:path' +import JSONTag from '@muze-nl/jsontag' +import Parser from '@muze-nl/od-jsontag/src/parse.mjs' +import {createHash} from 'node:crypto' +import {assertOdJsonTagFraming, getChangesetPath} from './recovery.mjs' +import {getDefaultIntegrityFile, loadIntegrityManifest, verifyIntegrity} from './integrity.mjs' + +export const hash = bytes => createHash('sha256').update(bytes).digest('hex') +export const validCommandId = id => typeof id === 'string' && id.length > 0 && !/[/\\\0]/.test(id) +export function storePaths(options = {}) { + const datafile = path.resolve(options.datafile || './data.od-jsontag') + return {datafile, commandLog: path.resolve(options.commandLog || './command-log.jsontag'), + commandStatus: path.resolve(options.commandStatus || './command-status.jsontag'), + integrityFile: path.resolve(options.integrityFile || getDefaultIntegrityFile(datafile)), + integrity: options.integrity === undefined ? Boolean(options.integrityFile) : Boolean(options.integrity), + requiredFiles: (options.requiredFiles || []).map(file => path.resolve(file)), + ...(options.schemaFile ? {schemaFile: path.resolve(options.schemaFile)} : {})} +} +export function mutableDirectories(config) { + return [...new Set([config.datafile, config.commandLog, config.commandStatus, + config.integrityFile, ...config.requiredFiles].map(file => path.dirname(file)))].sort() +} +export async function inventory(config) { + const files = {} + const expected = new Set([config.datafile,config.commandLog,config.commandStatus,config.integrityFile,...config.requiredFiles,...(config.schemaFile?[config.schemaFile]:[])]) + const extension = path.extname(config.datafile), stem = path.basename(config.datafile,extension)+'.' + for (const directory of mutableDirectories(config)) { + const entries = await fs.readdir(directory, {withFileTypes: true}) + for (const entry of entries.sort((a,b) => a.name.localeCompare(b.name))) { + if (entry.name === '.simplystore-lock') continue + const file = path.join(directory, entry.name) + const dataArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith(stem) && (entry.name.endsWith(extension) || entry.name.endsWith('.tmp')) + const indexArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith('index.') + const temporary = [...expected].some(name=>file.startsWith(name+'.') && file.endsWith('.tmp')) + if (!expected.has(file) && !dataArtifact && !indexArtifact && !temporary) continue + if (entry.isSymbolicLink()) throw new Error(`Symlink in store artifact: ${file}`) + if (entry.isFile()) files[file] = hash(await fs.readFile(file)) + } + } + for (const file of [config.datafile, config.commandLog, config.commandStatus, + ...(config.integrity ? [config.integrityFile] : []), ...config.requiredFiles, + ...(config.schemaFile ? [config.schemaFile] : [])]) { + if (!(file in files)) { + try { files[file] = hash(await fs.readFile(file)) } catch (error) { + if (error.code !== 'ENOENT') throw error + files[file] = null + } + } + } + return Object.fromEntries(Object.entries(files).sort(([a],[b]) => a.localeCompare(b))) +} + +async function records(file, errors, kind) { + let text + try { text = await fs.readFile(file, 'utf8') } catch (error) { + errors.push(`${kind}: ${error.message}`); return [] + } + if (text && !text.endsWith('\n')) errors.push(`${kind}: incomplete final record in ${file}`) + const result = [] + for (const [index, line] of text.split('\n').entries()) { + if (!line) continue + try { + const value = JSONTag.parse(line) + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('record is not an object') + result.push({value, line, lineNumber: index + 1}) + } catch (error) { errors.push(`${kind} ${file}:${index + 1}: ${error.message}`) } + } + return result +} + +// Reads canonical files only; deliberately never imports user handlers/index modules. +export async function inspectStore(options = {}) { + const config = storePaths(options) + const errors = [], warnings = [] + const canonicalPaths = await Promise.all([config.datafile, config.commandLog, config.commandStatus, config.integrityFile].map(async file => path.join(await fs.realpath(path.dirname(file)),path.basename(file)))) + if (new Set(canonicalPaths).size !== canonicalPaths.length) errors.push('Configured canonical artifact paths overlap') + const before = await inventory(config) + const log = await records(config.commandLog, errors, 'command log') + const statuses = await records(config.commandStatus, errors, 'command status') + const commands = [], byId = new Map() + for (const record of log) { + const {id, name} = record.value + if (!validCommandId(id) || typeof name !== 'string' || !name) { + errors.push(`Invalid command at log line ${record.lineNumber}`); continue + } + if (byId.has(id)) { + if (JSONTag.stringify(byId.get(id).command) !== JSONTag.stringify(record.value)) errors.push(`Conflicting command ID ${id}`) + continue + } + const command = {id, position: commands.length, command: record.value, line: record.line, + history: [], accepted: false, status: null, file: getChangesetPath(config.datafile, id)} + byId.set(id, command); commands.push(command) + } + const doneOrder = new Map() + for (const {value, lineNumber} of statuses) { + const command = byId.get(value.command) + if (!command) { errors.push(`Status line ${lineNumber} has no logged command: ${value.command}`); continue } + if (!['accepted', 'active', 'done', 'failed', 'unsafe'].includes(value.status)) { + errors.push(`Unknown status for ${command.id}: ${value.status}`); continue + } + if (value.status === 'accepted') command.accepted = true + if (value.status !== 'accepted' && !command.accepted) errors.push(`No acceptance evidence for ${command.id}`) + if (value.status === 'done') { + doneOrder.delete(command.id); doneOrder.set(command.id, command.position) + } + command.history.push(value); command.status = value.status + } + let lastDonePosition = -1 + for (const [id, position] of doneOrder) { + if (byId.get(id).status !== 'done') continue + if (position < lastDonePosition) errors.push(`Committed execution differs from command-log order at ${id}`) + lastDonePosition = position + } + let manifest + try { + // Preserve optional existence-based compatibility; required integrity is explicit. + const enabled = config.integrity || before[config.integrityFile] != null + if (enabled) { + if (before[config.integrityFile] == null) throw new Error(`Missing integrity manifest ${config.integrityFile}`) + await records(config.integrityFile, errors, 'integrity manifest') + manifest = await loadIntegrityManifest(config.integrityFile) + } + } catch (error) { errors.push(error.message) } + const parser = new Parser() + const buffers = [], committed = [] + let data, prefixValid = errors.length === 0 + async function validateData(file) { + const bytes = await fs.readFile(file) + assertOdJsonTagFraming(bytes, file) + if (manifest) verifyIntegrity(manifest, config.integrityFile, file, bytes, {required: true}) + return bytes + } + try { + const bytes = await validateData(config.datafile) + if (!bytes.length) throw new Error('Empty base dataset') + data = parser.parse(bytes); buffers.push(bytes) + } catch (error) { errors.push(error.message); prefixValid = false } + for (const command of commands) { + command.present = before[command.file] != null + command.condition = command.present ? 'present' : 'missing' + if (!command.accepted) errors.push(`Orphan command without acceptance: ${command.id}`) + if (command.present) { + try { await validateData(command.file) } catch (error) { + command.condition = 'corrupt'; command.problem = error.message + } + } + if (command.status === 'done') { + if (!command.present || command.condition === 'corrupt') { + command.problem ||= `Missing changeset for committed command ${command.id}: ${command.file}` + prefixValid = false + } else if (prefixValid) { + try { + const bytes = await validateData(command.file) + data = parser.parse(bytes); buffers.push(bytes); committed.push(command.id) + } catch (error) { command.problem = error.message; prefixValid = false } + } + } else if (command.status === 'accepted' || command.status === 'active') { + prefixValid = false + } + command.laterDatasets = commands.filter(later => later.position > command.position && later.accepted && before[later.file] != null).map(later => later.id) + } + const extension = path.extname(config.datafile), stem = path.basename(config.datafile, extension) + '.' + const known = new Set(commands.map(command => command.file)) + for (const file of Object.keys(before)) { + if (path.dirname(file) === path.dirname(config.datafile) && path.basename(file).startsWith(stem) && file.endsWith(extension) && file !== config.datafile && file !== config.integrityFile && !known.has(file)) { + errors.push(`Unexplained changeset: ${file}`) + } + } + for (const file of config.requiredFiles) if (before[file] == null) errors.push(`Missing required artifact: ${file}`) + const after = await inventory(config) + if (JSON.stringify(before) !== JSON.stringify(after)) errors.push('Store changed during inspection') + const pending = commands.filter(c => c.status === 'accepted' || c.status === 'active') + const damaged = commands.filter(c => c.problem || (c.present && c.status !== 'done')) + const ready = errors.length === 0 && pending.length === 0 && damaged.length === 0 + warnings.push('History completeness and original code/hidden inputs require independent administrator evidence; current files alone cannot prove them.') + return {config, files: before, fingerprint: hash(JSON.stringify(before)), errors, warnings, + commands, committed, ready, data, buffers} +} diff --git a/src/store-ownership.mjs b/src/store-ownership.mjs new file mode 100644 index 0000000..bad2d09 --- /dev/null +++ b/src/store-ownership.mjs @@ -0,0 +1,51 @@ +import fs from 'node:fs/promises' +import path from 'node:path' +import os from 'node:os' +import process from 'node:process' +import {randomUUID} from 'node:crypto' +import {publishFile, syncDirectory, syncAncestors} from './storage.mjs' + +export async function acquireOwnership(directories, {ancestorToken, purpose, auditDir} = {}) { + const canonical = [...new Set(await Promise.all(directories.map(dir => fs.realpath(dir))))].sort() + for (const directory of canonical) { + for (let parent = path.dirname(directory); ; parent = path.dirname(parent)) { + if (!canonical.includes(parent)) { + const lock = path.join(parent, '.simplystore-lock') + try { + await fs.access(lock) + let owner + try { owner = JSON.parse(await fs.readFile(path.join(lock,'owner.json'),'utf8')) } catch { /* An incomplete lock remains exclusive. */ } + if (!ancestorToken || owner?.token !== ancestorToken) throw new Error(`Store is locked by unfinished parent operation: ${lock}`) + } catch (error) { if (error.code !== 'ENOENT') throw error } + } + if (path.dirname(parent) === parent) break + } + } + const token = randomUUID(), held = [] + try { + for (const directory of canonical) { + const lock = path.join(directory, '.simplystore-lock') + try { await fs.mkdir(lock) } catch (error) { + if (error.code === 'EEXIST') throw new Error(`Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.`) + throw error + } + held.push(lock) + await publishFile(path.join(lock, 'owner.json'), JSON.stringify({token, purpose, auditDir, pid: process.pid, host: os.hostname(), started: new Date().toISOString()})) + await syncAncestors(directory) + } + } catch (error) { + // No store writes have begun, so only our own acquired locks can be released. + for (const lock of held.reverse()) { await fs.rm(lock, {recursive: true}); await syncDirectory(path.dirname(lock)) } + throw error + } + return {token, directories: canonical, async release() { + for (const lock of [...held].reverse()) { + const owner = JSON.parse(await fs.readFile(path.join(lock, 'owner.json'), 'utf8')) + if (owner.token !== token) throw new Error(`Ownership changed: ${lock}`) + await fs.unlink(path.join(lock, 'owner.json')) + await fs.rmdir(lock) + await syncDirectory(path.dirname(lock)) + } + held.length = 0 + }} +} diff --git a/src/util.mjs b/src/util.mjs index 311266c..527557c 100644 --- a/src/util.mjs +++ b/src/util.mjs @@ -1,5 +1,5 @@ import JSONTag from '@muze-nl/jsontag' -import fs from 'node:fs/promises' +import {appendRecord} from './storage.mjs' export function deepFreeze(obj) { Object.freeze(obj) @@ -32,15 +32,4 @@ export function joinArgs(args) { * @param {string} data The line to write * @return {void} */ -export async function appendFile(filename, data) { - let handle; - try { - handle = await fs.open(filename, 'a+') - await handle.appendFile(data+"\n") - await handle.datasync() - return true - } finally { - await handle.close() - } -} - +export const appendFile = appendRecord diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs new file mode 100644 index 0000000..4bcab66 --- /dev/null +++ b/test/admin-recovery.mjs @@ -0,0 +1,211 @@ +import {Buffer} from 'node:buffer' +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import path from 'node:path' +import os from 'node:os' +import JSONTag from '@muze-nl/jsontag' +import {makeServerFixture} from './durability-helpers.mjs' +import {planRecovery,applyRecovery,backupStore,restoreBackup,verifyCandidate} from '../src/admin-recovery.mjs' +import {inspectStore} from '../src/store-inspection.mjs' +import {acquireOwnership} from '../src/store-ownership.mjs' + +async function setup(t, entries) { + const store=await makeServerFixture(t) + const outputs=await fs.mkdtemp(path.join(os.tmpdir(),'simplystore-admin-output-')) + t.after(()=>fs.rm(outputs,{recursive:true,force:true})) + await fs.writeFile(store.commandLog,entries.map(([id])=>JSONTag.stringify({id,name:'addPerson',value:{name:id}})).join('\n')+'\n') + await fs.writeFile(store.commandStatus,entries.flatMap(([id,status])=>[{command:id,status:'accepted',code:202},...status==='accepted'?[]:[{command:id,status,code:200}]]).map(s=>JSONTag.stringify(s)).join('\n')+'\n') + return {store,outputs} +} +async function runRecovery(store,outputs,plan) { + return applyRecovery(plan,{to:path.join(outputs,'recovered'),auditDir:path.join(outputs,'audit'),approveRerun:plan.rerun,operator:'test admin',reason:'Verified complete log and command inputs; side effects assessed'}) +} +test('administrator recovers trailing missing/waiting commands in log order on a copy',async t=>{ + const {store,outputs}=await setup(t,[['A','done'],['B','accepted']]) + const before=await fs.readFile(store.commandStatus) + const plan=await planRecovery(store,{quiescent:true}) + assert.equal(plan.actionable,true);assert.deepEqual(plan.rerun,['A','B']) + const result=await runRecovery(store,outputs,plan) + const actual=await inspectStore(result.config) + assert.equal(actual.ready,true) + assert.deepEqual(actual.data.persons.map(p=>p.name),['A','B']) + assert.deepEqual(await fs.readFile(store.commandStatus),before) + assert.ok((await fs.readFile(result.config.commandStatus)).subarray(0,before.length).equals(before)) + assert.deepEqual((await fs.readFile(result.config.commandLog)).toString(),(await fs.readFile(store.commandLog)).toString()) +}) +for (const state of ['accepted','done','unsafe']) test(`later ${state} dataset blocks rerun, including zero-byte no-op`,async t=>{ + const {store}=await setup(t,[['A','done'],['B',state]]) + await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') + const plan=await planRecovery(store,{quiescent:true}) + assert.equal(plan.actionable,false) + assert.ok(plan.blocks.some(s=>s.includes('later accepted datasets'))) +}) +test('new later file makes an approved plan stale before executing a handler',async t=>{ + const {store,outputs}=await setup(t,[['A','done'],['B','accepted']]) + const plan=await planRecovery(store,{quiescent:true}) + await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') + await assert.rejects(runRecovery(store,outputs,plan),/Stale recovery plan/) + await assert.rejects(fs.access(path.join(outputs,'recovered')),/ENOENT/) +}) +test('changed code and missing explicit approval reject apply',async t=>{ + const {store,outputs}=await setup(t,[['A','accepted']]) + const plan=await planRecovery(store,{quiescent:true}) + await assert.rejects(applyRecovery(plan,{to:path.join(outputs,'x'),auditDir:path.join(outputs,'y'),operator:'admin',reason:'test'}),/approval/) + await fs.appendFile(store.commandsFile,'\n// changed\n') + await assert.rejects(runRecovery(store,outputs,plan),/Stale recovery plan/) +}) +test('cooperative ownership excludes competing writer and path aliases',async t=>{ + const {store,outputs}=await setup(t,[]) + const owner=await acquireOwnership([store.dir]) + try { + await fs.symlink(store.dir,path.join(outputs,'alias')) + await assert.rejects(acquireOwnership([path.join(outputs,'alias')]),/locked/) + } finally {await owner.release()} +}) +test('backup and restore validate saved bytes without executing commands',async t=>{ + const {store,outputs}=await setup(t,[['A','done']]) + const result=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) + const backup=path.join(outputs,'backup') + await backupStore(result.config,{to:backup,quiescent:true}) + const restored=await restoreBackup(backup,{to:path.join(outputs,'restored'),auditDir:path.join(outputs,'restore-audit'),source:store,sourceQuiescent:true}) + assert.deepEqual((await inspectStore(restored.config)).data.persons.map(p=>p.name),['A']) + assert.deepEqual(restored.missingFromBackup,[]) + const saved=JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) + await fs.appendFile(saved.config.commandLog,' ') + await assert.rejects(restoreBackup(backup,{to:path.join(outputs,'bad'),auditDir:path.join(outputs,'bad-audit')}),/Backup contents/) +}) + +test('read-only inspection never imports handlers and rejects conflicting done order',async t=>{ + const {store}=await setup(t,[['A','accepted'],['B','accepted']]) + await fs.writeFile(store.commandsFile,"throw new Error('must not be imported')") + const before=await fs.readFile(store.commandStatus) + const plan=await planRecovery(store,{quiescent:true}) + assert.equal(plan.actionable,true);assert.deepEqual(await fs.readFile(store.commandStatus),before) + await fs.appendFile(store.commandStatus,[{command:'B',status:'done'},{command:'A',status:'done'}].map(JSON.stringify).join('\n')+'\n') + assert.ok((await planRecovery(store,{quiescent:true})).blocks.some(s=>s.includes('differs from command-log order'))) +}) +test('historical status order never makes a later log dataset eligible for bypass',async t=>{ + const {store}=await setup(t,[['A','accepted'],['B','accepted'],['C','accepted']]) + await fs.writeFile(store.commandStatus,[{command:'C',status:'accepted'},{command:'B',status:'accepted'},{command:'A',status:'accepted'}].map(JSON.stringify).join('\n')+'\n') + await fs.writeFile(path.join(store.dir,'data.C.jsontag'),'') + const plan=await planRecovery(store,{quiescent:true}) + assert.ok(plan.blocks.some(s=>s.includes('B: later accepted datasets exist: C'))) +}) +test('interrupted recovery preserves completed attempts and blocks an unfinished candidate',async t=>{ + const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted']]) + const witness=path.join(outputs,'effects') + await fs.writeFile(store.commandsFile,`import fs from 'node:fs';export default {addPerson(data,c){fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n');if(c.id==='B')throw new Error('external effect already happened');data.persons.push(c.value)}}`) + const plan=await planRecovery(store,{quiescent:true}) + await assert.rejects(runRecovery(store,outputs,plan),/uncertain or failed/) + const authorization=JSON.parse((await fs.readFile(path.join(outputs,'audit','attempts.jsonl'),'utf8')).split('\n')[0]) + const partial=await inspectStore(authorization.config) + assert.equal(partial.ready,false);assert.deepEqual(partial.committed,['A']) + assert.equal(partial.commands.find(c=>c.id==='B').status,'active') + assert.equal(await fs.readFile(witness,'utf8'),'A\nB\n') + await assert.rejects(acquireOwnership([path.dirname(authorization.config.datafile)]),/locked/) + await assert.rejects(fs.access(path.join(outputs,'audit','complete.json')),/ENOENT/) + const preview=await planRecovery({...authorization.config,commandsFile:store.commandsFile,indexFile:store.indexFile},{quiescent:true}) + assert.deepEqual(preview.rerun,['B'],'completed A is reconstructed, never selected for re-execution') +}) +test('retained digest mismatch stops recovery before replacing integrity or appending done',async t=>{ + const {store,outputs}=await setup(t,[['A','done']]) + const {appendIntegrityRecord}=await import('../src/integrity.mjs') + const integrity=path.join(store.dir,'data.integrity.jsontag') + await appendIntegrityRecord(integrity,store.datafile,await fs.readFile(store.datafile)) + await appendIntegrityRecord(integrity,path.join(store.dir,'data.A.jsontag'),Buffer.from('different historical bytes')) + store.integrityFile=integrity + const plan=await planRecovery(store,{quiescent:true}) + await assert.rejects(runRecovery(store,outputs,plan),/Integrity mismatch/) + const authorization=JSON.parse((await fs.readFile(path.join(outputs,'audit','attempts.jsonl'),'utf8')).split('\n')[0]) + assert.deepEqual(await fs.readFile(authorization.config.integrityFile),await fs.readFile(integrity)) + const statuses=(await fs.readFile(authorization.config.commandStatus,'utf8')).trim().split('\n').map(JSONTag.parse) + assert.equal(statuses.at(-1).status,'active') +}) +test('malformed tails and overlapping paths cannot be accepted as valid stores',async t=>{ + const {store}=await setup(t,[['A','accepted']]) + await fs.appendFile(store.commandLog,'{"id":"unfinished"') + assert.equal((await planRecovery(store,{quiescent:true})).actionable,false) + assert.ok((await inspectStore({...store,commandStatus:store.commandLog})).errors.some(e=>e.includes('paths overlap'))) +}) + +test('approved recovery resumes from completed candidate prefix without repeating external effects',async t=>{ + const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted']]) + const witness=path.join(outputs,'idempotent-effects') + await fs.writeFile(store.commandsFile,`import fs from 'node:fs';export default {addPerson(data,c){ + const previous=fs.existsSync(${JSON.stringify(witness)})?fs.readFileSync(${JSON.stringify(witness)},'utf8'):''; + if(!previous.split('\\n').includes(c.id)){ + fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n'); + if(c.id==='B')throw new Error('connection lost after effect'); + } + data.persons.push(c.value) + }}`) + await assert.rejects(runRecovery(store,outputs,await planRecovery(store,{quiescent:true})),/uncertain or failed/) + const authorization=JSON.parse((await fs.readFile(path.join(outputs,'audit','attempts.jsonl'),'utf8')).split('\n')[0]) + const partial={...authorization.config,commandsFile:store.commandsFile,indexFile:store.indexFile} + const plan=await planRecovery(partial,{quiescent:true}) + assert.deepEqual(plan.rerun,['B']) + const {releaseOfflineLocks}=await import('../src/admin-recovery.mjs') + const release=await releaseOfflineLocks(partial,{operator:'test admin',reason:'Inspected retained recovery audit; idempotent B can retry',confirmedStopped:true}) + await fs.writeFile(path.join(outputs,'unlock-evidence.json'),JSON.stringify(release)) + await release.finish() + const retry=path.join(outputs,'retry');await fs.mkdir(retry) + const result=await runRecovery(partial,retry,plan) + assert.deepEqual((await inspectStore(result.config)).data.persons.map(p=>p.name),['A','B']) + assert.equal(await fs.readFile(witness,'utf8'),'A\nB\n') +}) +test('finishing an interrupted completion report reconstructs data without executing again',async t=>{ + const {store,outputs}=await setup(t,[['A','accepted']]) + const result=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) + const audit=path.join(outputs,'audit') + // Fixture models the boundary after done, before completion report/lock release. + await fs.unlink(path.join(audit,'complete.json')) + await acquireOwnership([path.dirname(result.config.datafile)],{purpose:'recovery',auditDir:audit}) + const {finishRecovery,releaseOfflineLocks}=await import('../src/admin-recovery.mjs') + await assert.rejects(releaseOfflineLocks(result.config,{operator:'admin',reason:'test',confirmedStopped:true}),/requires finish/) + const before=await fs.readFile(result.config.commandStatus) + const finished=await finishRecovery(audit,{operator:'admin',reason:'All authorized commands already committed',confirmedStopped:true}) + assert.deepEqual(await fs.readFile(result.config.commandStatus),before) + assert.deepEqual(finished.committed,['A']) + const owner=await acquireOwnership([path.dirname(result.config.datafile)]);await owner.release() +}) +test('multi-directory integrity paths and file permissions survive backup and restore',async t=>{ + const {store,outputs}=await setup(t,[]) + const logdir=path.join(store.dir,'logs');await fs.mkdir(logdir) + store.commandLog=path.join(logdir,'commands.jsontag');store.commandStatus=path.join(logdir,'status.jsontag') + await fs.writeFile(store.commandLog,'');await fs.writeFile(store.commandStatus,'') + store.integrityFile=path.join(logdir,'integrity.jsontag') + const {appendIntegrityRecord}=await import('../src/integrity.mjs') + await appendIntegrityRecord(store.integrityFile,store.datafile,await fs.readFile(store.datafile)) + await fs.chmod(store.datafile,0o600) + const backup=path.join(outputs,'backup');await backupStore(store,{to:backup,quiescent:true}) + const restored=await restoreBackup(backup,{to:path.join(outputs,'restored'),auditDir:path.join(outputs,'audit')}) + assert.equal((await inspectStore(restored.config)).ready,true) + assert.equal((await fs.stat(restored.config.datafile)).mode&0o777,0o600) + assert.deepEqual(await fs.readFile(restored.config.integrityFile),await fs.readFile(store.integrityFile)) +}) + +test('backup excludes unrelated secrets while retaining declared custom artifacts',async t=>{ + const {store,outputs}=await setup(t,[]) + await fs.writeFile(path.join(store.dir,'.env'),'private application secret') + const required=path.join(store.dir,'derived-custom.txt');await fs.writeFile(required,'required output') + const report=await backupStore({...store,requiredFiles:[required]},{to:path.join(outputs,'backup'),quiescent:true}) + assert.equal(Object.keys(report.files).some(file=>path.basename(file)==='.env'),false) + assert.equal(await fs.readFile(report.config.requiredFiles[0],'utf8'),'required output') +}) + +test('a backup remains restorable after transfer to a different directory',async t=>{ + const {store,outputs}=await setup(t,[]) + const old=path.join(outputs,'old-backup');await backupStore(store,{to:old,quiescent:true}) + const moved=path.join(outputs,'transferred-backup');await fs.rename(old,moved) + const result=await restoreBackup(moved,{to:path.join(outputs,'restored'),auditDir:path.join(outputs,'restore-audit')}) + assert.equal((await inspectStore(result.config)).ready,true) +}) + +test('promotion verification rejects code changed after completed recovery',async t=>{ + const {store,outputs}=await setup(t,[['A','accepted']]) + const result=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) + assert.equal((await verifyCandidate(result)).ready,true) + await fs.appendFile(store.commandsFile,'\n// altered after recovery\n') + await assert.rejects(verifyCandidate(result),/Selected code changed/) +}) diff --git a/test/after-change-handlers.mjs b/test/after-change-handlers.mjs index eaab756..f2dc5c0 100644 --- a/test/after-change-handlers.mjs +++ b/test/after-change-handlers.mjs @@ -1,3 +1,4 @@ +import {waitForExit} from './durability-helpers.mjs' import assert from 'node:assert/strict' import fs from 'node:fs/promises' import path from 'node:path' @@ -118,14 +119,16 @@ test('handler lifecycle: final offset write failure prevents command commit', as const server = startServer(t, fixture, {port}) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { + const response = await postCommand(port, { id: commandId, name: 'addPerson', value: {name: 'Should Not Commit'} - }, 'failed') + }) + assert.equal(response.status,202) + assert.equal((await waitForExit(server.child)).code,1) + assert.match(server.getOutput(),/Storage outcome uncertain/) - assert.match(status.message, /EISDIR|EEXIST|ENOTEMPTY/) - assert.deepEqual(await queryNames(port), ['Initial']) + assert.match(server.getOutput(), /EISDIR|EEXIST|ENOTEMPTY/) assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) }) @@ -139,11 +142,13 @@ test('handler lifecycle: custom finalizer rejection prevents command commit', as const server = startServer(t, fixture, {port}) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { + const response = await postCommand(port, { id: 'custom-finalizer-fails', name: 'addPerson', value: {name: 'Should Not Commit'} - }, 'failed') - assert.match(status.message, /custom finalization failed/) - assert.deepEqual(await queryNames(port), ['Initial']) + }) + assert.equal(response.status,202) + assert.equal((await waitForExit(server.child)).code,1) + assert.match(server.getOutput(),/Storage outcome uncertain/) + assert.match(server.getOutput(), /custom finalization failed/) assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) }) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs index f228aca..00bde87 100644 --- a/test/conversion-offsets.mjs +++ b/test/conversion-offsets.mjs @@ -125,3 +125,16 @@ test('conversion propagates rejection from custom finalization', async t => { await assert.rejects(convert(files, indexFile), /custom finalization failed/) await assert.rejects(fs.access(path.join(files.dir, 'index.offset.json')), /ENOENT/) }) + +test('conversion initializes existing-format logs and optional base integrity without overwriting a store',async t=>{ + const files=await fixture(t) + await run(process.execPath,[converter,files.input,files.output,'--integrity'],{timeout:10000}) + const {loadIntegrityManifest,verifyIntegrity}=await import('../src/integrity.mjs') + const integrity=path.join(files.dir,'data.integrity.jsontag') + assert.equal(verifyIntegrity(await loadIntegrityManifest(integrity),integrity,files.output,await fs.readFile(files.output),{required:true}),true) + assert.equal(await fs.readFile(path.join(files.dir,'command-log.jsontag'),'utf8'),'') + assert.equal(await fs.readFile(path.join(files.dir,'command-status.jsontag'),'utf8'),'') + const before=await fs.readFile(files.output) + await assert.rejects(convert(files),/existing store is not overwritten/) + assert.deepEqual(await fs.readFile(files.output),before) +}) diff --git a/test/durability-crash.mjs b/test/durability-crash.mjs index 3d4be48..4008f55 100644 --- a/test/durability-crash.mjs +++ b/test/durability-crash.mjs @@ -7,6 +7,7 @@ import { appendIntegrityRecord, getDefaultIntegrityFile, loadIntegrityManifest } import { getChangesetPath } from '../src/recovery.mjs' import { assertRuntimeEnvironmentConfiguration, getRuntimeEnvironment } from '../src/runtime-environment.mjs' import { + unlockStoppedFixture, getCommandStatus, getOpenPort, makeServerFixture, @@ -82,110 +83,27 @@ test('fault points are inert outside test environment', async () => { }), false) }) -test('crash after command log but before accepted status is not committed', async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const command = { - id: 'crash-after-log', - name: 'addPerson', - value: {name: 'After Log'} - } - - const first = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: 'after-command-log-before-accepted-status' - }) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, {port}) - await waitForServer(second.child, second.getOutput, port) - - await assertServerStateMatchesOracle(port, fixture, []) - assert.deepEqual(await readCommandStatusRecords(fixture), []) - assert.equal((await readCommandLogRecords(fixture)).length, 1) -}) - -test('duplicate command log records do not replay an accepted command twice', async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const command = { - id: 'duplicate-log-command', - name: 'addPerson', - value: {name: 'Once'} - } - - const first = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: 'after-command-log-before-accepted-status' - }) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: 'after-command-accepted-status-before-response' - }) - await waitForServer(second.child, second.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(second.child)).signal, 'SIGKILL') - - const third = startServer(t, fixture, {port}) - await waitForServer(third.child, third.getOutput, port) - - await assertServerStateMatchesOracle(port, fixture, ['Once']) - assert.equal((await readCommandLogRecords(fixture)).length, 2) - assert.equal((await readCommandStatusRecords(fixture)).filter(record => record.status === 'done').length, 1) -}) - -test('accepted command crash boundaries replay to one committed state', async t => { - const replayFaultPoints = [ - 'after-command-accepted-status-before-response', - 'after-active-status-before-command-worker', - 'before-command-changeset-write', - 'after-command-changeset-write', - 'before-command-done-status' - ] - - for (const faultPointName of replayFaultPoints) { - await t.test(faultPointName, async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const command = { - id: faultPointName, - name: 'addPerson', - value: {name: faultPointName} - } - - const first = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: faultPointName - }) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, {port}) - await waitForServer(second.child, second.getOutput, port) - - const retryStatus = await postCommandStatus(port, { - id: command.id, - name: 'addPerson', - value: {name: `${faultPointName} duplicate`} - }) - assert.equal(retryStatus.command, command.id) - assert.equal(retryStatus.status, 'done') - - await assertServerStateMatchesOracle(port, fixture, [faultPointName]) - assert.equal((await getCommandStatus(port, command.id)).status, 'done') - }) - } +for (const faultPointName of [ + 'after-command-log-before-accepted-status', + 'after-command-accepted-status-before-response', + 'after-active-status-before-command-worker', + 'before-command-changeset-write', + 'after-command-changeset-write', + 'before-command-done-status' +]) test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { + const fixture=await makeServerFixture(t) + const port=await getOpenPort() + const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:faultPointName}) + await waitForServer(first.child,first.getOutput,port) + await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) + assert.equal((await waitForExit(first.child)).signal,'SIGKILL') + const before=await fs.readFile(fixture.commandStatus) + await unlockStoppedFixture(t,fixture) + const second=startServer(t,fixture,{port}) + assert.equal((await waitForExit(second.child)).code,1) + assert.match(second.getOutput(),/Administrative recovery required/) + assert.deepEqual(await fs.readFile(fixture.commandStatus),before) + assert.equal((await readCommandLogRecords(fixture))[0].id,'A') }) test('crash after done status but before query update recovers committed state without replay', async t => { @@ -206,6 +124,7 @@ test('crash after done status but before query update recovers committed state w await postCommandExpectingCrash(port, command) assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') + await unlockStoppedFixture(t,fixture) const statusAfterCrash = await readCommandStatusRecords(fixture) assert.equal(statusAfterCrash.at(-1).status, 'done') @@ -225,55 +144,16 @@ test('crash after done status but before query update recovers committed state w assert.equal((await getCommandStatus(port, command.id)).status, 'done') }) -test('repeated active command crashes are marked unsafe instead of replaying forever', async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const commandId = 'poison-command' - const crashOptions = { - port, - runtimeEnvironment: 'test', - faultPoint: 'before-command-done-status', - maxCommandCrashAttempts: 2 - } - - const first = startServer(t, fixture, crashOptions) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, { - id: commandId, - name: 'addPerson', - value: {name: 'Poison'} - }) - - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, crashOptions) - assert.equal((await waitForExit(second.child)).signal, 'SIGKILL') - - const third = startServer(t, fixture, { - port, - maxCommandCrashAttempts: 2 - }) - await waitForServer(third.child, third.getOutput, port) - - const commandStatus = await getCommandStatus(port, commandId) - assert.equal(commandStatus.status, 'unsafe') - assert.equal(commandStatus.attempt, 2) - - const retryStatus = await postCommandStatus(port, { - id: commandId, - name: 'addPerson', - value: {name: 'Retry'} - }) - assert.equal(retryStatus.command, commandId) - assert.equal(retryStatus.status, 'unsafe') - assert.equal(retryStatus.attempt, 2) - - await assertServerStateMatchesOracle(port, fixture, []) - - const statusFile = await fs.readFile(fixture.commandStatus, 'utf8') - assert.match(statusFile, /"status":"active","attempt":1/) - assert.match(statusFile, /"status":"active","attempt":2/) - assert.match(statusFile, /"status":"unsafe"/) +test('crash ownership cannot be stolen automatically',async t=>{ + const fixture=await makeServerFixture(t),port=await getOpenPort() + const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:'before-command-done-status'}) + await waitForServer(first.child,first.getOutput,port) + await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) + await waitForExit(first.child) + const second=startServer(t,fixture,{port}) + assert.equal((await waitForExit(second.child)).code,1) + assert.match(second.getOutput(),/Store is locked/) + assert.equal((await readCommandStatusRecords(fixture)).filter(s=>s.status==='active').length,1) }) test('hanging command times out unsafe and later accepted command commits', async t => { diff --git a/test/durability-helpers.mjs b/test/durability-helpers.mjs index fdc92f6..ec493a5 100644 --- a/test/durability-helpers.mjs +++ b/test/durability-helpers.mjs @@ -35,6 +35,8 @@ export async function makeServerFixture(t, options = {}) { const commandStatus = path.join(dir, 'command-status.jsontag') const runner = path.join(dir, 'run-server.mjs') + await fs.writeFile(commandLog, '') + await fs.writeFile(commandStatus, '') await fs.writeFile(datafile, serialize(JSONTag.parse(options.initialData || '{"persons":[]}'))) await fs.writeFile(commandsFile, options.commandsSource || `export default { addPerson: (dataspace, command) => { @@ -100,17 +102,22 @@ export async function stopServer(child) { } export async function waitForServer(child, getOutput, port) { + let finished = false await new Promise((resolve, reject) => { const timeout = setTimeout(() => { + finished = true child.off('exit', onExit) reject(new Error(`Server did not start on port ${port}:\n${getOutput()}`)) }, 5000) const onExit = (code, signal) => { clearTimeout(timeout) + finished = true reject(new Error(`Server exited before startup (${code || signal}):\n${getOutput()}`)) } const checkReady = () => { + if (finished) return if (getOutput().includes(`SimplyStore listening on port ${port}`)) { + finished = true clearTimeout(timeout) child.off('exit', onExit) resolve() @@ -230,3 +237,14 @@ export async function reconstructCommittedPersonNames(fixture) { const data = await reconstructCommittedDataset(fixture) return data.persons.map(person => person.name) } + +// Explicit administrator action in disposable fixtures whose writer has exited. +export async function unlockStoppedFixture(t, fixture) { + const {releaseOfflineLocks} = await import('../src/admin-recovery.mjs') + const {publishFile} = await import('../src/storage.mjs') + const audit = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-unlock-audit-')) + t.after(() => fs.rm(audit, {recursive:true,force:true})) + const release = await releaseOfflineLocks(fixture, {operator:'test administrator',reason:'Fixture writer has exited; inspect before startup',confirmedStopped:true}) + await publishFile(path.join(audit,'release.json'),JSON.stringify(release)) + await release.finish() +} diff --git a/test/durability-latency.mjs b/test/durability-latency.mjs new file mode 100644 index 0000000..8504aca --- /dev/null +++ b/test/durability-latency.mjs @@ -0,0 +1,22 @@ +import process from 'node:process' +import test from 'node:test' +import assert from 'node:assert/strict' +import {performance} from 'node:perf_hooks' +import {execFileSync} from 'node:child_process' +import {makeServerFixture,getOpenPort,startServer,waitForServer,postCommand,getCommandStatus} from './durability-helpers.mjs' + +test('measure local startup, acceptance, and completion latency',async t=>{ + const store=await makeServerFixture(t),port=await getOpenPort() + const boot=performance.now(),server=startServer(t,store,{port}) + await waitForServer(server.child,server.getOutput,port) + const startupMs=performance.now()-boot,accepted=[],completed=[] + for(let i=0;i<30;i++){ + const start=performance.now(),id=`latency-${i}` + assert.equal((await postCommand(port,{id,name:'addPerson',value:{name:id}})).status,202) + accepted.push(performance.now()-start) + for(;;){const status=await getCommandStatus(port,id);if(status.status==='done')break;assert.equal(['accepted','active'].includes(status.status),true);await new Promise(r=>setTimeout(r,2))} + completed.push(performance.now()-start) + } + const stats=values=>{const v=[...values].sort((a,b)=>a-b);return {p50Ms:v[Math.floor(v.length*.5)],p95Ms:v[Math.floor(v.length*.95)],maxMs:v.at(-1)}} + t.diagnostic(JSON.stringify({samples:30,startupMs,acceptance:stats(accepted),completion:stats(completed),filesystem:execFileSync('stat',['-f','-c','%T',store.dir],{encoding:'utf8'}).trim(),node:process.version,notes:'Single localhost client; completion includes a fresh worker and polling; no production performance claim'})) +}) diff --git a/test/durability-startup.mjs b/test/durability-startup.mjs index c8d3d92..a9b91e3 100644 --- a/test/durability-startup.mjs +++ b/test/durability-startup.mjs @@ -11,16 +11,13 @@ import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import { appendIntegrityRecord, getDefaultIntegrityFile } from '../src/integrity.mjs' import { - activeCommandStatus, getCommittedCommandIds, getChangesetPath, assertOdJsonTagFraming, loadCommandLog, loadCommandStatus, - pendingCommandStatus, recoverActiveCommands, RecoveryIntegrityError, - unsafeCommandStatus } from '../src/recovery.mjs' const rootDir = path.dirname(path.dirname(fileURLToPath(import.meta.url))) @@ -252,52 +249,19 @@ test('durable command log replays only accepted commands', async t => { ...taskDefaults, id: 'accepted-command', command: JSONTag.stringify({id: 'accepted-command', name: 'addPerson', value: {name: 'Accepted'}}), - request: null }) assert.equal(JSONTag.parse(commands[0].command).name, 'addPerson') }) -test('active command below crash threshold is accepted for replay on startup', async t => { - const fixture = await makeFixture(t) - const statusFile = path.join(fixture.dir, 'command-status.jsontag') - const commandLog = path.join(fixture.dir, 'command-log.jsontag') - - await writeJsonTagLines(statusFile, [ - {command: 'crashed-command', code: 102, status: activeCommandStatus, attempt: 1} - ]) - await writeJsonTagLines(commandLog, [ - {id: 'crashed-command', name: 'addPerson', value: {name: 'Retry'}} - ]) - - const status = loadCommandStatus(statusFile) - await recoverActiveCommands(status, statusFile, {maxCrashAttempts: 2}) - const commands = loadCommandLog(status, commandLog) - - assert.equal(status.get('crashed-command').status, pendingCommandStatus) - assert.equal(commands.length, 1) - assert.equal(commands[0].id, 'crashed-command') - assert.match(await fs.readFile(statusFile, 'utf8'), /"status":"accepted"/) -}) - -test('active command at crash threshold is marked unsafe and not replayed', async t => { - const fixture = await makeFixture(t) - const statusFile = path.join(fixture.dir, 'command-status.jsontag') - const commandLog = path.join(fixture.dir, 'command-log.jsontag') - - await writeJsonTagLines(statusFile, [ - {command: 'poison-command', code: 102, status: activeCommandStatus, attempt: 2} - ]) - await writeJsonTagLines(commandLog, [ - {id: 'poison-command', name: 'addPerson', value: {name: 'Unsafe'}} - ]) - - const status = loadCommandStatus(statusFile) - await recoverActiveCommands(status, statusFile, {maxCrashAttempts: 2}) - const commands = loadCommandLog(status, commandLog) - - assert.equal(status.get('poison-command').status, unsafeCommandStatus) - assert.equal(commands.length, 0) - assert.match(await fs.readFile(statusFile, 'utf8'), /"status":"unsafe"/) +for (const attempt of [1, 2]) test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { + const fixture = await makeFixture(t) + const file = path.join(fixture.dir, 'status.jsontag') + await writeJsonTagLines(file, [{command:'A',status:'active',attempt}]) + const before = await fs.readFile(file) + const status = loadCommandStatus(file) + await assert.rejects(recoverActiveCommands(status,file,{maxCrashAttempts:2}), /administrator assessment/) + assert.equal(status.get('A').status,'active') + assert.deepEqual(await fs.readFile(file),before) }) test('malformed durable status record refuses recovery with explicit integrity error', async t => { diff --git a/test/power-loss-baseline.mjs b/test/power-loss-baseline.mjs index 9960405..d136e66 100644 --- a/test/power-loss-baseline.mjs +++ b/test/power-loss-baseline.mjs @@ -1,331 +1,148 @@ -import assert from 'node:assert/strict' +import {Buffer} from 'node:buffer' +// The original baseline is retained at EVD-20260919-TTZ7C-23's Git version. +// These assertions implement the corrected command-only / authoritative-log contracts. import test from 'node:test' +import assert from 'node:assert/strict' import fs from 'node:fs/promises' import path from 'node:path' -import {Buffer} from 'node:buffer' import JSONTag from '@muze-nl/jsontag' -import {getChangesetPath, loadCommandLog, loadCommandStatus} from '../src/recovery.mjs' -import {getOpenPort, waitForServer, waitForExit, queryPersons, readCommandLogRecords, - readCommandStatusRecords, reconstructCommittedPersonNames} from './durability-helpers.mjs' -import {fixture, rows, submit, completion, eventually, launch, crash, copyImage, - loseUnpublishedName, requirement} from './power-loss-helpers.mjs' - -async function ready(t, store, options = {}) { - const port = await getOpenPort() - const server = await launch(t, store, port, options) - await waitForServer(server.child, server.getOutput, port) - return {...server, port} -} - -async function commit(store, server, id = 'A', value = id) { - const result = await submit(store, server.port, {id, name: 'addPerson', value: {name: value}}) - assert.equal(result.http, 202) - assert.equal((await completion(store, server.port, id)).status, 'done') -} - -async function restartedFailure(t, image) { - const server = await launch(t, image, await getOpenPort()) - const exit = await waitForExit(server.child) - assert.notEqual(exit.code, 0) - assert.doesNotMatch(server.getOutput(), /SimplyStore listening/) - return server.getOutput() +import {fixture,launch,submit,completion,eventually,rows,crash,copyImage,loseUnpublishedName} from './power-loss-helpers.mjs' +import {getOpenPort,waitForServer,waitForExit,queryPersons,readCommandLogRecords,readCommandStatusRecords,unlockStoppedFixture} from './durability-helpers.mjs' +import {inspectStore} from '../src/store-inspection.mjs' + +async function ready(t,store,options={}) { + const port=await getOpenPort(), server=await launch(t,store,port,options) + await waitForServer(server.child,server.getOutput,port) + return {...server,port} } - -async function expectedNames(store) { - // Fixture oracle uses only externally observed completions and original inputs. - const journal = await rows(store.journal) - const issued = new Map(journal.filter(e => e.event === 'issued').map(e => [e.command.id, e.command])) - return journal.filter(e => e.event === 'completion' && e.body.status === 'done') - .map(e => issued.get(e.id).value.name) +const command=id=>({id,name:'addPerson',value:{name:id}}) +async function commit(store,server,id='A') { + assert.equal((await submit(store,server.port,command(id))).http,202) + assert.equal((await completion(store,server.port,id)).status,'done') } -test('PL01 control: trace observes actual file syncs before completed state survives restart', async t => { - const store = await fixture(t) - // Existing log names let this control isolate synchronized contents. - await fs.writeFile(store.commandLog, '') - await fs.writeFile(store.commandStatus, '') - const server = await ready(t, store) - await commit(store, server) - const trace = await rows(store.trace) - assert.ok(trace.some(e => e.op === 'datasync' && e.file === store.commandLog)) - assert.ok(trace.some(e => e.op === 'datasync' && e.file === store.commandStatus)) - const changeset = getChangesetPath(store.datafile, 'A') - const rename = trace.findIndex(e => e.op === 'rename' && e.file === changeset) - assert.ok(rename > 0) - assert.ok(trace.slice(0, rename).some(e => e.op === 'sync' && e.file === trace[rename].from)) - assert.ok(trace[rename].threadId > 0, 'preload also instruments command worker threads') - await crash(server.child) - const image = await copyImage(t, store) - const restarted = await ready(t, image) - assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), await expectedNames(store)) - assert.deepEqual(await reconstructCommittedPersonNames(image), await expectedNames(store)) +test('PL01-05 durable publication survives modeled loss of unsynced names and restart',async t=>{ + const store=await fixture(t),server=await ready(t,store) + await commit(store,server) + const trace=await rows(store.trace) + const changeset=path.join(store.dir,'data.A.jsontag') + const rename=trace.findIndex(e=>e.op==='rename'&&e.file===changeset) + assert.ok(rename>0) + assert.ok(trace.slice(0,rename).some(e=>e.op==='sync'&&e.file===trace[rename].from)) + const dirSync=trace.findIndex((e,i)=>i>rename&&e.op==='sync'&&e.file===store.dir) + const doneSync=trace.findIndex((e,i)=>i>dirSync&&e.op==='datasync'&&e.file===store.commandStatus&&Buffer.from(e.bytes,'base64').toString().includes('"status":"done"')) + assert.ok(dirSync>rename&&doneSync>dirSync) + await crash(server.child) + const image=await copyImage(t,store) + assert.equal(await loseUnpublishedName(store,image,changeset),false) + await unlockStoppedFixture(t,image) + const restarted=await ready(t,image) + assert.deepEqual((await queryPersons(restarted.port)).map(p=>p.name),['A']) }) -test('PL02 new status filename can be lost after externally observed done', async t => { - const store = await fixture(t) - const server = await ready(t, store) - await commit(store, server) - await crash(server.child) - const expected = await expectedNames(store) - assert.deepEqual(expected, ['A']) - const originalStatus = await fs.readFile(store.commandStatus) - const image = await copyImage(t, store) - await loseUnpublishedName(store, image, store.commandStatus) - const restarted = await ready(t, image) - const actual = (await queryPersons(restarted.port)).map(p => p.name) - assert.deepEqual(await fs.readFile(store.commandStatus), originalStatus, 'original evidence is preserved') - await requirement(t, 'P2', actual, expected, 'completed command must not silently disappear with its status filename') +test('PL10 an actual short changeset write is completed before acknowledgment',async t=>{ + const store=await fixture(t),server=await ready(t,store,{fault:{op:'write',file:'data.A.jsontag.',action:'short-write'}}) + await commit(store,server) + const trace=await rows(store.trace) + assert.ok(trace.some(e=>e.op==='injected'&&e.action==='short-write')) + assert.ok(trace.filter(e=>e.op==='write'&&path.basename(e.file).startsWith('data.A.jsontag.')).length>=2) + assert.deepEqual((await inspectStore(store)).data.persons.map(p=>p.name),['A']) }) - -test('PL03 new command-log filename can be lost after observed acceptance', async t => { - const store = await fixture(t, {commandsSource: 'export default { hang() { while (true) {} } }'}) - const server = await ready(t, store) - assert.equal((await submit(store, server.port, {id: 'waiting', name: 'hang'})).http, 202) - await eventually(async () => (await rows(store.trace)).some(e => e.file === store.commandStatus && - e.op === 'datasync' && Buffer.from(e.bytes, 'base64').toString().includes('active')), 'active is synced') - await crash(server.child) - const image = await copyImage(t, store) - await loseUnpublishedName(store, image, store.commandLog) - const restarted = await ready(t, image) - assert.deepEqual(await queryPersons(restarted.port), []) - const status = loadCommandStatus(image.commandStatus) - const queued = loadCommandLog(status, image.commandLog) - const witness = (await rows(store.journal)).filter(e => e.event === 'response' && e.http === 202).map(e => e.id) - assert.deepEqual(witness, ['waiting']) - await requirement(t, 'P1', queued.map(c => c.id), witness, 'accepted execution inputs must remain recoverable') +for (const [name,op,file,contains] of [ + ['log sync','datasync','command-log.jsontag','"id":"A"'], + ['acceptance sync','datasync','command-status.jsontag','"status":"accepted"'], + ['log close','close:after','command-log.jsontag','"id":"A"'], + ['changeset sync','sync','data.A.jsontag.'], + ['changeset rename','rename','data.A.jsontag'], + ['done sync','datasync','command-status.jsontag','"status":"done"'] +]) test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`,async t=>{ + const store=await fixture(t),server=await ready(t,store,{fault:{op,file,contains,action:'error'}}) + try {await submit(store,server.port,command('A'))} catch(error){assert.match(error.message,/fetch failed|terminated|socket/)} + assert.equal((await waitForExit(server.child)).code,1) + assert.match(server.getOutput(),/Storage outcome uncertain/) + assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) + const status=await readCommandStatusRecords(store) + // Failed sync can leave a complete but uncertain done record; never append failed after it. + const done=status.findIndex(s=>s.status==='done') + if(done>=0)assert.equal(status.slice(done+1).some(s=>s.status==='failed'),false) + else assert.equal(status.some(s=>s.status==='done'),false) }) - -test('PL04 unpersisted changeset rename causes explicit failure after observed done', async t => { - const store = await fixture(t) - const server = await ready(t, store) - await commit(store, server) - await crash(server.child) - const image = await copyImage(t, store) - const lost = await loseUnpublishedName(store, image, getChangesetPath(store.datafile, 'A')) - assert.equal(lost, true, 'baseline changeset publication lacks a directory barrier') - assert.match(await restartedFailure(t, image), /Missing changeset|missing changeset/) - assert.deepEqual(await expectedNames(store), ['A']) - await requirement(t, 'P2', lost, false, 'completed changeset filename must survive the supported cut') +test('PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip',async t=>{ + const store=await fixture(t,{commandsSource:`export default {addPerson(data,command,request,meta){ + if(request!==undefined)throw new Error('unlogged input leaked') + if(!meta.index.id)throw new Error('index unavailable') + data.persons.push(command.value) + }}`}) + const worker=path.join(store.dir,'custom-worker.mjs') + await fs.writeFile(worker,`import {parentPort} from 'node:worker_threads' +import runCommand,{initialize} from ${JSON.stringify(new URL('../src/command-worker-module.mjs',import.meta.url).href)} +parentPort.on('message',async task=>{if('request' in task)throw new Error('request leaked to worker');await initialize(task);parentPort.postMessage(await runCommand(task.command,{query:{name:'wrong'}}))})`) + const server=await ready(t,store,{server:{commandWorker:worker}}) + const input={id:'A',name:'addPerson',value:{name:'cafΓ© ζΌ’ε­— πŸ˜€',date:new Date('2026-09-19T00:00:00Z')}} + assert.equal((await submit(store,server.port,input,'?name=wrong')).http,202) + assert.equal((await completion(store,server.port,'A')).status,'done') + assert.equal(JSONTag.stringify((await readCommandLogRecords(store))[0]),JSONTag.stringify(input)) + assert.equal((await queryPersons(server.port))[0].name,input.value.name) }) - -test('PL05 default index sidecars are currently rebuildable at startup', async t => { - const store = await fixture(t, {indexSource: `export {default} from ${JSON.stringify(new URL('../src/index.mjs', import.meta.url).href)}`}) - const server = await ready(t, store) - await commit(store, server) - await crash(server.child) - const image = await copyImage(t, store) - const sidecars = (await fs.readdir(image.dir)).filter(name => /^index\.(id|offset).*\.json$/.test(name)) - assert.ok(sidecars.length >= 2) - for (const name of sidecars) await fs.rm(path.join(image.dir, name)) - const restarted = await ready(t, image) - assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), await expectedNames(store)) +for(const duplicate of [false,true])test(`PL16 delayed append cannot be overtaken${duplicate?' by duplicate ID':''}`,async t=>{ + const store=await fixture(t),server=await ready(t,store,{fault:{op:'datasync:after',file:'command-log.jsontag',contains:'"id":"A"',action:'pause'}}) + const a=submit(store,server.port,command('A')) + await eventually(async()=> (await rows(store.trace)).some(e=>e.op==='injected'),'A append gate') + const b=submit(store,server.port,command(duplicate?'A':'B')) + await new Promise(resolve=>setTimeout(resolve,30)) + assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),['A']) + await fs.writeFile(store.release,'') + assert.equal((await a).http,202);assert.ok([200,202].includes((await b).http)) + await completion(store,server.port,duplicate?'A':'B') + const expected=duplicate?['A']:['A','B'] + assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),expected) + assert.deepEqual((await readCommandStatusRecords(store)).filter(s=>s.status==='active').map(s=>s.command),expected) + assert.deepEqual((await queryPersons(server.port)).map(p=>p.name),expected) }) - -test('PL06 adversarial torn status tail fails explicitly and preserves the original', async t => { - const store = await fixture(t) - const server = await ready(t, store) - await commit(store, server) - await crash(server.child) - const original = await fs.readFile(store.commandStatus) - const image = await copyImage(t, store) - // Corruption fixture, NOT an allowed loss of previously synced bytes. - await fs.appendFile(image.commandStatus, '{"command":"next","status":') - assert.match(await restartedFailure(t, image), /Invalid command status record/) - assert.deepEqual(await fs.readFile(store.commandStatus), original) +for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.jsontag'])test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`,async t=>{ + const store=await fixture(t),server=await ready(t,store);await commit(store,server) + await crash(server.child);await unlockStoppedFixture(t,store) + await fs.unlink(path.join(store.dir,artifact)) + const before=await fs.readdir(store.dir) + const restarted=await launch(t,store,await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code,1) + assert.match(restarted.getOutput(),/Administrative recovery required/) + assert.deepEqual((await fs.readdir(store.dir)).sort(),before.sort()) }) - -for (const [label, fault] of [ - ['PL07 changeset fsync error', {op: 'fsync', file: 'data.A.jsontag.', action: 'error'}], - ['PL08 changeset rename error', {op: 'rename', file: 'data.A.jsontag', action: 'error'}], - ['PL09 changeset write ENOSPC', {op: 'write', file: 'data.A.jsontag.', action: 'error', code: 'ENOSPC'}] -]) { - test(`${label} prevents command completion`, async t => { - const store = await fixture(t) - const server = await ready(t, store, {fault}) - assert.equal((await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}})).http, 202) - assert.equal((await completion(store, server.port, 'A')).status, 'failed') - assert.ok((await rows(store.trace)).some(e => e.op === 'injected' && e.operation === fault.op)) - assert.deepEqual(await queryPersons(server.port), []) - assert.ok(!(await readCommandStatusRecords(store)).some(e => e.status === 'done')) - }) -} - -test('PL10 a successful short changeset write can still be acknowledged as done', async t => { - const store = await fixture(t) - const server = await ready(t, store, {fault: {op: 'write', file: 'data.A.jsontag.', action: 'short-write'}}) - await commit(store, server) - assert.ok((await rows(store.trace)).some(e => e.op === 'injected' && e.action === 'short-write')) - assert.deepEqual((await queryPersons(server.port)).map(p => p.name), ['A'], 'live workers receive intact memory bytes') - await crash(server.child) - const image = await copyImage(t, store) - assert.match(await restartedFailure(t, image), /truncated|record length|record payload/i) - await requirement(t, 'P3', (await expectedNames(store)).length, 0, 'a partial changeset write must not be acknowledged as complete') -}) - -test('PL11 done-status datasync failure exits with an uncertain on-disk outcome', async t => { - const store = await fixture(t) - const server = await ready(t, store, {fault: {op: 'datasync', file: 'command-status.jsontag', contains: '"status":"done"', action: 'error'}}) - assert.equal((await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}})).http, 202) - const exit = await waitForExit(server.child) - assert.notEqual(exit.code, 0) - assert.match(server.getOutput(), /Injected EIO/) - const trace = await rows(store.trace) - const injection = trace.findIndex(e => e.op === 'injected') - assert.ok(injection >= 0) - assert.ok(!trace.slice(injection + 1).some(e => e.file === store.commandStatus && e.op === 'datasync')) - assert.deepEqual(await expectedNames(store), []) - // The written done bytes may nevertheless exist. Failure is not proof of rollback. - assert.ok((await readCommandStatusRecords(store)).some(e => e.status === 'done')) -}) - -test('PL12 loss before done sync causes automatic replay of an external effect', async t => { - const store = await fixture(t) - const effects = path.join(store.audit, 'effects.jsonl') - await fs.writeFile(effects, '') - await fs.writeFile(store.commandsFile, `import fs from 'node:fs' -export default { addPerson(data, command) { - fs.appendFileSync(${JSON.stringify(effects)}, JSON.stringify(command.id) + '\\n') - data.persons.push(command.value) -} }`) - // Persisted initial names are part of the fixture's starting state. - await fs.writeFile(store.commandStatus, '') - const server = await ready(t, store, {fault: {op: 'datasync', file: 'command-status.jsontag', contains: '"status":"done"', action: 'pause'}}) - assert.equal((await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}})).http, 202) - await eventually(async () => (await rows(store.trace)).some(e => e.op === 'injected'), 'pause before done sync') - await crash(server.child) - const trace = await rows(store.trace) - const stable = trace.findLast(e => e.file === store.commandStatus && e.op === 'datasync') - assert.ok(stable) - assert.doesNotMatch(Buffer.from(stable.bytes, 'base64').toString(), /"status":"done"/) - const image = await copyImage(t, store) - await fs.writeFile(image.commandStatus, Buffer.from(stable.bytes, 'base64')) - await loseUnpublishedName(store, image, getChangesetPath(store.datafile, 'A')) - assert.deepEqual(await rows(effects), ['A']) - const restarted = await ready(t, image) - assert.equal((await completion(image, restarted.port, 'A')).status, 'done') - assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), ['A']) - await requirement(t, 'P7', await rows(effects), ['A'], 'uncertain execution needs administrator approval before repeating external effects') -}) - -test('PL13 recovery queue ignores later accepted datasets (policy fixture)', async t => { - const store = await fixture(t) - const server = await ready(t, store) - await commit(store, server, 'C') - await crash(server.child) - const image = await copyImage(t, store) - const commands = [{id: 'B', name: 'addPerson', value: {name: 'B'}}, {id: 'C', name: 'addPerson', value: {name: 'C'}}] - await fs.writeFile(image.commandLog, commands.map(c => JSONTag.stringify(c)).join('\n') + '\n') - // Synthetic administrative input: do not claim this is an observed power-cut image. - for (const laterStatus of ['accepted', 'done']) { - const status = new Map([['B', {status: 'accepted'}], ['C', {status: laterStatus}]]) - assert.ok((await fs.stat(getChangesetPath(image.datafile, 'C'))).isFile()) - const queued = loadCommandLog(status, image.commandLog) - await requirement(t, 'P6', queued.some(c => c.id === 'B'), false, - `block B when later C has a dataset and status ${laterStatus}`) - } +test('PL12/17 startup preserves uncertain active work without repeating external effects',async t=>{ + const store=await fixture(t,{commandsSource:`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify('/tmp/placeholder-effect')},'') ;data.persons.push(command.value)}}`}) + const witness=path.join(store.audit,'effects') + await fs.writeFile(store.commandsFile,`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify(witness)},command.id+'\\n');data.persons.push(command.value)}}`) + const server=await ready(t,store,{fault:{op:'write',file:'command-status.jsontag',contains:'"status":"done"',action:'error'}}) + await submit(store,server.port,command('A'));await waitForExit(server.child).catch(error=>{throw new Error(server.getOutput(),{cause:error})}) + const before=await fs.readFile(store.commandStatus) + await unlockStoppedFixture(t,store) + const restarted=await launch(t,store,await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code,1) + assert.deepEqual(await fs.readFile(store.commandStatus),before) + assert.equal(await fs.readFile(witness,'utf8'),'A\n') }) -test('PL14 complete command body survives JSONTag logging and queue reconstruction', async t => { - const store = await fixture(t) - const command = {id: 'tagged', name: 'addPerson', value: {name: 'caf\u00e9 \u6f22\u5b57'}, - when: new Date('2026-09-19T10:00:00.000Z'), nested: {zero: 0, flag: false, values: [null, 'x']}} - const server = await ready(t, store) - assert.equal((await submit(store, server.port, command)).http, 202) - assert.equal((await completion(store, server.port, command.id)).status, 'done') - await crash(server.child) - const queued = loadCommandLog(new Map([[command.id, {status: 'accepted'}]]), store.commandLog) - assert.equal(queued.length, 1) - assert.equal(JSONTag.stringify(JSONTag.parse(queued[0].command)), JSONTag.stringify(command)) +for(const operation of ['sync','close:after'])test(`directory ${operation} failure after rename prevents done`,async t=>{ + const store=await fixture(t) + const server=await ready(t,store,{fault:{op:operation,file:path.basename(store.dir),action:'error',after:{op:'rename',file:'data.A.jsontag'}}}) + await submit(store,server.port,command('A')) + assert.equal((await waitForExit(server.child)).code,1) + assert.ok((await rows(store.trace)).some(e=>e.op==='injected'&&e.file===store.dir)) + assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) }) - -test('PL15 configured worker can use request inputs that are absent from the log', async t => { - const store = await fixture(t, {commandsSource: 'export default { addPerson(data, command, request) { data.persons.push({name: request.query.name}) } }'}) - const worker = path.join(store.dir, 'request-worker.mjs') - await fs.writeFile(worker, `import {parentPort} from 'node:worker_threads' -import runCommand, {initialize} from ${JSON.stringify(new URL('../src/command-worker-module.mjs', import.meta.url).href)} -parentPort.on('message', async task => { - await initialize(task) - parentPort.postMessage(await runCommand(task.command, task.request)) -})`) - const server = await ready(t, store, {server: {commandWorker: worker}}) - assert.equal((await submit(store, server.port, {id: 'request', name: 'addPerson'}, '?name=from-request')).http, 202) - assert.equal((await completion(store, server.port, 'request')).status, 'done') - assert.deepEqual((await queryPersons(server.port)).map(p => p.name), ['from-request']) - await crash(server.child) - const [queued] = loadCommandLog(new Map([['request', {status: 'accepted'}]]), store.commandLog) - assert.equal((await rows(store.journal)).find(e => e.event === 'issued').query, '?name=from-request') - await requirement(t, 'P8', queued.request?.query?.name, 'from-request', 'recovery must retain the request input used by the configured worker') +for(const operation of ['open','write','close:after'])test(`changeset ${operation} error is not masked or acknowledged`,async t=>{ + const store=await fixture(t) + const server=await ready(t,store,{fault:{op:operation,file:'data.A.jsontag.',action:'error',code:operation==='write'?'ENOSPC':'EIO'}}) + await submit(store,server.port,command('A')) + assert.equal((await waitForExit(server.child)).code,1) + assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) + assert.match(server.getOutput(),/Storage outcome uncertain/) }) - -test('PL16 concurrent acceptance can reorder waiting commands on restart', async t => { - const store = await fixture(t, {commandsSource: `export default { - block() { while (true) {} }, - addPerson(data, command) { data.persons.push(command.value) } -}`}) - const server = await ready(t, store, {fault: {op: 'datasync:after', file: 'command-log.jsontag', contains: '"id":"A"', action: 'pause'}, - server: {maxCommandCrashAttempts: 1}}) - assert.equal((await submit(store, server.port, {id: 'block', name: 'block'})).http, 202) - const a = submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}}) - await eventually(async () => (await rows(store.trace)).some(e => e.op === 'injected'), 'A log completion gate') - assert.equal((await submit(store, server.port, {id: 'B', name: 'addPerson', value: {name: 'B'}})).http, 202) - await fs.writeFile(store.release, '') - assert.equal((await a).http, 202) - const accepted = (await rows(store.journal)).filter(e => e.event === 'response' && e.id !== 'block').map(e => e.id) - assert.deepEqual(accepted, ['B', 'A'], 'external observer records acceptance independently') - assert.deepEqual((await readCommandLogRecords(store)).map(c => c.id), ['block', 'A', 'B']) - assert.deepEqual((await readCommandStatusRecords(store)).filter(c => c.status === 'accepted' && c.command !== 'block').map(c => c.command), accepted) - await crash(server.child) - const image = await copyImage(t, store) - const restarted = await ready(t, image, {server: {maxCommandCrashAttempts: 1}}) - assert.equal((await completion(image, restarted.port, 'B')).status, 'done') - assert.equal((await completion(image, restarted.port, 'A')).status, 'done') - const actual = (await queryPersons(restarted.port)).map(p => p.name) - await requirement(t, 'P4/P6', actual, accepted, 'restart must preserve established waiting-queue order') -}) - -test('PL17 failed startup reclassifies active work before discovering missing committed data', async t => { - const store = await fixture(t) - await fs.writeFile(store.commandStatus, [ - {command: 'A', status: 'active', attempt: 1}, - {command: 'C', status: 'done', code: 200} - ].map(c => JSONTag.stringify(c)).join('\n') + '\n') - const before = await fs.readFile(store.commandStatus, 'utf8') - assert.match(await restartedFailure(t, store), /Missing changeset|missing changeset/) - assert.ok((await fs.readFile(store.commandStatus, 'utf8')).startsWith(before), 'original records remain as an append-only prefix') - await requirement(t, 'P7', loadCommandStatus(store.commandStatus).get('A').status, 'active', - 'uncertain active work must not become replayable before administrator review') -}) - -for (const [id, file, op] of [ - ['PL18', 'command-log.jsontag', 'datasync'], - ['PL19', 'command-status.jsontag', 'datasync'], - ['PL20', 'command-log.jsontag', 'close:after'] -]) { - test(`${id} ${file} ${op} failure prevents acceptance`, async t => { - const store = await fixture(t) - const server = await ready(t, store, {fault: {op, file, action: 'error'}}) - const response = await submit(store, server.port, {id: 'A', name: 'addPerson', value: {name: 'A'}}) - assert.equal(response.http, 500) - assert.ok((await rows(store.trace)).some(e => e.op === 'injected' && e.operation === op)) - assert.deepEqual(await queryPersons(server.port), []) - assert.ok(!(await rows(store.journal)).some(e => e.event === 'response' && e.http === 202)) - }) -} - -test('PL21 model control: an explicit directory sync prevents modeled filename loss', async t => { - const store = await fixture(t, {indexSource: `import fs from 'node:fs/promises' -import index from ${JSON.stringify(new URL('../src/index.mjs', import.meta.url).href)} -export default { update() {}, async finalize(bytes, meta, id) { - await index.finalize(bytes, meta, id) - const directory = await fs.open(meta.data, 'r') - try { await directory.sync() } finally { await directory.close() } -} }`}) - const server = await ready(t, store) - await commit(store, server) - await crash(server.child) - const image = await copyImage(t, store) - assert.equal(await loseUnpublishedName(store, image, getChangesetPath(store.datafile, 'A')), false) - assert.equal(await loseUnpublishedName(store, image, store.commandStatus), false) - const restarted = await ready(t, image) - assert.deepEqual((await queryPersons(restarted.port)).map(p => p.name), await expectedNames(store)) +test('short command-log append preserves its entire JSONTag record',async t=>{ + const store=await fixture(t) + const server=await ready(t,store,{fault:{op:'write',file:'command-log.jsontag',contains:'"id":"A"',action:'short-write'}}) + await commit(store,server) + assert.deepEqual((await readCommandLogRecords(store))[0],command('A')) + assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) }) diff --git a/test/power-loss-preload.mjs b/test/power-loss-preload.mjs index 6062119..cf0386b 100644 --- a/test/power-loss-preload.mjs +++ b/test/power-loss-preload.mjs @@ -14,6 +14,7 @@ if (configFile) { 'appendFileSync', 'readFileSync', 'existsSync', 'statSync', 'writeFileSync'] .map(key => [key, fs[key].bind(fs)])) const open = fsp.open.bind(fsp) + const rename = fsp.rename.bind(fsp) const handles = new Map() let injected = false const scoped = file => typeof file === 'string' && @@ -30,6 +31,7 @@ if (configFile) { if (injected || !rule || rule.op !== op || !scoped(file) || !path.basename(file).startsWith(rule.file) || (rule.contains && !content.includes(rule.contains))) return false + if (rule.after && !raw.readFileSync(config.trace,'utf8').split('\n').filter(Boolean).map(JSON.parse).some(event => event.op === rule.after.op && path.basename(event.file) === rule.after.file)) return false injected = true record('injected', file, {operation: op, action: rule.action}) if (rule.action === 'pause') { @@ -46,21 +48,30 @@ if (configFile) { return false } + fsp.rename = async function(from, to) { + const file = path.resolve(String(to)) + await fault('rename', file) + await rename(from, to) + record('rename', file, {from:path.resolve(String(from))}) + } fsp.open = async function(file, ...args) { const absolute = path.resolve(String(file)) const existed = raw.existsSync(absolute) + await fault('open', absolute) const handle = await open(file, ...args) if (!scoped(absolute)) return handle record('open', absolute, {existed, flags: args[0]}) let content = '' - for (const method of ['appendFile', 'datasync', 'sync', 'close']) { + for (const method of ['appendFile', 'write', 'datasync', 'sync', 'close']) { const original = handle[method].bind(handle) handle[method] = async (...params) => { if (method === 'appendFile') content = String(params[0]) - await fault(method, absolute, content) + if (method === 'write') content = String(params[0].subarray(params[1], params[1]+params[2])) + const short = await fault(method, absolute, content) + if (short && method === 'write') params[2] = Math.max(1, Math.floor(params[2]/2)) const result = await original(...params) record(method, absolute, method === 'datasync' || method === 'sync' - ? snapshot(absolute) : {}) + ? snapshot(absolute) : method === 'write' ? {bytesWritten:result.bytesWritten} : {}) await fault(method + ':after', absolute, content) return result } diff --git a/test/storage.mjs b/test/storage.mjs new file mode 100644 index 0000000..9a5e8b9 --- /dev/null +++ b/test/storage.mjs @@ -0,0 +1,34 @@ +import {Buffer} from 'node:buffer' +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import {writeAll,publishFile,publishFileSync,appendRecord,serialWriter} from '../src/storage.mjs' + +test('complete-write loop preserves bytes through repeated short writes and rejects zero progress', async()=>{ + const bytes = Buffer.from('cafΓ© ζΌ’ε­— πŸ˜€'), written=[] + await writeAll({async write(buffer,offset,length){const n=Math.min(3,length);written.push(buffer.subarray(offset,offset+n));return {bytesWritten:n}}},bytes) + assert.deepEqual(Buffer.concat(written),bytes) + for (const count of [0,-1,NaN,1000]) await assert.rejects(writeAll({async write(){return {bytesWritten:count}}},bytes), /progress write/) +}) +test('serialized writer preserves order and stops after an uncertain operation',async()=>{ + const run=serialWriter(), seen=[] + let release + const gate=new Promise(resolve=>{release=resolve}) + const first=run(async()=>{seen.push('A');await gate;throw new Error('uncertain')}) + const second=run(async()=>seen.push('B')) + release();await assert.rejects(first,/uncertain/);await assert.rejects(second,/uncertain/) + assert.deepEqual(seen,['A']) +}) +test('durable publication replaces entire contents and concurrent append retains each record once',async t=>{ + const dir=await fs.mkdtemp(path.join(os.tmpdir(),'simplystore-storage-'));t.after(()=>fs.rm(dir,{recursive:true,force:true})) + const file=path.join(dir,'data') + publishFileSync(file,'old') + await publishFile(file,'new cafΓ©');assert.equal(await fs.readFile(file,'utf8'),'new cafΓ©') + await publishFile(file,'');assert.equal((await fs.stat(file)).size,0) + await Promise.all(Array.from({length:40},(_,i)=>appendRecord(file,String(i)))) + assert.equal(await fs.readFile(file,'utf8'),Array.from({length:40},(_,i)=>i+'\n').join('')) + await assert.rejects(publishFile(path.join(dir,'missing','x'),'x'),error=>error.code==='ENOENT' && error.storageFailure) + assert.deepEqual((await fs.readdir(dir)).sort(),['data']) +}) From 13db3bf55926cf3a3a1b2769c422881a875193c8 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 14:40:44 +0200 Subject: [PATCH 22/37] Verify exact suffix recovery and backup loss examples --- .../implementations/IMP-20260919-TTZ7C-26.md | 5 +++ .../implementations/IMP-20260919-TTZ7C-26.ttl | 1 + test/admin-recovery.mjs | 34 +++++++++++++++++++ 3 files changed, 40 insertions(+) diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md index f7f4876..0f479a1 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.md +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -56,3 +56,8 @@ Operational documentation is in `DURABILITY.md`, `docs/recovery.md`, and `docs/power-loss-filesystem.md`. Full rebuild remains contract-only. Physical hardware certification, exhaustive faults, arbitrary effect rollback, deployment, release and integration are outside this implementation. + +The final acceptance examples additionally exercise exactly the verified A / missing +B / waiting C / missing D suffix, disclosed loss from an older backup, and +identical versus conflicting historical duplicate IDs. These add verification +coverage without changing the committed runtime or VM exercise. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl index a0eaa4e..6e88331 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -34,4 +34,5 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ; sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260917-TTZ7C-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-22 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "049c45642fe60146bd511bb87abb4b032f8df36a" ] ; sd:implementationChangeKind sd:MixedChange . diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs index 4bcab66..7396d93 100644 --- a/test/admin-recovery.mjs +++ b/test/admin-recovery.mjs @@ -209,3 +209,37 @@ test('promotion verification rejects code changed after completed recovery',asyn await fs.appendFile(store.commandsFile,'\n// altered after recovery\n') await assert.rejects(verifyCandidate(result),/Selected code changed/) }) + +test('verified A prefix permits only missing B, waiting C and missing D in log order',async t=>{ + const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted'],['C','accepted'],['D','accepted']]) + const complete=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) + const source={...complete.config,commandsFile:store.commandsFile,indexFile:store.indexFile} + for(const id of ['B','C','D'])await fs.unlink(path.join(path.dirname(source.datafile),`data.${id}.jsontag`)) + await fs.appendFile(source.commandStatus,JSONTag.stringify({command:'C',status:'accepted',code:202})+'\n') + const prefix=await fs.readFile(path.join(path.dirname(source.datafile),'data.A.jsontag')) + const plan=await planRecovery(source,{quiescent:true}) + assert.equal(plan.actionable,true);assert.deepEqual(plan.committed,['A']);assert.deepEqual(plan.rerun,['B','C','D']) + const next=path.join(outputs,'next');await fs.mkdir(next) + const result=await runRecovery(source,next,plan) + assert.deepEqual((await inspectStore(result.config)).data.persons.map(p=>p.name),['A','B','C','D']) + assert.deepEqual(await fs.readFile(path.join(path.dirname(result.config.datafile),'data.A.jsontag')),prefix) +}) + +test('restore reports accepted commands absent from an older backup',async t=>{ + const {store,outputs}=await setup(t,[]) + const backup=path.join(outputs,'old-backup');await backupStore(store,{to:backup,quiescent:true}) + await fs.appendFile(store.commandLog,JSONTag.stringify({id:'later',name:'addPerson',value:{name:'later'}})+'\n') + await fs.appendFile(store.commandStatus,JSONTag.stringify({command:'later',status:'accepted',code:202})+'\n') + const result=await restoreBackup(backup,{to:path.join(outputs,'rollback'),auditDir:path.join(outputs,'audit'),source:store,sourceQuiescent:true}) + assert.equal(result.sameBase,true);assert.deepEqual(result.missingFromBackup,['later']) + assert.deepEqual(result.committed,[]) +}) + +test('identical historical duplicate retains first log position; conflicting duplicate blocks',async t=>{ + const {store}=await setup(t,[['A','accepted'],['B','accepted']]) + await fs.appendFile(store.commandLog,JSONTag.stringify({id:'A',name:'addPerson',value:{name:'A'}})+'\n') + const plan=await planRecovery(store,{quiescent:true}) + assert.equal(plan.actionable,true);assert.deepEqual(plan.rerun,['A','B']) + await fs.appendFile(store.commandLog,JSONTag.stringify({id:'A',name:'addPerson',value:{name:'different'}})+'\n') + assert.ok((await planRecovery(store,{quiescent:true})).blocks.some(s=>s.includes('Conflicting command ID A'))) +}) From 14c640088420bba49c96333946f1e370cf0a5c60 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 14:44:13 +0200 Subject: [PATCH 23/37] Record regression and ext4 power-cut recovery evidence --- .../EVD-20260919-TTZ7C-27-results.json | 226 ++++++++++++++++++ .../evidence/EVD-20260919-TTZ7C-27-tests.txt | 138 +++++++++++ .spiral/evidence/EVD-20260919-TTZ7C-27.md | 112 +++++++++ .spiral/evidence/EVD-20260919-TTZ7C-27.ttl | 11 + 4 files changed, 487 insertions(+) create mode 100644 .spiral/evidence/EVD-20260919-TTZ7C-27-results.json create mode 100644 .spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt create mode 100644 .spiral/evidence/EVD-20260919-TTZ7C-27.md create mode 100644 .spiral/evidence/EVD-20260919-TTZ7C-27.ttl diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json new file mode 100644 index 0000000..2ceb1bd --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json @@ -0,0 +1,226 @@ +{ + "implementation": "049c45642fe60146bd511bb87abb4b032f8df36a", + "additionalAcceptanceTests": "13db3bf55926cf3a3a1b2769c422881a875193c8", + "regression": { + "passed": 101, + "failed": 0, + "skipped": 0, + "todo": 0 + }, + "administratorTestsAfterThreeAdditions": { + "passed": 22, + "failed": 0, + "skipped": 0, + "todo": 0 + }, + "vmEnvironment": { + "qemu": "QEMU emulator version 10.2.1 (Debian 1:10.2.1+ds-1ubuntu3.2)\nCopyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers\n", + "node": "v24.7.0\n", + "kernelSha256": "19fd789cd5e6b4adfe18c9c6b92a1ce8c8b2eef6ff88456cef80ee2dca9b3f72", + "repositoryHead": "049c45642fe60146bd511bb87abb4b032f8df36a", + "worktreeDiffSha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "filesystem": "ext4 data=ordered,barrier=1", + "device": "virtio-blk, QEMU cache=writeback, volatile NBD backend honoring FLUSH/FUA", + "acceleration": "TCG, 2 CPUs, 1024 MiB", + "physicalPowerCut": false + }, + "guestEnvelope": { + "kernel": "Linux version 7.0.0-31-generic (buildd@lcy02-amd64-091) (x86_64-linux-gnu-gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, GNU ld (GNU Binutils for Ubuntu) 2.46) #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 04:26:38 UTC 2026\n", + "mounts": "rootfs / rootfs rw,size=178920k,nr_inodes=44730,inode64 0 0\nproc /proc proc rw,relatime 0 0\nsysfs /sys sysfs rw,relatime 0 0\ndevtmpfs /dev devtmpfs rw,relatime,size=178920k,nr_inodes=44730,mode=755,inode64 0 0\n/dev/vda /store ext4 rw,relatime,data=ordered 0 0\n", + "writeCache": "write back\n" + }, + "vmObserver": [ + { + "deviceControl": "flushed A survives; unflushed B lost", + "discardedDirtyPages": 1 + }, + { + "scenario": "done", + "recover": false, + "marker": "CUT_DONE {\"samples\":[{\"id\":\"A\",\"doneMs\":2520.205792},{\"id\":\"L0\",\"acceptMs\":44.68866600000001,\"doneMs\":868.762804},{\"id\":\"L1\",\"acceptMs\":33.01929300000029,\"doneMs\":872.86582},{\"id\":\"L2\",\"acceptMs\":31.05912999999964,\"doneMs\":755.6233089999996},{\"id\":\"L3\",\"acceptMs\":31.991556999999375,\"doneMs\":829.8371459999998}]}", + "flushes": 101, + "writes": 134, + "discardedDirtyPages": 0 + }, + { + "scenario": "done", + "recover": true, + "marker": "VERIFY {\"ready\":true,\"errors\":[],\"commands\":[{\"id\":\"A\",\"status\":\"done\",\"present\":true},{\"id\":\"L0\",\"status\":\"done\",\"present\":true},{\"id\":\"L1\",\"status\":\"done\",\"present\":true},{\"id\":\"L2\",\"status\":\"done\",\"present\":true},{\"id\":\"L3\",\"status\":\"done\",\"present\":true}],\"persons\":[{\"name\":\"A\"},{\"name\":\"L0\"},{\"name\":\"L1\"},{\"name\":\"L2\"},{\"name\":\"L3\"}],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 8, + "discardedDirtyPages": 0 + }, + { + "scenario": "accepted", + "recover": false, + "marker": "CUT_ACCEPTED", + "flushes": 25, + "writes": 30, + "discardedDirtyPages": 0 + }, + { + "scenario": "accepted", + "recover": true, + "marker": "VERIFY {\"ready\":false,\"errors\":[],\"commands\":[{\"id\":\"A\",\"status\":\"accepted\",\"present\":false}],\"persons\":[],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 8, + "discardedDirtyPages": 0 + }, + { + "scenario": "active", + "recover": false, + "marker": "CUT_ACTIVE", + "flushes": 27, + "writes": 33, + "discardedDirtyPages": 0 + }, + { + "scenario": "active", + "recover": true, + "marker": "VERIFY {\"ready\":false,\"errors\":[],\"commands\":[{\"id\":\"A\",\"status\":\"active\",\"present\":false}],\"persons\":[],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 8, + "discardedDirtyPages": 0 + }, + { + "scenario": "unsynced", + "recover": false, + "marker": "CUT_UNSYNCED", + "flushes": 13, + "writes": 17, + "discardedDirtyPages": 0 + }, + { + "scenario": "unsynced", + "recover": true, + "marker": "VERIFY {\"ready\":true,\"errors\":[],\"commands\":[],\"persons\":[],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 7, + "discardedDirtyPages": 0 + } + ], + "vmResults": [ + { + "scenario": "done", + "cut": "CUT_DONE {\"samples\":[{\"id\":\"A\",\"doneMs\":2520.205792},{\"id\":\"L0\",\"acceptMs\":44.68866600000001,\"doneMs\":868.762804},{\"id\":\"L1\",\"acceptMs\":33.01929300000029,\"doneMs\":872.86582},{\"id\":\"L2\",\"acceptMs\":31.05912999999964,\"doneMs\":755.6233089999996},{\"id\":\"L3\",\"acceptMs\":31.991556999999375,\"doneMs\":829.8371459999998}]}", + "recovered": { + "ready": true, + "errors": [], + "commands": [ + { + "id": "A", + "status": "done", + "present": true + }, + { + "id": "L0", + "status": "done", + "present": true + }, + { + "id": "L1", + "status": "done", + "present": true + }, + { + "id": "L2", + "status": "done", + "present": true + }, + { + "id": "L3", + "status": "done", + "present": true + } + ], + "persons": [ + { + "name": "A" + }, + { + "name": "L0" + }, + { + "name": "L1" + }, + { + "name": "L2" + }, + { + "name": "L3" + } + ], + "unsyncedPresent": false + } + }, + { + "scenario": "accepted", + "cut": "CUT_ACCEPTED", + "recovered": { + "ready": false, + "errors": [], + "commands": [ + { + "id": "A", + "status": "accepted", + "present": false + } + ], + "persons": [], + "unsyncedPresent": false + } + }, + { + "scenario": "active", + "cut": "CUT_ACTIVE", + "recovered": { + "ready": false, + "errors": [], + "commands": [ + { + "id": "A", + "status": "active", + "present": false + } + ], + "persons": [], + "unsyncedPresent": false + } + }, + { + "scenario": "unsynced", + "cut": "CUT_UNSYNCED", + "recovered": { + "ready": true, + "errors": [], + "commands": [], + "persons": [], + "unsyncedPresent": false + } + } + ], + "tmpfsLatency": { + "samples": 30, + "startupMs": 127.17863, + "acceptance": { + "p50Ms": 0.9827060000000074, + "p95Ms": 2.0263760000000275, + "maxMs": 35.388232999999985 + }, + "completion": { + "p50Ms": 39.87711200000001, + "p95Ms": 44.079794999999876, + "maxMs": 79.74672899999999 + }, + "filesystem": "tmpfs", + "node": "v24.7.0", + "notes": "Single localhost client; completion includes a fresh worker and polling; no production performance claim" + }, + "cliSmoke": [ + "convert --integrity", + "inspect --quiescent", + "backup --quiescent", + "restore --store --quiescent", + "verify" + ] +} diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt new file mode 100644 index 0000000..a81dc0e --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt @@ -0,0 +1,138 @@ +simplystore-acceptance-tests.log +βœ” atomicity: command mutation followed by failure does not become visible or committed (458.010713ms) +βœ” isolation: mutating data inside a JavaScript query does not affect later queries (320.714073ms) +βœ” consistency: query during an active command sees the previous committed state (1049.932589ms) +βœ” idempotency: duplicate command ID is acknowledged but not applied twice (504.230852ms) +βœ” idempotency: retry during active command reports current status and does not enqueue twice (1013.644938ms) +βœ” administrator recovers trailing missing/waiting commands in log order on a copy (218.649497ms) +βœ” later accepted dataset blocks rerun, including zero-byte no-op (12.996066ms) +βœ” later done dataset blocks rerun, including zero-byte no-op (10.972497ms) +βœ” later unsafe dataset blocks rerun, including zero-byte no-op (17.307829ms) +βœ” new later file makes an approved plan stale before executing a handler (15.721993ms) +βœ” changed code and missing explicit approval reject apply (17.297063ms) +βœ” cooperative ownership excludes competing writer and path aliases (11.828539ms) +βœ” backup and restore validate saved bytes without executing commands (279.629517ms) +βœ” read-only inspection never imports handlers and rejects conflicting done order (19.357133ms) +βœ” historical status order never makes a later log dataset eligible for bypass (17.97534ms) +βœ” interrupted recovery preserves completed attempts and blocks an unfinished candidate (187.880494ms) +βœ” retained digest mismatch stops recovery before replacing integrity or appending done (78.7803ms) +βœ” malformed tails and overlapping paths cannot be accepted as valid stores (7.249875ms) +βœ” approved recovery resumes from completed candidate prefix without repeating external effects (266.76745ms) +βœ” finishing an interrupted completion report reconstructs data without executing again (98.479939ms) +βœ” multi-directory integrity paths and file permissions survive backup and restore (29.938398ms) +βœ” backup excludes unrelated secrets while retaining declared custom artifacts (27.105721ms) +βœ” a backup remains restorable after transfer to a different directory (55.572793ms) +βœ” promotion verification rejects code changed after completed recovery (79.208975ms) +βœ” handler lifecycle: index update failure prevents command commit (432.338152ms) +βœ” handler lifecycle: derived index file can remain after failed command (397.476824ms) +βœ” handler lifecycle: final offset write failure prevents command commit (289.95578ms) +βœ” handler lifecycle: custom finalizer rejection prevents command commit (264.197927ms) +βœ” handler lifecycle: current index update can mutate canonical state before commit (276.577028ms) +βœ” command edit indexes only its changed record using changeset UTF-8 offsets (17.352132ms) +βœ” command addition indexes new entities and preserves interior skip numbers (9.110173ms) +βœ” command offsets include mutations and new records introduced by the index hook (7.558886ms) +βœ” successive command indexes are relative to their own changeset files (8.252224ms) +βœ” no-op commands replace any stale sidecar with an empty offset index (5.626016ms) +βœ” failure to write final offsets rejects the command before success (5.18124ms) +βœ” commands await custom finalization that delegates to the default index (28.165176ms) +βœ” custom finalization can replace the default without an unconditional offset write (4.302294ms) +βœ” custom finalization rejection propagates from the command worker (5.008679ms) +βœ” default conversion offsets match every final record (100.442045ms) +βœ” conversion finalizes offsets after custom indexing changes record sizes and adds records (98.071236ms) +βœ” conversion replaces stale offsets even when the custom hook writes no index (130.345011ms) +βœ” conversion awaits custom finalization with the original receiver and final bytes (99.235133ms) +βœ” conversion propagates rejection from custom finalization (60.646362ms) +βœ” conversion initializes existing-format logs and optional base integrity without overwriting a store (202.842664ms) +βœ” runtime environment defaults to production and ignores production fault points (1.555028ms) +βœ” fault points are inert outside test environment (0.183785ms) +βœ” crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (518.498482ms) +βœ” crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (396.358861ms) +βœ” crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (364.429233ms) +βœ” crash at before-command-changeset-write preserves evidence and never automatically reruns (379.091026ms) +βœ” crash at after-command-changeset-write preserves evidence and never automatically reruns (338.042732ms) +βœ” crash at before-command-done-status preserves evidence and never automatically reruns (357.359279ms) +βœ” crash after done status but before query update recovers committed state without replay (463.445396ms) +βœ” crash ownership cannot be stolen automatically (301.306069ms) +βœ” hanging command times out unsafe and later accepted command commits (322.625473ms) +βœ” hanging load worker fails startup explicitly (193.730659ms) +βœ” integrity-enabled command writes changeset digest before done status (329.91289ms) +βœ” normal restart preserves committed state according to reconstruction oracle (410.676844ms) +βœ” accepted command changeset is not treated as committed state on startup (89.67717ms) +βœ” done command with missing changeset refuses recovery instead of silently using base data (61.171916ms) +βœ” malformed base OD-JSONTag file refuses recovery (79.053377ms) +βœ” malformed committed changeset refuses recovery (92.273027ms) +βœ” truncated committed changeset payload refuses recovery before lazy parsing can hide it (49.439526ms) +βœ” malformed uncommitted changeset is ignored during committed startup reconstruction (54.649112ms) +βœ” durable status file selects only done command changesets for startup (63.786153ms) +βœ” durable command log replays only accepted commands (3.713886ms) +βœ” active command at attempt 1 remains unchanged pending administrator assessment (6.586364ms) +βœ” active command at attempt 2 remains unchanged pending administrator assessment (2.793342ms) +βœ” malformed durable status record refuses recovery with explicit integrity error (4.069897ms) +βœ” structurally invalid durable status record refuses recovery (5.262939ms) +βœ” malformed durable command log record refuses recovery with explicit integrity error (1.916061ms) +βœ” structurally invalid durable command log record refuses recovery (2.074256ms) +βœ” OD-JSONTag framing validation catches truncated lazy records (0.327636ms) +βœ” OD-JSONTag framing validation accepts changeset skip records (0.241231ms) +βœ” integrity manifest detects same-length altered base OD-JSONTag payload (75.395083ms) +βœ” integrity manifest detects same-length altered committed changeset payload (89.357652ms) +βœ” integrity enabled requires manifest entry for base data (78.988138ms) +βœ” PL01-05 durable publication survives modeled loss of unsynced names and restart (834.651879ms) +βœ” PL10 an actual short changeset write is completed before acknowledgment (290.104519ms) +βœ” PL07-09/11/18-20 log sync failure stops mutation without false terminal success (213.085435ms) +βœ” PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (202.657879ms) +βœ” PL07-09/11/18-20 log close failure stops mutation without false terminal success (180.136444ms) +βœ” PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (255.984928ms) +βœ” PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (254.343941ms) +βœ” PL07-09/11/18-20 done sync failure stops mutation without false terminal success (266.18276ms) +βœ” PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (249.577537ms) +βœ” PL16 delayed append cannot be overtaken (317.039566ms) +βœ” PL16 delayed append cannot be overtaken by duplicate ID (271.982054ms) +βœ” PL02-04/06 missing command-log.jsontag fails open rather than recreating history (319.898933ms) +βœ” PL02-04/06 missing command-status.jsontag fails open rather than recreating history (310.826762ms) +βœ” PL02-04/06 missing data.A.jsontag fails open rather than recreating history (316.136257ms) +βœ” PL12/17 startup preserves uncertain active work without repeating external effects (278.472846ms) +βœ” directory sync failure after rename prevents done (189.041135ms) +βœ” directory close:after failure after rename prevents done (186.196454ms) +βœ” changeset open error is not masked or acknowledged (188.168142ms) +βœ” changeset write error is not masked or acknowledged (185.555103ms) +βœ” changeset close:after error is not masked or acknowledged (190.574387ms) +βœ” short command-log append preserves its entire JSONTag record (195.428932ms) +βœ” complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.23133ms) +βœ” serialized writer preserves order and stops after an uncertain operation (0.477868ms) +βœ” durable publication replaces entire contents and concurrent append retains each record once (33.390561ms) +β„Ή tests 101 +β„Ή pass 101 +β„Ή fail 0 +β„Ή skipped 0 +β„Ή todo 0 +β„Ή duration_ms 5928.321784 + +simplystore-acceptance-examples-detail.log +βœ” administrator recovers trailing missing/waiting commands in log order on a copy (121.693187ms) +βœ” later accepted dataset blocks rerun, including zero-byte no-op (3.134899ms) +βœ” later done dataset blocks rerun, including zero-byte no-op (2.581528ms) +βœ” later unsafe dataset blocks rerun, including zero-byte no-op (3.826567ms) +βœ” new later file makes an approved plan stale before executing a handler (4.010908ms) +βœ” changed code and missing explicit approval reject apply (5.358755ms) +βœ” cooperative ownership excludes competing writer and path aliases (1.982249ms) +βœ” backup and restore validate saved bytes without executing commands (75.292536ms) +βœ” read-only inspection never imports handlers and rejects conflicting done order (4.477334ms) +βœ” historical status order never makes a later log dataset eligible for bypass (2.365027ms) +βœ” interrupted recovery preserves completed attempts and blocks an unfinished candidate (90.496786ms) +βœ” retained digest mismatch stops recovery before replacing integrity or appending done (50.347143ms) +βœ” malformed tails and overlapping paths cannot be accepted as valid stores (2.83327ms) +βœ” approved recovery resumes from completed candidate prefix without repeating external effects (150.275693ms) +βœ” finishing an interrupted completion report reconstructs data without executing again (57.460823ms) +βœ” multi-directory integrity paths and file permissions survive backup and restore (19.356728ms) +βœ” backup excludes unrelated secrets while retaining declared custom artifacts (9.276232ms) +βœ” a backup remains restorable after transfer to a different directory (13.518402ms) +βœ” promotion verification rejects code changed after completed recovery (53.110458ms) +βœ” verified A prefix permits only missing B, waiting C and missing D in log order (309.225479ms) +βœ” restore reports accepted commands absent from an older backup (11.018172ms) +βœ” identical historical duplicate retains first log position; conflicting duplicate blocks (6.571817ms) +β„Ή tests 22 +β„Ή pass 22 +β„Ή fail 0 +β„Ή skipped 0 +β„Ή todo 0 +β„Ή duration_ms 1023.688896 diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md new file mode 100644 index 0000000..2779531 --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -0,0 +1,112 @@ +# Evidence: Power-Loss Durability And Administrative Recovery + +Evaluation of IMP-20260919-TTZ7C-26 at +`13db3bf55926cf3a3a1b2769c422881a875193c8`. Runtime and VM harness are unchanged +from `049c45642fe60146bd511bb87abb4b032f8df36a`; the later commit adds three +specific administrator acceptance examples. The VM captured the latter runtime +revision with an empty tracked worktree diff. Human cycle acceptance is pending. + +Machine-readable environment, observations, recovered data and timings are in +[EVD-27-results.json](EVD-20260919-TTZ7C-27-results.json); individual test outcomes +are retained in [EVD-27-tests.txt](EVD-20260919-TTZ7C-27-tests.txt). + +## Automated regression and policy evidence + +```sh +node --test test/durability-startup.mjs test/conversion-offsets.mjs \ + test/command-offsets.mjs test/acid-baseline.mjs test/durability-crash.mjs \ + test/after-change-handlers.mjs test/storage.mjs test/admin-recovery.mjs \ + test/power-loss-baseline.mjs +node --test --test-isolation=none test/admin-recovery.mjs +node --test test/durability-latency.mjs +``` + +The complete relevant suite passed 101/101, zero failures/skips/TODOs. After three +additional acceptance examples, the administrator suite passed 22/22. This covers +104 distinct tests across those runs; 19 administrator tests were rerun. The +latency test passed separately. Tests use disposable stores and bounded localhost +listeners. Conversion/inspect/backup/restore/verify also passed a direct CLI smoke +exercise with an integrity-enabled store. Python harness syntax parsing passed. + +The earlier baseline remains immutable in EVD-23. PL15's request-retention +expectation was explicitly corrected to request removal with complete logged +JSONTag inputs; PL16's old waiting-queue-preservation expectation was corrected to +command-log authority. Other changes replace unsafe automatic-replay assertions +with byte-preserving refusal, explicit assessment and guarded recovery evidence. +A failed sync can leave visible done bytes: tests assert no false public success +and conservative termination, not that failed sync proves rollback. + +| Requirement | Evidence and disposition | +|---|---| +| P1 acceptance | Complete short appends; command/accepted sync and close failures; serialized concurrent A/B/duplicate acceptance; ext4 cut after observed 202 retains accepted A | +| P2 completion | Changeset file-sync/rename/directory-sync trace precedes done; default offset/conversion/integrity regressions; ext4 cut retains all five completed transitions | +| P3 I/O uncertainty | Short/zero/invalid writes; open/write/ENOSPC/file sync/rename/directory sync/close injection; poisoned writer and stopped mutation, preserved uncertainty | +| P4 independent oracle | Client commands/status witnesses outside damaged stores; namespace-loss controls; host serial observer expects A/L0/L1/L2/L3 independently of recovered logs | +| P5 inspection | No user module import or source-status mutation; framing/tails, missing files, conflicting IDs/order, integrity and existing uncertain data diagnosed | +| P6 ordered rerun | Later accepted/done/unsafe zero-byte data blocks; contrary status order cannot bypass guard; verified A prefix selects only missing B, waiting C, missing D and recovers A/B/C/D | +| P7 no bypass | Crash locks retained, alias exclusion, pending startup refusal without rerun, source/code/later-file changes rejected, partial recovery requires new assessment | +| P8 inputs | Configured worker cannot access request; third handler argument undefined and fourth metadata retained; nested/date/Unicode JSONTag body survives; exact original command bytes retained in recovered copy | +| P9 backup | Consistent integrity-enabled multi-directory copy, permissions and portable backup; no command execution on restore; changed backup rejected; older-backup comparison reports missing accepted command | +| P10 evidence/promotion | Original source/log unchanged, external authorization and attempt records; completed prefix never rerun on resume; finish missing report without execution; changed candidate/code fails verification; CLI emits activation preview only | +| P11 filesystem | Four disposable ext4 VM cut/reboot scenarios plus independent volatile-device negative control; exact envelope and timing recorded below | +| P12 rebuild | Separate new-store selected-code/history and cooperative effect-suppression contract retained in DES-25 and administrator guide; no ordinary recovery override or implemented rebuild claim | + +## Filesystem experiment + +Run the command in `docs/power-loss-filesystem.md` with the packaged harness. +This evaluation used QEMU 10.2.1, Linux 7.0.0-31-generic, Node v24.7.0, ext4 +`data=ordered,barrier=1`, virtio-blk with QEMU `cache=writeback`, and a purpose-built +volatile NBD backend honoring FLUSH/FUA. TCG used two CPUs and 1024 MiB RAM; no +KVM or host block-device writes. QEMU/kernel files were extracted under `/tmp`, +without installing host packages. Exact kernel digest and observed mounts/cache +are retained in the JSON evidence. + +The host kills the whole guest on an observed boundary, discards volatile block +writes, then boots the same surviving image for ext4 recovery and engine +inspection. Results: + +- Done: five completed commands A/L0/L1/L2/L3 survive in order; ready=true. +- Accepted: A's complete accepted record survives; no dataset; ready=false. +- Active: A remains active with no dataset; ready=false; inspection executes no handler. +- Unsynced filesystem control: an ordinary unsynced file disappears after the cut. +- Independent device control: flushed A persists, unflushed B is lost (one dirty page discarded). + +At the engine boundaries no dirty device pages remained: completed flushes had +drained them. The separate device control verifies that the backend really loses +unflushed writes; the filesystem control demonstrates guest-cache loss. The +test does not fabricate deletion of acknowledged synced data. Output images, +initramfs and full serial logs were retained locally under +`/tmp/simplystore-vm-acceptance`; compact evidence is committed here, not images. + +## Latency and practical bounds + +On the ext4/TCG guest, the first cold command completed in 2520 ms. Four subsequent +commands accepted in 31.1–44.7 ms and completed in 755.6–872.9 ms. These include +emulation, fresh workers and polling and are too few for throughput or percentile +claims. Flush correctness, not production performance, is this experiment's aim. + +A separate 30-command localhost/tmpfs sample measured startup 127.2 ms; +acceptance p50/p95 0.98/2.03 ms; completion p50/p95 39.88/44.08 ms. Maximums were +35.39/79.75 ms respectively. These are tmpfs timings, not disk measurements or a +before/after performance comparison. A deployment should measure its own storage. + +Evidence is bounded: no physical power removal, random exhaustive campaign, +remote-filesystem promise, lying-device tolerance, or universal custom-hook +certification. The preload instruments the newly used async paths; synchronous +index helpers have functional/conversion coverage, not exhaustive injected faults +at every native call. Interrupted recovery boundaries combine real worker effects +with explicit fixtures; deployment promotion is documented and previewed, not +performed against a service manager. These are test limits, not stronger claims. + +No canonical format change or migration is required. Pending/uncertain historical +work deliberately reduces automatic availability. Surviving files cannot prove +complete old history, original code/dependencies, missing business inputs or +external effects. Missing/corrupt unrecoverable state may require a disclosed +backup rollback. Full rebuild tooling and wider stress/platform testing remain +follow-up work within their separately recorded boundaries. + +ESLint passed the new modules, changed server/worker/recovery/converter paths, +harness and revised tests. Full changed-file lint also exposes two pre-existing +undefined `meta` references in the untouched `load` methods of `index.id.mjs` and +`index.offset.mjs`, confirmed identical on master; this cycle changes their write +paths only. This is not represented as a clean repository-wide lint result. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl new file mode 100644 index 0000000..557c774 --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260919-TTZ7C-27 a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260919-TTZ7C-27" ; + sd:repositoryPath ".spiral/evidence/EVD-20260919-TTZ7C-27.md" ; + sd:status sd:Active ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "13db3bf55926cf3a3a1b2769c422881a875193c8" ] . From eafac5d407e5675699201e666eae2980b1c6691c Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 14:44:54 +0200 Subject: [PATCH 24/37] Prepare power-loss recovery cycle for human acceptance --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 101 ++++++++++++++++-------- .spiral/project-context.md | 4 +- 2 files changed, 68 insertions(+), 37 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index 23bbac1..951be4f 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -6,11 +6,10 @@ Created from authoritative `master` at Branch creation and correspondence were verified with Git. The pre-existing untracked `package.json~` is unrelated and remains untouched. -Status: **Active**. The maintainer committed to this cycle on 2026-09-19: -"Lets make a cycle to do all this work." This accepts the proposed work boundary, -not the eventual runtime result. The first baseline test slice is implemented -and evidenced. The maintainer subsequently authorized the whole implementation -and evaluation cycle; only human acceptance may close it. +Status: **Active β€” implementation and evaluation complete; awaiting human acceptance**. +The maintainer authorized the whole cycle with the corrected no-format-change +design. Engineering completion does not mark the cycle Accepted or authorize +integration. See Evaluate for the evidence and practical limits. ## Analyze @@ -20,7 +19,7 @@ Make command acceptance and completion durable across the supported power-loss model, and provide an administrator with a tested, evidence-preserving recovery route when committed storage cannot be reconstructed automatically. -### Current effective behavior and gap +### Baseline effective behavior and gap (at cycle opening) The project has crash/retry, malformed-storage, handler, timeout, integrity, and persisted-offset evidence. The latest offset cycle is accepted and merged. @@ -118,7 +117,7 @@ human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. | Rerun unsafe or prohibited | Demonstrate backup restoration, disclose verified coverage, loss, and uncertainty | | Interrupted administrative recovery | Original evidence remains intact; subsequent inspection can explain the partial recovery attempt | -All P1-P12 criteria remain open until evidenced; P12 is documentation of a +P1-P12 were open until evidenced; the evaluation below supplies that evidence. P12 is documentation of a follow-up contract, not acceptance of an implemented rebuild feature. Existing startup/offset, handler, ACID, and crash suites must remain green with any intentional retry-policy changes reflected by justified new tests. @@ -189,34 +188,66 @@ requires special recovery transitions or a migration. Rebuild remains a follow-u contract. Both designs map baseline gaps and remaining acceptance tests to concrete implementation responsibilities. -Step 2 is revised under the design-only instruction. P1-P12 remain design targets -within the existing formats. The material limits are explicit: prospective -acknowledgment guarantees cannot prove lost historical data; old ordering or -inputs may be unresolved; pending commands on restart need administrator -assessment; recovery cannot promise lossless repair without adequate surviving -evidence or a backup. PL15's requested-context-retention assertion is a superseded -interpretation, while the baseline observation remains valid evidence. Its future -replacement must verify removal of request context and command-input equivalence. -The maintainer subsequently confirmed command-log authority after verification -showed that sequential execution can still disagree with log order during -concurrent acceptance. PL16's queue-preservation assertion must likewise be -corrected: an A/B log requires A/B execution and restart, even if the old queue -was B/A. A complete log defines order; historical out-of-order changesets may -still require diagnosis because their preceding state was wrong. Runtime/tests -remain unchanged in this design-only correction. - -When implementation resumes, step 3 begins with byte-complete durable publication/ -append primitives and directory barriers, including short writes, errors, -conversion, and integrity cases. Then ownership, acceptance ordering, the worker -input boundary, and guarded recovery follow. Runtime fixes, new passing invariants, -backup/restore tooling, latency, and real filesystem evidence remain outstanding. -No new test run or runtime guarantee is claimed by this design-only revision. +The earlier design-only phase removed the proposed new format and migration. +The maintainer subsequently authorized completing the whole cycle. The corrected +PL15 test removes request context; corrected PL16 makes live/restarted execution +follow the command log. Historical failures remain recorded in EVD-23. + +### Durable runtime and recovery β€” completed + +IMP-20260919-TTZ7C-26 implements complete durable writes, file/directory barriers, +conversion initialization, ownership, serial acceptance through queue insertion, +command-only workers and strict read-only startup validation. Storage uncertainty +stops mutation; pending/active work is never automatically rerun. + +Administrative inspect/apply/finish/unlock/backup/restore/verify operates on +preserved evidence and new candidates, with explicit effect assessment, exact +suffix approval, later-dataset guards, source/code revalidation and deliberate +promotion. Backup copies preserve relative integrity paths and permissions and +can be transferred. Full rebuild remains the separate documented contract. + +The administrator guide, durability contract and packaged disposable VM harness +are complete. EVD-20260919-TTZ7C-27 records test outcomes, named filesystem/cache +conditions, negative controls, recovered state and measured latency. ## Evaluate -Pending. Cycle creation is complete; the engineering cycle is Active. -Before review, verify the acceptance matrix, runtime evidence, administrator -procedure, commit latency, and remaining limits. Validate staged provenance before -each semantic commit and range/tree provenance before integration. Only human -acceptance can mark the cycle Accepted; then validate the exact prospective -integration against current `master`. Do not merge this Active cycle outward. +Engineering result: **ready for human cycle acceptance within the stated envelope**. +All P1-P12 rows have bounded evidence in EVD-20260919-TTZ7C-27 +at `14c640088420bba49c96333946f1e370cf0a5c60`. The relevant suite +passed 101/101; the expanded administrator suite passed 22/22, adding three exact +acceptance examples (104 distinct tests across those runs), with no skips/TODOs. +The direct CLI conversion/inspection/backup/restore/verification procedure passed. +Sixteen provenance-validator regression tests passed. + +| Criteria | Result | +|---|---| +| P1-P4 | Durable complete records/publications, fault propagation, independent observed acknowledgments, authoritative acceptance/execution order | +| P5-P8 | Read-only diagnosis, approved ordered suffix, no automatic replay, exact command bytes and removal of request context | +| P9-P10 | Consistent portable backup/restore, known-loss reporting, preserved originals/audit, resumable assessment and verified promotion preview | +| P11 | QEMU/ext4 cut/reboot after accepted, active and five done commands; filesystem/device negative controls; exact environment and latency retained | +| P12 | Separate cooperative rebuild contract documented; rebuild tooling explicitly remains a following cycle | + +The VM used Linux 7.0.0-31-generic, QEMU 10.2.1, ext4 ordered data/barriers, +virtio-blk/writeback and a volatile NBD backend honoring FLUSH/FUA. Completed +A/L0/L1/L2/L3 survived in order; accepted/active A remained pending/uncertain. +Unsynced filesystem and device writes were lost in negative controls. This is +real guest-filesystem recovery over a modeled block cache, not physical hardware +certification or exhaustive fault enumeration. Timings and test limits are in +EVD-27; emulated ext4 and tmpfs numbers are not production performance promises. + +Compatibility: no canonical format change or migration. Commands no longer get +HTTP request inputs. Existing pending work requires administrator assessment, +and older/noncooperating writers must not share a store. Historical completeness, +original code/undeclared dependencies and external effects require independent +evidence; irrecoverable stores may need an explicitly disclosed backup rollback. + +The unchanged legacy index `load` methods have two pre-existing lint errors; +new/changed behavior checks pass. They are outside this cycle's write-path work. +Randomized/soak testing, additional filesystems/hardware and full changed-code +rebuild remain follow-up work. There is no release or production deployment. + +Final staged, range and tree provenance validation must pass before handing over +for acceptance. Keep this cycle Active until the maintainer accepts it; then +record acceptance in a new commit and validate the exact prospective integration +against current master. Do not merge this Active branch outward. diff --git a/.spiral/project-context.md b/.spiral/project-context.md index ad37f92..993bd29 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,11 +18,11 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-Loss Durability And Administrative Recovery`), committed by the maintainer on 2026-09-19 and opened on `spiral/CYC-20260919-TTZ7C-17-power-loss-recovery` from `master` at `44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. The cycle is Active; runtime work and acceptance evidence remain outstanding. Its source, understanding, and acceptance matrix are `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20`. +Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-Loss Durability And Administrative Recovery`), committed by the maintainer on 2026-09-19 and opened on `spiral/CYC-20260919-TTZ7C-17-power-loss-recovery` from `master` at `44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. The cycle is Active; implementation and evaluation are complete and human acceptance is pending. Its source, understanding, and acceptance matrix are `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20`. Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. -Active-cycle progress on 2026-09-19: the first baseline slice is implemented in `IMP-20260919-TTZ7C-22` and verified by `EVD-20260919-TTZ7C-23`. `docs/power-loss-baseline.md` maps 21 scenarios; 10 invariant assertions expose current gaps (explicit TODO in baseline mode, failing in strict mode). Existing suites pass 59/59. Findings include unsynced namespace loss, acknowledged short writes, unsafe replay, missing request inputs for a configured worker, and queue-order changes after restart. The design step is now recorded in `DES-20260919-TTZ7C-24` (durable publication and ordered records) and `DES-20260919-TTZ7C-25` (administrative recovery and consistent restore). No runtime fixes or filesystem power-loss guarantee are claimed. The maintainer subsequently requested a design-only revision retaining current store formats without migration. DES-24/25 now remove the v2 envelope, stored sequence/profile fields, and store descriptor. Commands themselves supply all execution inputs beyond the dataset; HTTP request context is removed from the designed worker contract. Command-log position is the sole authoritative order for live execution, reconstruction, restart, and the later-dataset guard; acceptance serialization must enforce it without stored sequence fields. Contrary status/queue order is a bug, not an alternative authority. PL16's old queue-preservation assertion must be corrected to compare execution against the log. A complete log settles order; historical out-of-order changesets can still require diagnosis because they may encode the wrong preceding state. Pending work on restart requires administrator assessment because existing records do not prove which protocol wrote them. Recovery audit/backup reports remain external tool outputs, not required store metadata. Historical missing inputs/order and unobservable history loss remain explicit limits. PL15's request-retention assertion is superseded by the corrected input contract; its observed baseline behavior remains evidence. This step is design only. Proposed implementation starts with complete-write loops, file/directory barriers, and error handling, then ownership/ordering and recovery; remaining test rows and the rest of the cycle stay open. +Active-cycle progress on 2026-09-19: `IMP-20260919-TTZ7C-26` implements the corrected `DES-20260919-TTZ7C-24/25` following explicit authorization to complete the whole cycle. Existing command/status/integrity and dataset formats remain unchanged, with no migration. Durable complete-write and file/directory barriers, ownership, log-authoritative serial acceptance/execution, command-only workers, strict startup assessment, and source-bound administrator recovery/backup/restore are implemented. `EVD-20260919-TTZ7C-27` records 101 passing regression tests plus 22 expanded administrator tests (three additional cases), direct CLI checks, and a passing four-scenario ext4/QEMU power-cut exercise with device/filesystem negative controls and latency. `DURABILITY.md` and `docs/recovery.md` define the bounded behavior and runnable procedure. Historical baseline gaps remain at `EVD-20260919-TTZ7C-23`; PL15/16 expectations were corrected by the maintainer, not silently weakened. Full rebuild remains a documented separate contract. The cycle is ready for human acceptance but not Accepted or integrated. Broader filesystem/hardware and randomized testing, undeclared custom dependencies and historical input/effect uncertainty remain explicit limits. Two unchanged legacy index `load` methods have pre-existing undefined-meta lint errors outside this write-path cycle. ## Intake State From 80a1633e1791ed33eb644439bdea511578de21dd Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 16:57:05 +0200 Subject: [PATCH 25/37] Record preference for explicit control-flow braces --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 11 +++++++++++ AGENTS.md | 2 ++ 2 files changed, 13 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index b5a95db..428e5ce 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -101,3 +101,14 @@ Exact recovery CLI syntax, durable command ordering, log-format implications, filesystem test configuration, and handling of ambiguous/corrupt evidence need design and executable evidence. No runtime outcome or cycle acceptance is claimed by this source. Creating the cycle is distinct from completing it. + +## Code-style correction during acceptance review β€” 2026-09-19 + +The maintainer has not yet accepted the cycle. After reviewing the net server +code growth, they requested: + +> That is a good number. One thing: I prefer to always use block { and }, even if there is only one line in an if or for statement. + +Use braces consistently in the cycle's JavaScript control flow and retain this +preference in repository instructions. This is a formatting correction, not a +change to the durability/recovery design or cycle acceptance. diff --git a/AGENTS.md b/AGENTS.md index 3665a8e..ebe6d28 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -58,6 +58,8 @@ Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifa ## Working Defaults +- Always use braces for control-flow bodies, including single-statement `if`, `else`, and loop bodies. + - Treat `master` as the authoritative branch unless the human states otherwise. - For ordinary repository-changing Spiral cycle work, create one dedicated branch from the authoritative branch, normally `spiral/CYC-###-short-goal`. - For new Spiral artifacts after CYC-018, use distributed-safe IDs allocated by `node .spiral-core/bin/spiral.mjs allocate ` instead of scanning for the next legacy number. Existing `SRC-001` / `CYC-017` style IDs remain valid historical artifacts and must not be renamed. From 941a96ee4158034e611018a03839d2b1e5c0c861 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 16:58:53 +0200 Subject: [PATCH 26/37] Use explicit control-flow blocks throughout cycle code --- .spiral/evidence/EVD-20260919-TTZ7C-27.md | 17 ++++ eslint.config.js | 1 + scripts/convert.mjs | 6 +- scripts/power-loss-vm/guest.mjs | 10 +-- scripts/recover.mjs | 4 +- src/admin-recovery.mjs | 94 +++++++++++------------ src/command-worker-module.mjs | 4 +- src/execute-worker.mjs | 8 +- src/server.mjs | 34 ++++---- src/storage.mjs | 30 ++++---- src/store-inspection.mjs | 40 +++++----- src/store-ownership.mjs | 10 +-- test/admin-recovery.mjs | 8 +- test/durability-crash.mjs | 4 +- test/durability-helpers.mjs | 2 +- test/durability-latency.mjs | 4 +- test/durability-startup.mjs | 4 +- test/power-loss-baseline.mjs | 28 +++---- test/power-loss-helpers.mjs | 4 +- test/power-loss-preload.mjs | 24 +++--- test/storage.mjs | 8 +- 21 files changed, 181 insertions(+), 163 deletions(-) diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md index 2779531..4201202 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.md +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -110,3 +110,20 @@ harness and revised tests. Full changed-file lint also exposes two pre-existing undefined `meta` references in the untouched `load` methods of `index.id.mjs` and `index.offset.mjs`, confirmed identical on master; this cycle changes their write paths only. This is not represented as a clean repository-wide lint result. + +## Braces correction during review + +After the maintainer's style instruction recorded at +`80a1633e1791ed33eb644439bdea511578de21dd`, checked all 26 JavaScript modules +changed in this cycle with ESLint's `curly: ["error", "all"]` rule. Nineteen +files needed braces or accompanying block spacing; the check reports zero brace +violations. Added the rule to ESLint configuration and retained the preference +in AGENTS.md. Unrelated legacy files were not reformatted. + +Parsed all 26 before/after files with Espree and compared their syntax trees, +ignoring source positions, literal raw formatting and single-statement blocks +in control-flow bodies. All trees match after that normalization. Embedded fixture +source strings were unchanged. This establishes that the edit adds control-flow +blocks/spacing without changing the contained expressions, statements or branches. +The runtime regression and VM evidence above remains tied to its exact original +revision; no new power-cut experiment is claimed for this formatting correction. diff --git a/eslint.config.js b/eslint.config.js index 2690113..7db434b 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -3,5 +3,6 @@ import globals from "globals"; import { defineConfig } from "eslint/config"; export default defineConfig([ + { rules: { curly: ["error", "all"] } }, { files: ["**/*.{js,mjs,cjs}"], plugins: { js }, extends: ["js/recommended"], languageOptions: { globals: globals.browser } }, ]); diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 61904f7..fb050e9 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -90,10 +90,10 @@ async function main() { await publishFile(outputFile, strData) // Custom indexes may change record sizes or add records after initial parsing. await finalizeIndex(index, strData, meta) - if (integrity) await appendIntegrityRecord(getDefaultIntegrityFile(outputFile), outputFile, Buffer.from(strData)) - for (const file of logs) await publishFile(file, '') + if (integrity) { await appendIntegrityRecord(getDefaultIntegrityFile(outputFile), outputFile, Buffer.from(strData)) } + for (const file of logs) { await publishFile(file, '') } for (const entry of fs.readdirSync(directory, {withFileTypes:true})) { - if (entry.isFile()) await syncFile(path.join(directory, entry.name)) + if (entry.isFile()) { await syncFile(path.join(directory, entry.name)) } } await ownership.release() console.log('Converted data written to ',outputFile) diff --git a/scripts/power-loss-vm/guest.mjs b/scripts/power-loss-vm/guest.mjs index b673c0a..44bd7b4 100644 --- a/scripts/power-loss-vm/guest.mjs +++ b/scripts/power-loss-vm/guest.mjs @@ -16,7 +16,7 @@ if(exists){ } await publishFile(config.datafile,serialize(JSONTag.parse('{"persons":[]}'))) await publishFile(config.commandLog,'');await publishFile(config.commandStatus,'') -if(mode==='unsynced'){await fs.writeFile('/store/unsynced','not guaranteed durable');console.log('CUT_UNSYNCED');await new Promise(()=>{})} +if(mode==='unsynced'){ await fs.writeFile('/store/unsynced','not guaranteed durable');console.log('CUT_UNSYNCED');await new Promise(()=>{}) } await fs.writeFile('/app/commands.mjs',`export default {addPerson(data,command){${mode==='active'?"console.log('CUT_ACTIVE');while(true){}":mode==='accepted'?"while(true){}":"data.persons.push(command.value)"}}}`) await fs.writeFile('/app/index.mjs','export default {update(){},create(){},load(){return {}}}') await server.run({...config,port:3000,maxWorkers:1,commandTimeout:0,wwwroot:'/app/www'}) @@ -24,16 +24,16 @@ await new Promise(r=>setTimeout(r,100)) const before=performance.now() const response=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id:'A',name:'addPerson',value:{name:'A'}})}) console.log('ACCEPTED '+JSON.stringify({http:response.status,ms:performance.now()-before})) -if(mode==='accepted')console.log('CUT_ACCEPTED') +if(mode==='accepted'){ console.log('CUT_ACCEPTED') } if(mode==='done'){ - for(;;){const status=await fetch('http://127.0.0.1:3000/command/A').then(r=>r.json());if(status.status==='done')break;await new Promise(r=>setTimeout(r,10))} + for(;;){ const status=await fetch('http://127.0.0.1:3000/command/A').then(r=>r.json());if(status.status==='done'){ break; }await new Promise(r=>setTimeout(r,10)) } const samples=[{id:'A',doneMs:performance.now()-before}] for(let i=0;i<4;i++){ const id='L'+i,start=performance.now() const accepted=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id,name:'addPerson',value:{name:id}})}) - if(accepted.status!==202)throw new Error('Acceptance failed') + if(accepted.status!==202){ throw new Error('Acceptance failed') } const acceptMs=performance.now()-start - for(;;){const status=await fetch('http://127.0.0.1:3000/command/'+id).then(r=>r.json());if(status.status==='done')break;await new Promise(r=>setTimeout(r,10))} + for(;;){ const status=await fetch('http://127.0.0.1:3000/command/'+id).then(r=>r.json());if(status.status==='done'){ break; }await new Promise(r=>setTimeout(r,10)) } samples.push({id,acceptMs,doneMs:performance.now()-start}) } console.log('CUT_DONE '+JSON.stringify({samples})) diff --git a/scripts/recover.mjs b/scripts/recover.mjs index 07a447a..f989b36 100644 --- a/scripts/recover.mjs +++ b/scripts/recover.mjs @@ -7,12 +7,12 @@ import {inspectStore} from '../src/store-inspection.mjs' const [action,...args] = process.argv.slice(2) const options = {} for (let i=0;i JSON.parse(await fs.readFile(file,'utf8')) -const required = key => { if (typeof options[key] !== 'string') throw new Error(`--${key} is required`); return options[key] } +const required = key => { if (typeof options[key] !== 'string') { throw new Error(`--${key} is required`); } return options[key] } async function main() { switch (action) { case 'inspect': { diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs index c9c47f4..25bd4fa 100644 --- a/src/admin-recovery.mjs +++ b/src/admin-recovery.mjs @@ -26,18 +26,18 @@ async function codeHashes(code) { async function canonicalNew(file) { const absolute = path.resolve(file) try { await fs.lstat(absolute); throw new Error(`Destination already exists: ${absolute}`) } - catch (error) { if (error.code !== 'ENOENT') throw error } + catch (error) { if (error.code !== 'ENOENT') { throw error } } return path.join(await fs.realpath(path.dirname(absolute)), path.basename(absolute)) } function outside(target, roots) { - for (const root of roots) if (target === root || target.startsWith(root + path.sep) || root.startsWith(target + path.sep)) { + for (const root of roots) { if (target === root || target.startsWith(root + path.sep) || root.startsWith(target + path.sep)) { throw new Error(`Overlapping source/destination: ${target} and ${root}`) - } + } } } async function assertFresh(plan) { const files = await inventory(plan.config) - if (hash(JSON.stringify(files)) !== plan.fingerprint) throw new Error('Stale recovery plan: source inventory changed') - if (JSON.stringify(await codeHashes(plan.code)) !== JSON.stringify(plan.codeHashes)) throw new Error('Stale recovery plan: selected code changed') + if (hash(JSON.stringify(files)) !== plan.fingerprint) { throw new Error('Stale recovery plan: source inventory changed') } + if (JSON.stringify(await codeHashes(plan.code)) !== JSON.stringify(plan.codeHashes)) { throw new Error('Stale recovery plan: selected code changed') } } export async function planRecovery(options, {quiescent = false} = {}) { const report = await inspectStore(options) @@ -45,10 +45,10 @@ export async function planRecovery(options, {quiescent = false} = {}) { const candidates = report.commands.filter(c => ['accepted','active'].includes(c.status) || (c.status === 'done' && !c.present)) const blocks = [...report.errors] for (const command of report.commands) { - if (command.condition === 'corrupt' || (command.present && command.status !== 'done')) blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) + if (command.condition === 'corrupt' || (command.present && command.status !== 'done')) { blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) } } for (const command of candidates) { - if (command.laterDatasets.length) blocks.push(`${command.id}: later accepted datasets exist: ${command.laterDatasets.join(',')}`) + if (command.laterDatasets.length) { blocks.push(`${command.id}: later accepted datasets exist: ${command.laterDatasets.join(',')}`) } } return {kind:'simplystore-recovery-plan', config:report.config, code, codeHashes:await codeHashes(code), files:report.files, fingerprint:report.fingerprint, actionable:quiescent && blocks.length === 0, @@ -65,19 +65,19 @@ export async function writePlan(file, plan) { async function copyStore(config, files, target) { const directories = [...new Set(Object.keys(files).map(file => path.dirname(file)))].sort() let common = directories[0] - while (!directories.every(dir => dir === common || dir.startsWith(common + path.sep) || common === path.parse(common).root)) common = path.dirname(common) + while (!directories.every(dir => dir === common || dir.startsWith(common + path.sep) || common === path.parse(common).root)) { common = path.dirname(common) } const mapFile = file => path.join(target, path.relative(common,file)) - for (const directory of directories) await durableMkdir(path.join(target,path.relative(common,directory))) + for (const directory of directories) { await durableMkdir(path.join(target,path.relative(common,directory))) } for (const [file,digest] of Object.entries(files)) { - if (digest === null) continue + if (digest === null) { continue } const bytes = await fs.readFile(file) - if (hash(bytes) !== digest) throw new Error(`Source changed while copying: ${file}`) + if (hash(bytes) !== digest) { throw new Error(`Source changed while copying: ${file}`) } await publishFile(mapFile(file), bytes, {mode:(await fs.stat(file)).mode & 0o777}) } const result = {...config, datafile:mapFile(config.datafile), commandLog:mapFile(config.commandLog), commandStatus:mapFile(config.commandStatus), integrityFile:mapFile(config.integrityFile), requiredFiles:config.requiredFiles.map(mapFile)} - if (config.schemaFile) result.schemaFile = mapFile(config.schemaFile) + if (config.schemaFile) { result.schemaFile = mapFile(config.schemaFile) } return result } async function prepareWorkspace(target, audit, roots) { @@ -88,16 +88,16 @@ async function prepareWorkspace(target, audit, roots) { } export async function applyRecovery(plan, {to, auditDir, approveRerun = [], operator, reason} = {}) { - if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) throw new Error('Plan is not actionable; inspect a quiescent source and resolve blocking evidence') - if (!operator || !reason) throw new Error('Operator and assessment reason are required') - if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) throw new Error('Explicit approval must match the complete ordered rerun list') + if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) { throw new Error('Plan is not actionable; inspect a quiescent source and resolve blocking evidence') } + if (!operator || !reason) { throw new Error('Operator and assessment reason are required') } + if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) { throw new Error('Explicit approval must match the complete ordered rerun list') } const sourceOwner = await acquireOwnership(mutableDirectories(plan.config)) let candidateOwner try { await assertFresh(plan) // Do not trust editable plan fields to bypass the source predicate. const fresh = await planRecovery({...plan.config,...plan.code}, {quiescent:true}) - if (!fresh.actionable || JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun)) throw new Error('Source no longer satisfies ordered recovery predicate') + if (!fresh.actionable || JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun)) { throw new Error('Source no longer satisfies ordered recovery predicate') } const workspace = await prepareWorkspace(to, auditDir, sourceOwner.directories) const {target,audit} = workspace const rootOwner = await acquireOwnership([target],{purpose:'recovery',auditDir:audit}) @@ -111,14 +111,14 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper const journal = path.join(audit,'attempts.jsonl') await appendRecord(journal, JSON.stringify({event:'authorized',planHash,operator,reason,rerun:plan.rerun,config,attestation:plan.attestation})) const inspection = await inspectStore(config) - if (inspection.errors.length) throw new Error(inspection.errors.join('; ')) + if (inspection.errors.length) { throw new Error(inspection.errors.join('; ')) } const parser = new Parser() let data - for (const bytes of inspection.buffers) data = parser.parse(bytes) + for (const bytes of inspection.buffers) { data = parser.parse(bytes) } // Serialize committed prefix, rebuilding parser indexes without custom hooks. let meta = {index:{id:new Map()}, data:path.dirname(config.datafile), parts:inspection.committed.length} const buffers = [serialize(data,{meta})] - if (config.schemaFile) meta.schema = JSONTag.parse(await fs.readFile(config.schemaFile,'utf8')) + if (config.schemaFile) { meta.schema = JSONTag.parse(await fs.readFile(config.schemaFile,'utf8')) } const expectedManifest = plan.files[plan.config.integrityFile] != null ? await loadIntegrityManifest(plan.config.integrityFile) : null for (const id of plan.rerun) { await assertFresh(plan) @@ -134,7 +134,7 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper await appendRecord(journal, JSON.stringify({event:'stopped',id,result:result && {status:result.status,message:result.message}})) throw new Error(`Recovery attempt ${id} uncertain or failed; inspect before approving another attempt`) } - for (const file of config.requiredFiles) await syncFile(file) + for (const file of config.requiredFiles) { await syncFile(file) } if (expectedManifest && command.status === 'done') { verifyIntegrity(expectedManifest, plan.config.integrityFile, command.file, result.data, {required:true}) } @@ -148,7 +148,7 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper await appendRecord(journal, JSON.stringify({event:'done',id,attempt:active.attempt})) } const complete = await inspectStore(config) - if (!complete.ready) throw new Error('Recovered candidate is not ready: '+JSON.stringify({errors:complete.errors,commands:complete.commands.filter(c=>c.problem)})) + if (!complete.ready) { throw new Error('Recovered candidate is not ready: '+JSON.stringify({errors:complete.errors,commands:complete.commands.filter(c=>c.problem)})) } const report = {kind:'simplystore-recovery-complete',planHash,config,code:plan.code,codeHashes:plan.codeHashes,files:complete.files, fingerprint:complete.fingerprint,committed:complete.committed,operator,reason,warnings:complete.warnings} await publishFile(path.join(audit,'complete.json'), JSON.stringify(report,null,2)) @@ -162,18 +162,18 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper } export async function backupStore(options, {to, quiescent = false} = {}) { - if (!quiescent) throw new Error('Backup requires a stopped/quiescent source') + if (!quiescent) { throw new Error('Backup requires a stopped/quiescent source') } const config = storePaths(options), owner = await acquireOwnership(mutableDirectories(config)) try { const report = await inspectStore(config) - if (!report.ready) throw new Error('Backup source is not a complete validated store') + if (!report.ready) { throw new Error('Backup source is not a complete validated store') } const target = await canonicalNew(to); outside(target, owner.directories) await durableMkdir(target) const backupOwner = await acquireOwnership([target]) const copied = await copyStore(config, report.files, path.join(target,'store')) const check = await inspectStore(copied) - if (!check.ready) throw new Error('Backup reconstruction failed') - if (hash(JSON.stringify(await inventory(config))) !== report.fingerprint) throw new Error('Backup source changed') + if (!check.ready) { throw new Error('Backup reconstruction failed') } + if (hash(JSON.stringify(await inventory(config))) !== report.fingerprint) { throw new Error('Backup source changed') } const manifest = {kind:'simplystore-backup-complete',root:target,config:copied,files:check.files,fingerprint:check.fingerprint, committed:check.committed,sourceFingerprint:report.fingerprint,warnings:report.warnings} await publishFile(path.join(target,'complete.json'),JSON.stringify(manifest,null,2)) @@ -185,29 +185,29 @@ export async function backupStore(options, {to, quiescent = false} = {}) { export async function restoreBackup(backupDirectory, {to, auditDir, source, sourceQuiescent=false} = {}) { const backup = await fs.realpath(backupDirectory) const manifest = JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) - if (manifest.kind !== 'simplystore-backup-complete') throw new Error('Backup is incomplete or unrecognized') + if (manifest.kind !== 'simplystore-backup-complete') { throw new Error('Backup is incomplete or unrecognized') } const oldRoot=manifest.root - if(typeof oldRoot!=='string' || !path.isAbsolute(oldRoot) || manifest.fingerprint!==hash(JSON.stringify(manifest.files)))throw new Error('Invalid backup inventory') + if(typeof oldRoot!=='string' || !path.isAbsolute(oldRoot) || manifest.fingerprint!==hash(JSON.stringify(manifest.files))){ throw new Error('Invalid backup inventory') } const rebase=file=>{ - if(typeof file!=='string' || file!==path.resolve(file) || !file.startsWith(oldRoot+path.sep))throw new Error('Backup manifest points outside backup') + if(typeof file!=='string' || file!==path.resolve(file) || !file.startsWith(oldRoot+path.sep)){ throw new Error('Backup manifest points outside backup') } return path.join(backup,path.relative(oldRoot,file)) } manifest.files=Object.fromEntries(Object.entries(manifest.files).map(([file,digest])=>[rebase(file),digest])) manifest.fingerprint=hash(JSON.stringify(manifest.files)) - for(const key of ['datafile','commandLog','commandStatus','integrityFile','schemaFile'])if(manifest.config[key])manifest.config[key]=rebase(manifest.config[key]) + for(const key of ['datafile','commandLog','commandStatus','integrityFile','schemaFile']){ if(manifest.config[key]){ manifest.config[key]=rebase(manifest.config[key]) } } manifest.config.requiredFiles=manifest.config.requiredFiles.map(rebase) - if (mutableDirectories(manifest.config).some(dir=>dir!==backup && !dir.startsWith(backup+path.sep))) throw new Error('Backup configuration points outside backup') - if (source && !sourceQuiescent) throw new Error('Source comparison requires a stopped/quiescent source') + if (mutableDirectories(manifest.config).some(dir=>dir!==backup && !dir.startsWith(backup+path.sep))) { throw new Error('Backup configuration points outside backup') } + if (source && !sourceQuiescent) { throw new Error('Source comparison requires a stopped/quiescent source') } const owner = await acquireOwnership([...mutableDirectories(manifest.config),...(source ? mutableDirectories(storePaths(source)) : [])]) try { const report = await inspectStore(manifest.config) - if (!report.ready || report.fingerprint !== manifest.fingerprint) throw new Error('Backup contents differ from completed manifest') + if (!report.ready || report.fingerprint !== manifest.fingerprint) { throw new Error('Backup contents differ from completed manifest') } const {target,audit} = await prepareWorkspace(to,auditDir,[backup,...owner.directories]) const rootOwner = await acquireOwnership([target],{purpose:'restore',auditDir:audit}) const config = await copyStore(manifest.config,manifest.files,target) const candidate = await acquireOwnership(mutableDirectories(config).filter(dir=>dir!==target),{ancestorToken:rootOwner.token}) const check = await inspectStore(config) - if (!check.ready) throw new Error('Restored candidate failed validation') + if (!check.ready) { throw new Error('Restored candidate failed validation') } let missingFromBackup = null, sameBase = null if (source) { const retained = await inspectStore(source) @@ -225,28 +225,28 @@ export async function restoreBackup(backupDirectory, {to, auditDir, source, sour // Explicit offline operation: no PID-age or liveness heuristic can authorize it. export async function releaseOfflineLocks(options, {operator, reason, confirmedStopped = false} = {}) { - if (!confirmedStopped || !operator || !reason) throw new Error('Offline release requires confirmed stopped writer, operator, and reason') + if (!confirmedStopped || !operator || !reason) { throw new Error('Offline release requires confirmed stopped writer, operator, and reason') } const config = storePaths(options), report = await inspectStore(config) const removed = [] const lockDirectories = new Set(await Promise.all(mutableDirectories(config).map(dir=>fs.realpath(dir)))) for (const directory of [...lockDirectories]) { for (let parent=path.dirname(directory); path.dirname(parent)!==parent; parent=path.dirname(parent)) { - try { await fs.access(path.join(parent,'.simplystore-lock')); lockDirectories.add(parent) } catch(error) { if(error.code!=='ENOENT')throw error } + try { await fs.access(path.join(parent,'.simplystore-lock')); lockDirectories.add(parent) } catch(error) { if(error.code!=='ENOENT'){ throw error } } } } for (const directory of [...lockDirectories].sort().reverse()) { const lock = path.join(directory,'.simplystore-lock') let entries - try { entries = await fs.readdir(lock) } catch (error) { if (error.code === 'ENOENT') continue; throw error } - if (entries.some(entry => entry !== 'owner.json' && !entry.endsWith('.tmp'))) throw new Error(`Unknown lock contents: ${lock}`) + try { entries = await fs.readdir(lock) } catch (error) { if (error.code === 'ENOENT') { continue; } throw error } + if (entries.some(entry => entry !== 'owner.json' && !entry.endsWith('.tmp'))) { throw new Error(`Unknown lock contents: ${lock}`) } const saved = [] - for (const entry of entries) saved.push({name:entry,bytes:(await fs.readFile(path.join(lock,entry))).toString('base64')}) + for (const entry of entries) { saved.push({name:entry,bytes:(await fs.readFile(path.join(lock,entry))).toString('base64')}) } const ownerRecord=saved.find(item=>item.name==='owner.json') const owner=ownerRecord ? JSON.parse(Buffer.from(ownerRecord.bytes,'base64').toString()) : null if (owner?.purpose==='recovery' && report.ready) { let completed try { completed=JSON.parse(await fs.readFile(path.join(owner.auditDir,'complete.json'),'utf8')) } catch { /* Missing report is not completion evidence. */ } - if(completed?.fingerprint!==report.fingerprint) throw new Error('Completed-looking recovery candidate requires finish with its retained audit before unlocking') + if(completed?.fingerprint!==report.fingerprint) { throw new Error('Completed-looking recovery candidate requires finish with its retained audit before unlocking') } } removed.push({lock,ownerFiles:saved}) } @@ -255,23 +255,23 @@ export async function releaseOfflineLocks(options, {operator, reason, confirmedS operator,reason,removed,async finish() { for (const {lock,ownerFiles} of removed) { for (const entry of ownerFiles) { - if ((await fs.readFile(path.join(lock,entry.name))).toString('base64') !== entry.bytes) throw new Error('Lock changed after preview') + if ((await fs.readFile(path.join(lock,entry.name))).toString('base64') !== entry.bytes) { throw new Error('Lock changed after preview') } } - for (const entry of ownerFiles) await fs.unlink(path.join(lock,entry.name)) + for (const entry of ownerFiles) { await fs.unlink(path.join(lock,entry.name)) } await fs.rmdir(lock); await syncDirectory(path.dirname(lock)) } }} } export async function finishRecovery(auditDirectory, {operator,reason,confirmedStopped=false}={}) { - if(!confirmedStopped || !operator || !reason)throw new Error('Finishing requires confirmed stopped writer and administrator assessment') + if(!confirmedStopped || !operator || !reason){ throw new Error('Finishing requires confirmed stopped writer and administrator assessment') } const audit=await fs.realpath(auditDirectory) const plan=JSON.parse(await fs.readFile(path.join(audit,'plan.json'),'utf8')) const authorization=JSON.parse((await fs.readFile(path.join(audit,'attempts.jsonl'),'utf8')).split('\n')[0]) - if(authorization.planHash!==hash(JSON.stringify(plan)))throw new Error('Audit authorization does not match retained plan') + if(authorization.planHash!==hash(JSON.stringify(plan))){ throw new Error('Audit authorization does not match retained plan') } await assertFresh(plan) const report=await inspectStore(authorization.config) - if(!report.ready || plan.rerun.some(id=>!report.committed.includes(id)))throw new Error('Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun') + if(!report.ready || plan.rerun.some(id=>!report.committed.includes(id))){ throw new Error('Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun') } const complete={kind:'simplystore-recovery-complete',planHash:authorization.planHash,config:report.config, code:plan.code,codeHashes:plan.codeHashes,files:report.files,fingerprint:report.fingerprint,committed:report.committed,operator,reason,warnings:report.warnings} await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) @@ -282,9 +282,9 @@ export async function finishRecovery(auditDirectory, {operator,reason,confirmedS } export async function verifyCandidate(report) { - if (!['simplystore-recovery-complete','simplystore-restore-complete'].includes(report.kind)) throw new Error('Expected completed recovery/restore report') + if (!['simplystore-recovery-complete','simplystore-restore-complete'].includes(report.kind)) { throw new Error('Expected completed recovery/restore report') } const current = await inspectStore(report.config) - if (!current.ready || current.fingerprint !== report.fingerprint) throw new Error('Candidate changed or is incomplete') - if (report.code && JSON.stringify(await codeHashes(report.code)) !== JSON.stringify(report.codeHashes)) throw new Error('Selected code changed after recovery') + if (!current.ready || current.fingerprint !== report.fingerprint) { throw new Error('Candidate changed or is incomplete') } + if (report.code && JSON.stringify(await codeHashes(report.code)) !== JSON.stringify(report.codeHashes)) { throw new Error('Selected code changed after recovery') } return current } diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 1144975..c5d4173 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -96,8 +96,8 @@ export default async function runCommand(commandStr) { let publishing = false try { let task = JSONTag.parse(commandStr, null, metaReadProxy) - if (!task.id) { throw new Error('missing command id')} - if (!task.name) { throw new Error('missing command name parameter')} + if (!task.id) { throw new Error('missing command id') } + if (!task.name) { throw new Error('missing command name parameter') } if (commands[task.name]) { let time = Date.now() await commands[task.name](dataspace, task, undefined, metaProxy) diff --git a/src/execute-worker.mjs b/src/execute-worker.mjs index 6638155..2936ade 100644 --- a/src/execute-worker.mjs +++ b/src/execute-worker.mjs @@ -5,7 +5,7 @@ export function executeWorker(filename, task, timeout = 30000) { const worker = new Worker(filename) let settled = false, timer const finish = async (settle, result) => { - if (settled) return + if (settled) { return } settled = true clearTimeout(timer) try { await worker.terminate() } catch (error) { reject(error); return } @@ -14,10 +14,10 @@ export function executeWorker(filename, task, timeout = 30000) { worker.on('message', result => { void finish(resolve, result) }) worker.on('error', error => { void finish(reject, error) }) worker.on('exit', code => { - if (!settled) void finish(reject, new Error(`Command worker exited without a result (${code})`)) + if (!settled) { void finish(reject, new Error(`Command worker exited without a result (${code})`)) } }) - if (timeout) timer = setTimeout(() => { void finish(resolve, - {status: 'unsafe', code: 504, message: `command worker timed out after ${timeout}ms`}) }, timeout) + if (timeout) { timer = setTimeout(() => { void finish(resolve, + {status: 'unsafe', code: 504, message: `command worker timed out after ${timeout}ms`}) }, timeout) } try { worker.postMessage(task) } catch (error) { void finish(reject, error) } }) } diff --git a/src/server.mjs b/src/server.mjs index 30d199c..358d558 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -85,11 +85,11 @@ async function main(options) { let inspection try { inspection = await inspectStore(config) - if (!inspection.ready) throw new Error('Administrative recovery required: ' + + if (!inspection.ready) { throw new Error('Administrative recovery required: ' + [...inspection.errors, ...inspection.commands.filter(c => c.problem).map(c => c.problem), ...inspection.commands.filter(c => c.status === 'accepted' || c.status === 'active').map(c => `${c.id}: ${c.status}; administrator assessment required`), - ...inspection.commands.filter(c => c.present && c.status !== 'done').map(c => `${c.id}: uncommitted dataset`)].join('; ')) - for (const [file, digest] of Object.entries(inspection.files)) if (digest !== null) await syncFile(file) + ...inspection.commands.filter(c => c.present && c.status !== 'done').map(c => `${c.id}: uncommitted dataset`)].join('; ')) } + for (const [file, digest] of Object.entries(inspection.files)) { if (digest !== null) { await syncFile(file) } } const data = await loadData(inspection.committed) jsontagBuffers = [data.data] meta = data.meta @@ -103,7 +103,7 @@ async function main(options) { let storageFailed = false, closing = false let runner = Promise.resolve() function failStorage(error) { - if (storageFailed) return + if (storageFailed) { return } storageFailed = true console.error('Storage outcome uncertain; stopping mutation. Administrator recovery required:', error) // Retain ownership evidence; a process exit must never imply a rollback. @@ -150,14 +150,14 @@ async function main(options) { }) listener.on('error', error => { failStorage(error) }) async function shutdown() { - if (closing) return + if (closing) { return } closing = true listener.close() try { await acceptSerial(async () => {}) await runner queryWorkerPool.close(); slowQueryWorkerPool.close() - if (!storageFailed) await ownership.release() + if (!storageFailed) { await ownership.release() } process.exit(storageFailed ? 1 : 0) } catch (error) { failStorage(error) } } @@ -303,20 +303,20 @@ async function main(options) { } async function handlePostCommand(req, res) { - if (storageFailed || closing) return sendResponse({code:503, body: JSON.stringify({message:'Store is unavailable'})}, res) + if (storageFailed || closing) { return sendResponse({code:503, body: JSON.stringify({message:'Store is unavailable'})}, res) } try { await acceptSerial(async () => { - if (storageFailed) throw new Error('Store is unavailable') - if (closing) return sendResponse({code:503,body:JSON.stringify({message:'Store is closing'})},res) + if (storageFailed) { throw new Error('Store is unavailable') } + if (closing) { return sendResponse({code:503,body:JSON.stringify({message:'Store is closing'})},res) } const accepted = await checkCommand(req, res) - if (!accepted) return - if (storageFailed) throw new Error('Store failed during acceptance') + if (!accepted) { return } + if (storageFailed) { throw new Error('Store failed during acceptance') } commandQueue.push({id: accepted.id, command: accepted.line}) sendResponse({code:202, body:JSON.stringify(status.get(accepted.id))}, res) }) void drainCommandQueue() } catch (error) { - if (!res.headersSent) sendResponse({code:500, body:JSON.stringify({message:'Storage outcome uncertain'})}, res) + if (!res.headersSent) { sendResponse({code:500, body:JSON.stringify({message:'Storage outcome uncertain'})}, res) } failStorage(error) } } @@ -338,7 +338,7 @@ async function main(options) { } function drainCommandQueue() { - if (commandRunnerActive || storageFailed) return runner + if (commandRunnerActive || storageFailed) { return runner } commandRunnerActive = true runner = (async () => { try { @@ -347,15 +347,15 @@ async function main(options) { console.log('starting command', command.id) const active = nextActiveCommandStatus(command.id, status.get(command.id)) await appendFile(commandStatus, JSONTag.stringify(active)) - if (storageFailed) throw new Error('Store failed before execution') + if (storageFailed) { throw new Error('Store failed before execution') } status.set(command.id, active) await faultPoint('after-active-status-before-command-worker') const result = await executeWorker(commandWorker, {...command, meta, data: jsontagBuffers, commandsFile, indexFile, datafile, integrityFile: integrityEnabled ? integrityFile : null, integrityRequired: integrityEnabled}, commandTimeout) - if (storageFailed) throw new Error('Store failed during execution') - if (result?.storageFailure) throw new Error(result.message || 'Worker persistence failure') + if (storageFailed) { throw new Error('Store failed during execution') } + if (result?.storageFailure) { throw new Error(result.message || 'Worker persistence failure') } if (!result || result.status === 'failed' || result.status === 'unsafe' || result.code >= 300) { const terminal = {command: command.id, status: result?.status === 'unsafe' ? 'unsafe' : 'failed', code: result?.code || 500, message: result?.message || 'Command failed', attempt: active.attempt} @@ -363,7 +363,7 @@ async function main(options) { status.set(command.id, terminal) continue } - for (const file of config.requiredFiles) await syncFile(file) + for (const file of config.requiredFiles) { await syncFile(file) } await faultPoint('before-command-done-status') const done = {command: command.id, code:200, status:'done'} await appendFile(commandStatus, JSONTag.stringify(done)) diff --git a/src/storage.mjs b/src/storage.mjs index ca679aa..88e1965 100644 --- a/src/storage.mjs +++ b/src/storage.mjs @@ -5,7 +5,7 @@ import {randomUUID} from 'node:crypto' import {Buffer} from 'node:buffer' export function storageError(error) { - if (!(error instanceof Error)) error = new Error(String(error)) + if (!(error instanceof Error)) { error = new Error(String(error)) } error.storageFailure = true return error } @@ -30,7 +30,7 @@ async function withHandle(file, flags, operation, mode) { if (handle) { try { await handle.close() } catch (error) { failure ??= error } } - if (failure) throw storageError(failure) + if (failure) { throw storageError(failure) } return result } @@ -55,7 +55,7 @@ export async function syncAncestors(directory) { for (let current = directory; ; current = path.dirname(current)) { await syncDirectory(current) const parent = path.dirname(current) - if (parent === current || (await fsp.stat(parent)).dev !== device) break + if (parent === current || (await fsp.stat(parent)).dev !== device) { break } } } @@ -69,7 +69,7 @@ export async function durableMkdir(directory) { if (error.code === 'ENOENT') { await durableMkdir(path.dirname(directory)) await fsp.mkdir(directory) - } else if (error.code !== 'EEXIST') throw storageError(error) + } else if (error.code !== 'EEXIST') { throw storageError(error) } } await syncDirectory(directory) await syncDirectory(path.dirname(directory)) @@ -81,11 +81,11 @@ export async function publishFile(file, data, {mode} = {}) { try { try { const stat = await fsp.lstat(file) - if (stat.isSymbolicLink()) throw new Error('Refusing to publish over a symlink') + if (stat.isSymbolicLink()) { throw new Error('Refusing to publish over a symlink') } mode ??= stat.mode & 0o777 - } catch (error) { if (error.code !== 'ENOENT') throw error } + } catch (error) { if (error.code !== 'ENOENT') { throw error } } await withHandle(temporary, 'wx', async handle => { - if (mode !== undefined) await handle.chmod(mode) + if (mode !== undefined) { await handle.chmod(mode) } await writeAll(handle, data) await handle.sync() }, mode) @@ -93,7 +93,7 @@ export async function publishFile(file, data, {mode} = {}) { renamed = true await syncDirectory(path.dirname(file)) } catch (error) { - if (!renamed) await fsp.unlink(temporary).catch(() => {}) + if (!renamed) { await fsp.unlink(temporary).catch(() => {}) } throw storageError(error) } } @@ -101,7 +101,7 @@ export async function publishFile(file, data, {mode} = {}) { const appenders = new Map() export function appendRecord(file, record) { const key = path.resolve(file) - if (!appenders.has(key)) appenders.set(key, serialWriter()) + if (!appenders.has(key)) { appenders.set(key, serialWriter()) } return appenders.get(key)(async () => { await withHandle(file, 'a', async handle => { await writeAll(handle, record + '\n') @@ -120,15 +120,15 @@ export function publishFileSync(file, data) { let mode try { const stat = fs.lstatSync(file) - if (stat.isSymbolicLink()) throw new Error('Refusing to publish over a symlink') + if (stat.isSymbolicLink()) { throw new Error('Refusing to publish over a symlink') } mode = stat.mode & 0o777 - } catch (error) { if (error.code !== 'ENOENT') throw error } + } catch (error) { if (error.code !== 'ENOENT') { throw error } } fd = fs.openSync(temporary, 'wx', mode) - if (mode !== undefined) fs.fchmodSync(fd,mode) + if (mode !== undefined) { fs.fchmodSync(fd,mode) } let offset = 0 while (offset < bytes.length) { const count = fs.writeSync(fd, bytes, offset, bytes.length - offset) - if (!Number.isInteger(count) || count <= 0 || count > bytes.length - offset) throw new Error('Invalid or zero-progress write') + if (!Number.isInteger(count) || count <= 0 || count > bytes.length - offset) { throw new Error('Invalid or zero-progress write') } offset += count } fs.fsyncSync(fd) @@ -137,14 +137,14 @@ export function publishFileSync(file, data) { try { fs.closeSync(fd) } catch (error) { failure ??= error } } try { - if (failure) throw failure + if (failure) { throw failure } fs.renameSync(temporary, file) renamed = true fd = fs.openSync(path.dirname(file), 'r') failure = undefined try { fs.fsyncSync(fd) } catch (error) { failure = error } try { fs.closeSync(fd) } catch (error) { failure ??= error } - if (failure) throw failure + if (failure) { throw failure } } catch (error) { if (!renamed) { try { fs.unlinkSync(temporary) } catch { /* Preserve the primary failure. */ } } throw storageError(error) diff --git a/src/store-inspection.mjs b/src/store-inspection.mjs index 95003f7..2eda147 100644 --- a/src/store-inspection.mjs +++ b/src/store-inspection.mjs @@ -28,14 +28,14 @@ export async function inventory(config) { for (const directory of mutableDirectories(config)) { const entries = await fs.readdir(directory, {withFileTypes: true}) for (const entry of entries.sort((a,b) => a.name.localeCompare(b.name))) { - if (entry.name === '.simplystore-lock') continue + if (entry.name === '.simplystore-lock') { continue } const file = path.join(directory, entry.name) const dataArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith(stem) && (entry.name.endsWith(extension) || entry.name.endsWith('.tmp')) const indexArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith('index.') const temporary = [...expected].some(name=>file.startsWith(name+'.') && file.endsWith('.tmp')) - if (!expected.has(file) && !dataArtifact && !indexArtifact && !temporary) continue - if (entry.isSymbolicLink()) throw new Error(`Symlink in store artifact: ${file}`) - if (entry.isFile()) files[file] = hash(await fs.readFile(file)) + if (!expected.has(file) && !dataArtifact && !indexArtifact && !temporary) { continue } + if (entry.isSymbolicLink()) { throw new Error(`Symlink in store artifact: ${file}`) } + if (entry.isFile()) { files[file] = hash(await fs.readFile(file)) } } } for (const file of [config.datafile, config.commandLog, config.commandStatus, @@ -43,7 +43,7 @@ export async function inventory(config) { ...(config.schemaFile ? [config.schemaFile] : [])]) { if (!(file in files)) { try { files[file] = hash(await fs.readFile(file)) } catch (error) { - if (error.code !== 'ENOENT') throw error + if (error.code !== 'ENOENT') { throw error } files[file] = null } } @@ -56,13 +56,13 @@ async function records(file, errors, kind) { try { text = await fs.readFile(file, 'utf8') } catch (error) { errors.push(`${kind}: ${error.message}`); return [] } - if (text && !text.endsWith('\n')) errors.push(`${kind}: incomplete final record in ${file}`) + if (text && !text.endsWith('\n')) { errors.push(`${kind}: incomplete final record in ${file}`) } const result = [] for (const [index, line] of text.split('\n').entries()) { - if (!line) continue + if (!line) { continue } try { const value = JSONTag.parse(line) - if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('record is not an object') + if (!value || typeof value !== 'object' || Array.isArray(value)) { throw new Error('record is not an object') } result.push({value, line, lineNumber: index + 1}) } catch (error) { errors.push(`${kind} ${file}:${index + 1}: ${error.message}`) } } @@ -74,7 +74,7 @@ export async function inspectStore(options = {}) { const config = storePaths(options) const errors = [], warnings = [] const canonicalPaths = await Promise.all([config.datafile, config.commandLog, config.commandStatus, config.integrityFile].map(async file => path.join(await fs.realpath(path.dirname(file)),path.basename(file)))) - if (new Set(canonicalPaths).size !== canonicalPaths.length) errors.push('Configured canonical artifact paths overlap') + if (new Set(canonicalPaths).size !== canonicalPaths.length) { errors.push('Configured canonical artifact paths overlap') } const before = await inventory(config) const log = await records(config.commandLog, errors, 'command log') const statuses = await records(config.commandStatus, errors, 'command status') @@ -85,7 +85,7 @@ export async function inspectStore(options = {}) { errors.push(`Invalid command at log line ${record.lineNumber}`); continue } if (byId.has(id)) { - if (JSONTag.stringify(byId.get(id).command) !== JSONTag.stringify(record.value)) errors.push(`Conflicting command ID ${id}`) + if (JSONTag.stringify(byId.get(id).command) !== JSONTag.stringify(record.value)) { errors.push(`Conflicting command ID ${id}`) } continue } const command = {id, position: commands.length, command: record.value, line: record.line, @@ -99,8 +99,8 @@ export async function inspectStore(options = {}) { if (!['accepted', 'active', 'done', 'failed', 'unsafe'].includes(value.status)) { errors.push(`Unknown status for ${command.id}: ${value.status}`); continue } - if (value.status === 'accepted') command.accepted = true - if (value.status !== 'accepted' && !command.accepted) errors.push(`No acceptance evidence for ${command.id}`) + if (value.status === 'accepted') { command.accepted = true } + if (value.status !== 'accepted' && !command.accepted) { errors.push(`No acceptance evidence for ${command.id}`) } if (value.status === 'done') { doneOrder.delete(command.id); doneOrder.set(command.id, command.position) } @@ -108,8 +108,8 @@ export async function inspectStore(options = {}) { } let lastDonePosition = -1 for (const [id, position] of doneOrder) { - if (byId.get(id).status !== 'done') continue - if (position < lastDonePosition) errors.push(`Committed execution differs from command-log order at ${id}`) + if (byId.get(id).status !== 'done') { continue } + if (position < lastDonePosition) { errors.push(`Committed execution differs from command-log order at ${id}`) } lastDonePosition = position } let manifest @@ -117,7 +117,7 @@ export async function inspectStore(options = {}) { // Preserve optional existence-based compatibility; required integrity is explicit. const enabled = config.integrity || before[config.integrityFile] != null if (enabled) { - if (before[config.integrityFile] == null) throw new Error(`Missing integrity manifest ${config.integrityFile}`) + if (before[config.integrityFile] == null) { throw new Error(`Missing integrity manifest ${config.integrityFile}`) } await records(config.integrityFile, errors, 'integrity manifest') manifest = await loadIntegrityManifest(config.integrityFile) } @@ -128,18 +128,18 @@ export async function inspectStore(options = {}) { async function validateData(file) { const bytes = await fs.readFile(file) assertOdJsonTagFraming(bytes, file) - if (manifest) verifyIntegrity(manifest, config.integrityFile, file, bytes, {required: true}) + if (manifest) { verifyIntegrity(manifest, config.integrityFile, file, bytes, {required: true}) } return bytes } try { const bytes = await validateData(config.datafile) - if (!bytes.length) throw new Error('Empty base dataset') + if (!bytes.length) { throw new Error('Empty base dataset') } data = parser.parse(bytes); buffers.push(bytes) } catch (error) { errors.push(error.message); prefixValid = false } for (const command of commands) { command.present = before[command.file] != null command.condition = command.present ? 'present' : 'missing' - if (!command.accepted) errors.push(`Orphan command without acceptance: ${command.id}`) + if (!command.accepted) { errors.push(`Orphan command without acceptance: ${command.id}`) } if (command.present) { try { await validateData(command.file) } catch (error) { command.condition = 'corrupt'; command.problem = error.message @@ -167,9 +167,9 @@ export async function inspectStore(options = {}) { errors.push(`Unexplained changeset: ${file}`) } } - for (const file of config.requiredFiles) if (before[file] == null) errors.push(`Missing required artifact: ${file}`) + for (const file of config.requiredFiles) { if (before[file] == null) { errors.push(`Missing required artifact: ${file}`) } } const after = await inventory(config) - if (JSON.stringify(before) !== JSON.stringify(after)) errors.push('Store changed during inspection') + if (JSON.stringify(before) !== JSON.stringify(after)) { errors.push('Store changed during inspection') } const pending = commands.filter(c => c.status === 'accepted' || c.status === 'active') const damaged = commands.filter(c => c.problem || (c.present && c.status !== 'done')) const ready = errors.length === 0 && pending.length === 0 && damaged.length === 0 diff --git a/src/store-ownership.mjs b/src/store-ownership.mjs index bad2d09..c510a63 100644 --- a/src/store-ownership.mjs +++ b/src/store-ownership.mjs @@ -15,10 +15,10 @@ export async function acquireOwnership(directories, {ancestorToken, purpose, aud await fs.access(lock) let owner try { owner = JSON.parse(await fs.readFile(path.join(lock,'owner.json'),'utf8')) } catch { /* An incomplete lock remains exclusive. */ } - if (!ancestorToken || owner?.token !== ancestorToken) throw new Error(`Store is locked by unfinished parent operation: ${lock}`) - } catch (error) { if (error.code !== 'ENOENT') throw error } + if (!ancestorToken || owner?.token !== ancestorToken) { throw new Error(`Store is locked by unfinished parent operation: ${lock}`) } + } catch (error) { if (error.code !== 'ENOENT') { throw error } } } - if (path.dirname(parent) === parent) break + if (path.dirname(parent) === parent) { break } } } const token = randomUUID(), held = [] @@ -26,7 +26,7 @@ export async function acquireOwnership(directories, {ancestorToken, purpose, aud for (const directory of canonical) { const lock = path.join(directory, '.simplystore-lock') try { await fs.mkdir(lock) } catch (error) { - if (error.code === 'EEXIST') throw new Error(`Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.`) + if (error.code === 'EEXIST') { throw new Error(`Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.`) } throw error } held.push(lock) @@ -41,7 +41,7 @@ export async function acquireOwnership(directories, {ancestorToken, purpose, aud return {token, directories: canonical, async release() { for (const lock of [...held].reverse()) { const owner = JSON.parse(await fs.readFile(path.join(lock, 'owner.json'), 'utf8')) - if (owner.token !== token) throw new Error(`Ownership changed: ${lock}`) + if (owner.token !== token) { throw new Error(`Ownership changed: ${lock}`) } await fs.unlink(path.join(lock, 'owner.json')) await fs.rmdir(lock) await syncDirectory(path.dirname(lock)) diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs index 7396d93..ff7c675 100644 --- a/test/admin-recovery.mjs +++ b/test/admin-recovery.mjs @@ -34,13 +34,13 @@ test('administrator recovers trailing missing/waiting commands in log order on a assert.ok((await fs.readFile(result.config.commandStatus)).subarray(0,before.length).equals(before)) assert.deepEqual((await fs.readFile(result.config.commandLog)).toString(),(await fs.readFile(store.commandLog)).toString()) }) -for (const state of ['accepted','done','unsafe']) test(`later ${state} dataset blocks rerun, including zero-byte no-op`,async t=>{ +for (const state of ['accepted','done','unsafe']) { test(`later ${state} dataset blocks rerun, including zero-byte no-op`,async t=>{ const {store}=await setup(t,[['A','done'],['B',state]]) await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') const plan=await planRecovery(store,{quiescent:true}) assert.equal(plan.actionable,false) assert.ok(plan.blocks.some(s=>s.includes('later accepted datasets'))) -}) +}) } test('new later file makes an approved plan stale before executing a handler',async t=>{ const {store,outputs}=await setup(t,[['A','done'],['B','accepted']]) const plan=await planRecovery(store,{quiescent:true}) @@ -61,7 +61,7 @@ test('cooperative ownership excludes competing writer and path aliases',async t= try { await fs.symlink(store.dir,path.join(outputs,'alias')) await assert.rejects(acquireOwnership([path.join(outputs,'alias')]),/locked/) - } finally {await owner.release()} + } finally { await owner.release() } }) test('backup and restore validate saved bytes without executing commands',async t=>{ const {store,outputs}=await setup(t,[['A','done']]) @@ -214,7 +214,7 @@ test('verified A prefix permits only missing B, waiting C and missing D in log o const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted'],['C','accepted'],['D','accepted']]) const complete=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) const source={...complete.config,commandsFile:store.commandsFile,indexFile:store.indexFile} - for(const id of ['B','C','D'])await fs.unlink(path.join(path.dirname(source.datafile),`data.${id}.jsontag`)) + for(const id of ['B','C','D']){ await fs.unlink(path.join(path.dirname(source.datafile),`data.${id}.jsontag`)) } await fs.appendFile(source.commandStatus,JSONTag.stringify({command:'C',status:'accepted',code:202})+'\n') const prefix=await fs.readFile(path.join(path.dirname(source.datafile),'data.A.jsontag')) const plan=await planRecovery(source,{quiescent:true}) diff --git a/test/durability-crash.mjs b/test/durability-crash.mjs index 4008f55..be13b1d 100644 --- a/test/durability-crash.mjs +++ b/test/durability-crash.mjs @@ -90,7 +90,7 @@ for (const faultPointName of [ 'before-command-changeset-write', 'after-command-changeset-write', 'before-command-done-status' -]) test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { +]) { test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { const fixture=await makeServerFixture(t) const port=await getOpenPort() const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:faultPointName}) @@ -104,7 +104,7 @@ for (const faultPointName of [ assert.match(second.getOutput(),/Administrative recovery required/) assert.deepEqual(await fs.readFile(fixture.commandStatus),before) assert.equal((await readCommandLogRecords(fixture))[0].id,'A') -}) +}) } test('crash after done status but before query update recovers committed state without replay', async t => { const fixture = await makeServerFixture(t) diff --git a/test/durability-helpers.mjs b/test/durability-helpers.mjs index ec493a5..0eb436a 100644 --- a/test/durability-helpers.mjs +++ b/test/durability-helpers.mjs @@ -115,7 +115,7 @@ export async function waitForServer(child, getOutput, port) { reject(new Error(`Server exited before startup (${code || signal}):\n${getOutput()}`)) } const checkReady = () => { - if (finished) return + if (finished) { return } if (getOutput().includes(`SimplyStore listening on port ${port}`)) { finished = true clearTimeout(timeout) diff --git a/test/durability-latency.mjs b/test/durability-latency.mjs index 8504aca..0c2e331 100644 --- a/test/durability-latency.mjs +++ b/test/durability-latency.mjs @@ -14,9 +14,9 @@ test('measure local startup, acceptance, and completion latency',async t=>{ const start=performance.now(),id=`latency-${i}` assert.equal((await postCommand(port,{id,name:'addPerson',value:{name:id}})).status,202) accepted.push(performance.now()-start) - for(;;){const status=await getCommandStatus(port,id);if(status.status==='done')break;assert.equal(['accepted','active'].includes(status.status),true);await new Promise(r=>setTimeout(r,2))} + for(;;){ const status=await getCommandStatus(port,id);if(status.status==='done'){ break; }assert.equal(['accepted','active'].includes(status.status),true);await new Promise(r=>setTimeout(r,2)) } completed.push(performance.now()-start) } - const stats=values=>{const v=[...values].sort((a,b)=>a-b);return {p50Ms:v[Math.floor(v.length*.5)],p95Ms:v[Math.floor(v.length*.95)],maxMs:v.at(-1)}} + const stats=values=>{ const v=[...values].sort((a,b)=>a-b);return {p50Ms:v[Math.floor(v.length*.5)],p95Ms:v[Math.floor(v.length*.95)],maxMs:v.at(-1)} } t.diagnostic(JSON.stringify({samples:30,startupMs,acceptance:stats(accepted),completion:stats(completed),filesystem:execFileSync('stat',['-f','-c','%T',store.dir],{encoding:'utf8'}).trim(),node:process.version,notes:'Single localhost client; completion includes a fresh worker and polling; no production performance claim'})) }) diff --git a/test/durability-startup.mjs b/test/durability-startup.mjs index a9b91e3..c4aaf01 100644 --- a/test/durability-startup.mjs +++ b/test/durability-startup.mjs @@ -253,7 +253,7 @@ test('durable command log replays only accepted commands', async t => { assert.equal(JSONTag.parse(commands[0].command).name, 'addPerson') }) -for (const attempt of [1, 2]) test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { +for (const attempt of [1, 2]) { test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { const fixture = await makeFixture(t) const file = path.join(fixture.dir, 'status.jsontag') await writeJsonTagLines(file, [{command:'A',status:'active',attempt}]) @@ -262,7 +262,7 @@ for (const attempt of [1, 2]) test(`active command at attempt ${attempt} remains await assert.rejects(recoverActiveCommands(status,file,{maxCrashAttempts:2}), /administrator assessment/) assert.equal(status.get('A').status,'active') assert.deepEqual(await fs.readFile(file),before) -}) +}) } test('malformed durable status record refuses recovery with explicit integrity error', async t => { const fixture = await makeFixture(t) diff --git a/test/power-loss-baseline.mjs b/test/power-loss-baseline.mjs index d136e66..78f12e9 100644 --- a/test/power-loss-baseline.mjs +++ b/test/power-loss-baseline.mjs @@ -55,18 +55,18 @@ for (const [name,op,file,contains] of [ ['changeset sync','sync','data.A.jsontag.'], ['changeset rename','rename','data.A.jsontag'], ['done sync','datasync','command-status.jsontag','"status":"done"'] -]) test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`,async t=>{ +]) { test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`,async t=>{ const store=await fixture(t),server=await ready(t,store,{fault:{op,file,contains,action:'error'}}) - try {await submit(store,server.port,command('A'))} catch(error){assert.match(error.message,/fetch failed|terminated|socket/)} + try { await submit(store,server.port,command('A')) } catch(error){ assert.match(error.message,/fetch failed|terminated|socket/) } assert.equal((await waitForExit(server.child)).code,1) assert.match(server.getOutput(),/Storage outcome uncertain/) assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) const status=await readCommandStatusRecords(store) // Failed sync can leave a complete but uncertain done record; never append failed after it. const done=status.findIndex(s=>s.status==='done') - if(done>=0)assert.equal(status.slice(done+1).some(s=>s.status==='failed'),false) - else assert.equal(status.some(s=>s.status==='done'),false) -}) + if(done>=0){ assert.equal(status.slice(done+1).some(s=>s.status==='failed'),false) } + else { assert.equal(status.some(s=>s.status==='done'),false) } +}) } test('PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip',async t=>{ const store=await fixture(t,{commandsSource:`export default {addPerson(data,command,request,meta){ if(request!==undefined)throw new Error('unlogged input leaked') @@ -84,7 +84,7 @@ parentPort.on('message',async task=>{if('request' in task)throw new Error('reque assert.equal(JSONTag.stringify((await readCommandLogRecords(store))[0]),JSONTag.stringify(input)) assert.equal((await queryPersons(server.port))[0].name,input.value.name) }) -for(const duplicate of [false,true])test(`PL16 delayed append cannot be overtaken${duplicate?' by duplicate ID':''}`,async t=>{ +for(const duplicate of [false,true]){ test(`PL16 delayed append cannot be overtaken${duplicate?' by duplicate ID':''}`,async t=>{ const store=await fixture(t),server=await ready(t,store,{fault:{op:'datasync:after',file:'command-log.jsontag',contains:'"id":"A"',action:'pause'}}) const a=submit(store,server.port,command('A')) await eventually(async()=> (await rows(store.trace)).some(e=>e.op==='injected'),'A append gate') @@ -98,8 +98,8 @@ for(const duplicate of [false,true])test(`PL16 delayed append cannot be overtake assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),expected) assert.deepEqual((await readCommandStatusRecords(store)).filter(s=>s.status==='active').map(s=>s.command),expected) assert.deepEqual((await queryPersons(server.port)).map(p=>p.name),expected) -}) -for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.jsontag'])test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`,async t=>{ +}) } +for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.jsontag']){ test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`,async t=>{ const store=await fixture(t),server=await ready(t,store);await commit(store,server) await crash(server.child);await unlockStoppedFixture(t,store) await fs.unlink(path.join(store.dir,artifact)) @@ -108,13 +108,13 @@ for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.js assert.equal((await waitForExit(restarted.child)).code,1) assert.match(restarted.getOutput(),/Administrative recovery required/) assert.deepEqual((await fs.readdir(store.dir)).sort(),before.sort()) -}) +}) } test('PL12/17 startup preserves uncertain active work without repeating external effects',async t=>{ const store=await fixture(t,{commandsSource:`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify('/tmp/placeholder-effect')},'') ;data.persons.push(command.value)}}`}) const witness=path.join(store.audit,'effects') await fs.writeFile(store.commandsFile,`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify(witness)},command.id+'\\n');data.persons.push(command.value)}}`) const server=await ready(t,store,{fault:{op:'write',file:'command-status.jsontag',contains:'"status":"done"',action:'error'}}) - await submit(store,server.port,command('A'));await waitForExit(server.child).catch(error=>{throw new Error(server.getOutput(),{cause:error})}) + await submit(store,server.port,command('A'));await waitForExit(server.child).catch(error=>{ throw new Error(server.getOutput(),{cause:error}) }) const before=await fs.readFile(store.commandStatus) await unlockStoppedFixture(t,store) const restarted=await launch(t,store,await getOpenPort()) @@ -123,22 +123,22 @@ test('PL12/17 startup preserves uncertain active work without repeating external assert.equal(await fs.readFile(witness,'utf8'),'A\n') }) -for(const operation of ['sync','close:after'])test(`directory ${operation} failure after rename prevents done`,async t=>{ +for(const operation of ['sync','close:after']){ test(`directory ${operation} failure after rename prevents done`,async t=>{ const store=await fixture(t) const server=await ready(t,store,{fault:{op:operation,file:path.basename(store.dir),action:'error',after:{op:'rename',file:'data.A.jsontag'}}}) await submit(store,server.port,command('A')) assert.equal((await waitForExit(server.child)).code,1) assert.ok((await rows(store.trace)).some(e=>e.op==='injected'&&e.file===store.dir)) assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) -}) -for(const operation of ['open','write','close:after'])test(`changeset ${operation} error is not masked or acknowledged`,async t=>{ +}) } +for(const operation of ['open','write','close:after']){ test(`changeset ${operation} error is not masked or acknowledged`,async t=>{ const store=await fixture(t) const server=await ready(t,store,{fault:{op:operation,file:'data.A.jsontag.',action:'error',code:operation==='write'?'ENOSPC':'EIO'}}) await submit(store,server.port,command('A')) assert.equal((await waitForExit(server.child)).code,1) assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) assert.match(server.getOutput(),/Storage outcome uncertain/) -}) +}) } test('short command-log append preserves its entire JSONTag record',async t=>{ const store=await fixture(t) const server=await ready(t,store,{fault:{op:'write',file:'command-log.jsontag',contains:'"id":"A"',action:'short-write'}}) diff --git a/test/power-loss-helpers.mjs b/test/power-loss-helpers.mjs index 1d60d13..f53d201 100644 --- a/test/power-loss-helpers.mjs +++ b/test/power-loss-helpers.mjs @@ -45,7 +45,7 @@ export async function eventually(fn, description, timeout = 6000) { const end = Date.now() + timeout while (Date.now() < end) { const result = await fn() - if (result) return result + if (result) { return result } await new Promise(resolve => setTimeout(resolve, 10)) } assert.fail(`Timed out: ${description}`) @@ -107,7 +107,7 @@ export async function loseUnpublishedName(store, image, original) { assert.ok(publication >= 0, `trace contains publication of ${original}`) const durable = trace.slice(publication + 1).some(e => e.file === path.dirname(original) && e.op === 'sync') - if (!durable) await fs.rm(path.join(image.dir, path.basename(original))) + if (!durable) { await fs.rm(path.join(image.dir, path.basename(original))) } return !durable } diff --git a/test/power-loss-preload.mjs b/test/power-loss-preload.mjs index cf0386b..f76b813 100644 --- a/test/power-loss-preload.mjs +++ b/test/power-loss-preload.mjs @@ -20,8 +20,8 @@ if (configFile) { const scoped = file => typeof file === 'string' && (file === root || file.startsWith(root + path.sep)) function record(op, file, extra = {}) { - if (scoped(file)) raw.appendFileSync(config.trace, - JSON.stringify({op, file, threadId, ...extra}) + '\n') + if (scoped(file)) { raw.appendFileSync(config.trace, + JSON.stringify({op, file, threadId, ...extra}) + '\n') } } function snapshot(file) { return raw.statSync(file).isDirectory() ? {} : {bytes: raw.readFileSync(file).toString('base64')} @@ -30,14 +30,14 @@ if (configFile) { const rule = config.fault if (injected || !rule || rule.op !== op || !scoped(file) || !path.basename(file).startsWith(rule.file) || - (rule.contains && !content.includes(rule.contains))) return false - if (rule.after && !raw.readFileSync(config.trace,'utf8').split('\n').filter(Boolean).map(JSON.parse).some(event => event.op === rule.after.op && path.basename(event.file) === rule.after.file)) return false + (rule.contains && !content.includes(rule.contains))) { return false } + if (rule.after && !raw.readFileSync(config.trace,'utf8').split('\n').filter(Boolean).map(JSON.parse).some(event => event.op === rule.after.op && path.basename(event.file) === rule.after.file)) { return false } injected = true record('injected', file, {operation: op, action: rule.action}) if (rule.action === 'pause') { const deadline = Date.now() + 15000 while (!raw.existsSync(config.release)) { - if (Date.now() > deadline) throw new Error('Baseline gate was not released') + if (Date.now() > deadline) { throw new Error('Baseline gate was not released') } await new Promise(resolve => setTimeout(resolve, 5)) } } else if (rule.action === 'short-write') { @@ -59,16 +59,16 @@ if (configFile) { const existed = raw.existsSync(absolute) await fault('open', absolute) const handle = await open(file, ...args) - if (!scoped(absolute)) return handle + if (!scoped(absolute)) { return handle } record('open', absolute, {existed, flags: args[0]}) let content = '' for (const method of ['appendFile', 'write', 'datasync', 'sync', 'close']) { const original = handle[method].bind(handle) handle[method] = async (...params) => { - if (method === 'appendFile') content = String(params[0]) - if (method === 'write') content = String(params[0].subarray(params[1], params[1]+params[2])) + if (method === 'appendFile') { content = String(params[0]) } + if (method === 'write') { content = String(params[0].subarray(params[1], params[1]+params[2])) } const short = await fault(method, absolute, content) - if (short && method === 'write') params[2] = Math.max(1, Math.floor(params[2]/2)) + if (short && method === 'write') { params[2] = Math.max(1, Math.floor(params[2]/2)) } const result = await original(...params) record(method, absolute, method === 'datasync' || method === 'sync' ? snapshot(absolute) : method === 'write' ? {bytesWritten:result.bytesWritten} : {}) @@ -102,7 +102,7 @@ if (configFile) { args[2] = Math.max(1, Math.floor(args[2] / 2)) } raw.write(fd, ...args, (error, bytes, buffer) => { - if (!error) record('write', file, {bytesWritten: bytes}) + if (!error) { record('write', file, {bytesWritten: bytes}) } callback(error, bytes, buffer) }) }).catch(callback) @@ -110,7 +110,7 @@ if (configFile) { fs.fsync = function(fd, callback) { const file = handles.get(fd) void fault('fsync', file).then(() => raw.fsync(fd, error => { - if (!error && scoped(file)) record('sync', file, snapshot(file)) + if (!error && scoped(file)) { record('sync', file, snapshot(file)) } callback(error) })).catch(callback) } @@ -127,7 +127,7 @@ if (configFile) { fs.rename = function(from, to, callback) { const file = path.resolve(String(to)) void fault('rename', file).then(() => raw.rename(from, to, error => { - if (!error) record('rename', file, {from: path.resolve(String(from))}) + if (!error) { record('rename', file, {from: path.resolve(String(from))}) } callback(error) })).catch(callback) } diff --git a/test/storage.mjs b/test/storage.mjs index 9a5e8b9..00b4e07 100644 --- a/test/storage.mjs +++ b/test/storage.mjs @@ -8,15 +8,15 @@ import {writeAll,publishFile,publishFileSync,appendRecord,serialWriter} from '.. test('complete-write loop preserves bytes through repeated short writes and rejects zero progress', async()=>{ const bytes = Buffer.from('cafΓ© ζΌ’ε­— πŸ˜€'), written=[] - await writeAll({async write(buffer,offset,length){const n=Math.min(3,length);written.push(buffer.subarray(offset,offset+n));return {bytesWritten:n}}},bytes) + await writeAll({async write(buffer,offset,length){ const n=Math.min(3,length);written.push(buffer.subarray(offset,offset+n));return {bytesWritten:n} }},bytes) assert.deepEqual(Buffer.concat(written),bytes) - for (const count of [0,-1,NaN,1000]) await assert.rejects(writeAll({async write(){return {bytesWritten:count}}},bytes), /progress write/) + for (const count of [0,-1,NaN,1000]) { await assert.rejects(writeAll({async write(){ return {bytesWritten:count} }},bytes), /progress write/) } }) test('serialized writer preserves order and stops after an uncertain operation',async()=>{ const run=serialWriter(), seen=[] let release - const gate=new Promise(resolve=>{release=resolve}) - const first=run(async()=>{seen.push('A');await gate;throw new Error('uncertain')}) + const gate=new Promise(resolve=>{ release=resolve }) + const first=run(async()=>{ seen.push('A');await gate;throw new Error('uncertain') }) const second=run(async()=>seen.push('B')) release();await assert.rejects(first,/uncertain/);await assert.rejects(second,/uncertain/) assert.deepEqual(seen,['A']) From f51752db2a0746425c5ef8c9401754fa585f52a1 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:01:25 +0200 Subject: [PATCH 27/37] Put block contents and closing braces on separate lines --- .spiral/evidence/EVD-20260919-TTZ7C-27.md | 14 ++ .spiral/sources/SRC-20260919-TTZ7C-18.md | 8 + AGENTS.md | 2 +- eslint.config.js | 2 +- scripts/convert.mjs | 31 +-- scripts/power-loss-vm/guest.mjs | 52 +++-- scripts/recover.mjs | 80 ++++---- src/admin-recovery.mjs | 226 +++++++++++++++++----- src/command-worker-module.mjs | 22 ++- src/command-worker.mjs | 5 +- src/execute-worker.mjs | 38 +++- src/index.id.mjs | 3 +- src/index.offset.mjs | 3 +- src/recovery.mjs | 17 +- src/server.mjs | 147 +++++++++----- src/storage.mjs | 131 ++++++++++--- src/store-inspection.mjs | 127 +++++++++--- src/store-ownership.mjs | 42 +++- src/util.mjs | 20 +- test/admin-recovery.mjs | 25 ++- test/conversion-offsets.mjs | 20 +- test/durability-crash.mjs | 65 ++++--- test/durability-helpers.mjs | 21 +- test/durability-latency.mjs | 10 +- test/durability-startup.mjs | 37 ++-- test/power-loss-baseline.mjs | 129 ++++++------ test/power-loss-helpers.mjs | 23 ++- test/power-loss-preload.mjs | 52 +++-- test/storage.mjs | 18 +- 29 files changed, 950 insertions(+), 420 deletions(-) diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md index 4201202..8a8fc82 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.md +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -127,3 +127,17 @@ source strings were unchanged. This establishes that the edit adds control-flow blocks/spacing without changing the contained expressions, statements or branches. The runtime regression and VM evidence above remains tied to its exact original revision; no new power-cut experiment is claimed for this formatting correction. + +The subsequent block-layout clarification expands all statement/function blocks +in the same 26 JavaScript files onto multiple lines, including empty blocks and +callbacks. ESLint uses `brace-style: ["error", "stroustrup", {allowSingleLine: +false}]` alongside `curly`; repository instructions also explicitly cover empty +blocks. `else`, `catch` and `finally` follow the closing-brace line. Existing +file indentation conventions are retained and trailing whitespace removed. + +Compared the resulting Espree syntax trees with the files at +`941a96ee4158034e611018a03839d2b1e5c0c861`, ignoring only source positions and +literal raw spelling. All 26 trees are identical; 728 block statements were +checked, with zero inline blocks. Curly/brace-style/indentation/trailing-whitespace +checks pass. No runtime tests or VM experiments were repeated for this verified +whitespace-only adjustment. diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index 428e5ce..3bc6329 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -112,3 +112,11 @@ code growth, they requested: Use braces consistently in the cycle's JavaScript control flow and retain this preference in repository instructions. This is a formatting correction, not a change to the durability/recovery design or cycle acceptance. + +The maintainer clarified the block layout: + +> After a block { always start on a new line, and a closing } needs its own line as well + +Expand block bodies, including empty blocks and callbacks, onto separate lines; +place `else`, `catch`, and `finally` after the closing-brace line. Object literals +and destructuring braces are not statement blocks. This remains formatting-only. diff --git a/AGENTS.md b/AGENTS.md index ebe6d28..8317399 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -58,7 +58,7 @@ Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifa ## Working Defaults -- Always use braces for control-flow bodies, including single-statement `if`, `else`, and loop bodies. +- Always use braces for control-flow bodies, including single-statement `if`, `else`, and loop bodies. Start block contents on the line after `{`, and put the closing `}` on its own line. This includes empty blocks and callbacks; `else`, `catch`, and `finally` start on the next line after `}`. - Treat `master` as the authoritative branch unless the human states otherwise. - For ordinary repository-changing Spiral cycle work, create one dedicated branch from the authoritative branch, normally `spiral/CYC-###-short-goal`. diff --git a/eslint.config.js b/eslint.config.js index 7db434b..aaaef59 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -3,6 +3,6 @@ import globals from "globals"; import { defineConfig } from "eslint/config"; export default defineConfig([ - { rules: { curly: ["error", "all"] } }, + { rules: { curly: ["error", "all"], "brace-style": ["error", "stroustrup", { allowSingleLine: false }] } }, { files: ["**/*.{js,mjs,cjs}"], plugins: { js }, extends: ["js/recommended"], languageOptions: { globals: globals.browser } }, ]); diff --git a/scripts/convert.mjs b/scripts/convert.mjs index fb050e9..79f160a 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -25,7 +25,8 @@ let outputFile = args[1] let indexFile = args[2] if (indexFile && indexFile[0]!='/') { indexFile = process.cwd()+'/'+indexFile -} else if (!indexFile) { +} +else if (!indexFile) { indexFile = __dirname+'/../src/index.mjs' } let schemaFile = args[3] @@ -34,16 +35,16 @@ if (schemaFile && schemaFile[0]!='/') { } async function main() { - const directory = path.resolve(path.dirname(outputFile)) - const logs = ['command-log.jsontag', 'command-status.jsontag'].map(name => path.join(directory, name)) - const ownership = await acquireOwnership([directory]) - if ([outputFile, ...logs, getDefaultIntegrityFile(outputFile)].some(file => fs.existsSync(file))) { - await ownership.release() - throw new Error('Conversion requires new output files; existing store is not overwritten') - } + const directory = path.resolve(path.dirname(outputFile)) + const logs = ['command-log.jsontag', 'command-status.jsontag'].map(name => path.join(directory, name)) + const ownership = await acquireOwnership([directory]) + if ([outputFile, ...logs, getDefaultIntegrityFile(outputFile)].some(file => fs.existsSync(file))) { + await ownership.release() + throw new Error('Conversion requires new output files; existing store is not overwritten') + } // now create indexes console.log('Using index library:', indexFile) - + const index = await import(indexFile).then(mod => { return mod.default }) @@ -90,10 +91,16 @@ async function main() { await publishFile(outputFile, strData) // Custom indexes may change record sizes or add records after initial parsing. await finalizeIndex(index, strData, meta) - if (integrity) { await appendIntegrityRecord(getDefaultIntegrityFile(outputFile), outputFile, Buffer.from(strData)) } - for (const file of logs) { await publishFile(file, '') } + if (integrity) { + await appendIntegrityRecord(getDefaultIntegrityFile(outputFile), outputFile, Buffer.from(strData)) + } + for (const file of logs) { + await publishFile(file, '') + } for (const entry of fs.readdirSync(directory, {withFileTypes:true})) { - if (entry.isFile()) { await syncFile(path.join(directory, entry.name)) } + if (entry.isFile()) { + await syncFile(path.join(directory, entry.name)) + } } await ownership.release() console.log('Converted data written to ',outputFile) diff --git a/scripts/power-loss-vm/guest.mjs b/scripts/power-loss-vm/guest.mjs index 44bd7b4..9dcfb4e 100644 --- a/scripts/power-loss-vm/guest.mjs +++ b/scripts/power-loss-vm/guest.mjs @@ -10,13 +10,17 @@ const config={datafile:'/store/data.jsontag',commandLog:'/store/command-log.json console.log('ENVELOPE '+JSON.stringify({kernel:await fs.readFile('/proc/version','utf8'),mounts:await fs.readFile('/proc/mounts','utf8'),writeCache:await fs.readFile('/sys/block/vda/queue/write_cache','utf8')})) const exists=await fs.access(config.datafile).then(()=>true,()=>false) if(exists){ - const report=await inspectStore(config) - console.log('VERIFY '+JSON.stringify({ready:report.ready,errors:report.errors,commands:report.commands.map(c=>({id:c.id,status:c.status,present:c.present})),persons:report.data?.persons,unsyncedPresent:await fs.access('/store/unsynced').then(()=>true,()=>false)})) - process.exit(0) + const report=await inspectStore(config) + console.log('VERIFY '+JSON.stringify({ready:report.ready,errors:report.errors,commands:report.commands.map(c=>({id:c.id,status:c.status,present:c.present})),persons:report.data?.persons,unsyncedPresent:await fs.access('/store/unsynced').then(()=>true,()=>false)})) + process.exit(0) } await publishFile(config.datafile,serialize(JSONTag.parse('{"persons":[]}'))) await publishFile(config.commandLog,'');await publishFile(config.commandStatus,'') -if(mode==='unsynced'){ await fs.writeFile('/store/unsynced','not guaranteed durable');console.log('CUT_UNSYNCED');await new Promise(()=>{}) } +if(mode==='unsynced'){ + await fs.writeFile('/store/unsynced','not guaranteed durable');console.log('CUT_UNSYNCED');await new Promise(()=>{ + + }) +} await fs.writeFile('/app/commands.mjs',`export default {addPerson(data,command){${mode==='active'?"console.log('CUT_ACTIVE');while(true){}":mode==='accepted'?"while(true){}":"data.persons.push(command.value)"}}}`) await fs.writeFile('/app/index.mjs','export default {update(){},create(){},load(){return {}}}') await server.run({...config,port:3000,maxWorkers:1,commandTimeout:0,wwwroot:'/app/www'}) @@ -24,18 +28,32 @@ await new Promise(r=>setTimeout(r,100)) const before=performance.now() const response=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id:'A',name:'addPerson',value:{name:'A'}})}) console.log('ACCEPTED '+JSON.stringify({http:response.status,ms:performance.now()-before})) -if(mode==='accepted'){ console.log('CUT_ACCEPTED') } +if(mode==='accepted'){ + console.log('CUT_ACCEPTED') +} if(mode==='done'){ - for(;;){ const status=await fetch('http://127.0.0.1:3000/command/A').then(r=>r.json());if(status.status==='done'){ break; }await new Promise(r=>setTimeout(r,10)) } - const samples=[{id:'A',doneMs:performance.now()-before}] - for(let i=0;i<4;i++){ - const id='L'+i,start=performance.now() - const accepted=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id,name:'addPerson',value:{name:id}})}) - if(accepted.status!==202){ throw new Error('Acceptance failed') } - const acceptMs=performance.now()-start - for(;;){ const status=await fetch('http://127.0.0.1:3000/command/'+id).then(r=>r.json());if(status.status==='done'){ break; }await new Promise(r=>setTimeout(r,10)) } - samples.push({id,acceptMs,doneMs:performance.now()-start}) - } - console.log('CUT_DONE '+JSON.stringify({samples})) + for(;;){ + const status=await fetch('http://127.0.0.1:3000/command/A').then(r=>r.json());if(status.status==='done'){ + break; + }await new Promise(r=>setTimeout(r,10)) + } + const samples=[{id:'A',doneMs:performance.now()-before}] + for(let i=0;i<4;i++){ + const id='L'+i,start=performance.now() + const accepted=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id,name:'addPerson',value:{name:id}})}) + if(accepted.status!==202){ + throw new Error('Acceptance failed') + } + const acceptMs=performance.now()-start + for(;;){ + const status=await fetch('http://127.0.0.1:3000/command/'+id).then(r=>r.json());if(status.status==='done'){ + break; + }await new Promise(r=>setTimeout(r,10)) + } + samples.push({id,acceptMs,doneMs:performance.now()-start}) + } + console.log('CUT_DONE '+JSON.stringify({samples})) } -await new Promise(()=>{}) +await new Promise(()=>{ + +}) diff --git a/scripts/recover.mjs b/scripts/recover.mjs index f989b36..205b555 100644 --- a/scripts/recover.mjs +++ b/scripts/recover.mjs @@ -7,49 +7,55 @@ import {inspectStore} from '../src/store-inspection.mjs' const [action,...args] = process.argv.slice(2) const options = {} for (let i=0;i JSON.parse(await fs.readFile(file,'utf8')) -const required = key => { if (typeof options[key] !== 'string') { throw new Error(`--${key} is required`); } return options[key] } +const required = key => { + if (typeof options[key] !== 'string') { + throw new Error(`--${key} is required`); + } return options[key] +} async function main() { switch (action) { - case 'inspect': { - const plan = await planRecovery(await read(required('store')), {quiescent:options.quiescent === true}) - await writePlan(required('out'),plan) - return plan - } - case 'apply': - return applyRecovery(await read(required('plan')), {to:required('to'),auditDir:required('audit-dir'), - approveRerun:typeof options['approve-rerun'] === 'string' ? options['approve-rerun'].split(',').filter(Boolean) : [], - operator:required('operator'),reason:required('reason')}) - case 'finish': - return finishRecovery(required('audit-dir'),{operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped']===true}) - case 'backup': - return backupStore(await read(required('store')), {to:required('to'),quiescent:options.quiescent === true}) - case 'restore': - return restoreBackup(required('backup'), {to:required('to'),auditDir:required('audit-dir'), - source:options.store ? await read(options.store) : undefined,sourceQuiescent:options.quiescent===true}) - case 'unlock': { - const store = await read(required('store')) - const release = await releaseOfflineLocks(store, {operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped'] === true}) - const audit = path.resolve(required('out')) - // writePlan enforces new output outside every source data directory. - await writePlan(audit, {...release,config:(await inspectStore(store)).config}) - await release.finish() - return {audit, released:release.removed.map(item=>item.lock)} - } - case 'verify': { - const report = await read(required('report')) - const current = await verifyCandidate(report) - const output = path.resolve(required('out')) - await writePlan(output,{config:report.config,kind:'simplystore-activation-preview',store:{...report.config,...report.code},committed:current.committed, - instruction:'Stop the old service, preserve its configuration, and explicitly select this store. This command does not deploy or switch a service.'}) - return {verified:true,activationPreview:output} - } - default: - throw new Error('Usage: recover.mjs inspect|apply|backup|restore|unlock|finish|verify --store config.json ...; see docs/recovery.md') + case 'inspect': { + const plan = await planRecovery(await read(required('store')), {quiescent:options.quiescent === true}) + await writePlan(required('out'),plan) + return plan + } + case 'apply': + return applyRecovery(await read(required('plan')), {to:required('to'),auditDir:required('audit-dir'), + approveRerun:typeof options['approve-rerun'] === 'string' ? options['approve-rerun'].split(',').filter(Boolean) : [], + operator:required('operator'),reason:required('reason')}) + case 'finish': + return finishRecovery(required('audit-dir'),{operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped']===true}) + case 'backup': + return backupStore(await read(required('store')), {to:required('to'),quiescent:options.quiescent === true}) + case 'restore': + return restoreBackup(required('backup'), {to:required('to'),auditDir:required('audit-dir'), + source:options.store ? await read(options.store) : undefined,sourceQuiescent:options.quiescent===true}) + case 'unlock': { + const store = await read(required('store')) + const release = await releaseOfflineLocks(store, {operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped'] === true}) + const audit = path.resolve(required('out')) + // writePlan enforces new output outside every source data directory. + await writePlan(audit, {...release,config:(await inspectStore(store)).config}) + await release.finish() + return {audit, released:release.removed.map(item=>item.lock)} + } + case 'verify': { + const report = await read(required('report')) + const current = await verifyCandidate(report) + const output = path.resolve(required('out')) + await writePlan(output,{config:report.config,kind:'simplystore-activation-preview',store:{...report.config,...report.code},committed:current.committed, + instruction:'Stop the old service, preserve its configuration, and explicitly select this store. This command does not deploy or switch a service.'}) + return {verified:true,activationPreview:output} + } + default: + throw new Error('Usage: recover.mjs inspect|apply|backup|restore|unlock|finish|verify --store config.json ...; see docs/recovery.md') } } main().then(result=>console.log(JSON.stringify(result,null,2))).catch(error=>{ diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs index 25bd4fa..ed7aa2c 100644 --- a/src/admin-recovery.mjs +++ b/src/admin-recovery.mjs @@ -25,19 +25,31 @@ async function codeHashes(code) { } async function canonicalNew(file) { const absolute = path.resolve(file) - try { await fs.lstat(absolute); throw new Error(`Destination already exists: ${absolute}`) } - catch (error) { if (error.code !== 'ENOENT') { throw error } } + try { + await fs.lstat(absolute); throw new Error(`Destination already exists: ${absolute}`) + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } return path.join(await fs.realpath(path.dirname(absolute)), path.basename(absolute)) } function outside(target, roots) { - for (const root of roots) { if (target === root || target.startsWith(root + path.sep) || root.startsWith(target + path.sep)) { - throw new Error(`Overlapping source/destination: ${target} and ${root}`) - } } + for (const root of roots) { + if (target === root || target.startsWith(root + path.sep) || root.startsWith(target + path.sep)) { + throw new Error(`Overlapping source/destination: ${target} and ${root}`) + } + } } async function assertFresh(plan) { const files = await inventory(plan.config) - if (hash(JSON.stringify(files)) !== plan.fingerprint) { throw new Error('Stale recovery plan: source inventory changed') } - if (JSON.stringify(await codeHashes(plan.code)) !== JSON.stringify(plan.codeHashes)) { throw new Error('Stale recovery plan: selected code changed') } + if (hash(JSON.stringify(files)) !== plan.fingerprint) { + throw new Error('Stale recovery plan: source inventory changed') + } + if (JSON.stringify(await codeHashes(plan.code)) !== JSON.stringify(plan.codeHashes)) { + throw new Error('Stale recovery plan: selected code changed') + } } export async function planRecovery(options, {quiescent = false} = {}) { const report = await inspectStore(options) @@ -45,10 +57,14 @@ export async function planRecovery(options, {quiescent = false} = {}) { const candidates = report.commands.filter(c => ['accepted','active'].includes(c.status) || (c.status === 'done' && !c.present)) const blocks = [...report.errors] for (const command of report.commands) { - if (command.condition === 'corrupt' || (command.present && command.status !== 'done')) { blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) } + if (command.condition === 'corrupt' || (command.present && command.status !== 'done')) { + blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) + } } for (const command of candidates) { - if (command.laterDatasets.length) { blocks.push(`${command.id}: later accepted datasets exist: ${command.laterDatasets.join(',')}`) } + if (command.laterDatasets.length) { + blocks.push(`${command.id}: later accepted datasets exist: ${command.laterDatasets.join(',')}`) + } } return {kind:'simplystore-recovery-plan', config:report.config, code, codeHashes:await codeHashes(code), files:report.files, fingerprint:report.fingerprint, actionable:quiescent && blocks.length === 0, @@ -65,19 +81,29 @@ export async function writePlan(file, plan) { async function copyStore(config, files, target) { const directories = [...new Set(Object.keys(files).map(file => path.dirname(file)))].sort() let common = directories[0] - while (!directories.every(dir => dir === common || dir.startsWith(common + path.sep) || common === path.parse(common).root)) { common = path.dirname(common) } + while (!directories.every(dir => dir === common || dir.startsWith(common + path.sep) || common === path.parse(common).root)) { + common = path.dirname(common) + } const mapFile = file => path.join(target, path.relative(common,file)) - for (const directory of directories) { await durableMkdir(path.join(target,path.relative(common,directory))) } + for (const directory of directories) { + await durableMkdir(path.join(target,path.relative(common,directory))) + } for (const [file,digest] of Object.entries(files)) { - if (digest === null) { continue } + if (digest === null) { + continue + } const bytes = await fs.readFile(file) - if (hash(bytes) !== digest) { throw new Error(`Source changed while copying: ${file}`) } + if (hash(bytes) !== digest) { + throw new Error(`Source changed while copying: ${file}`) + } await publishFile(mapFile(file), bytes, {mode:(await fs.stat(file)).mode & 0o777}) } const result = {...config, datafile:mapFile(config.datafile), commandLog:mapFile(config.commandLog), commandStatus:mapFile(config.commandStatus), integrityFile:mapFile(config.integrityFile), requiredFiles:config.requiredFiles.map(mapFile)} - if (config.schemaFile) { result.schemaFile = mapFile(config.schemaFile) } + if (config.schemaFile) { + result.schemaFile = mapFile(config.schemaFile) + } return result } async function prepareWorkspace(target, audit, roots) { @@ -88,16 +114,24 @@ async function prepareWorkspace(target, audit, roots) { } export async function applyRecovery(plan, {to, auditDir, approveRerun = [], operator, reason} = {}) { - if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) { throw new Error('Plan is not actionable; inspect a quiescent source and resolve blocking evidence') } - if (!operator || !reason) { throw new Error('Operator and assessment reason are required') } - if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) { throw new Error('Explicit approval must match the complete ordered rerun list') } + if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) { + throw new Error('Plan is not actionable; inspect a quiescent source and resolve blocking evidence') + } + if (!operator || !reason) { + throw new Error('Operator and assessment reason are required') + } + if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) { + throw new Error('Explicit approval must match the complete ordered rerun list') + } const sourceOwner = await acquireOwnership(mutableDirectories(plan.config)) let candidateOwner try { await assertFresh(plan) // Do not trust editable plan fields to bypass the source predicate. const fresh = await planRecovery({...plan.config,...plan.code}, {quiescent:true}) - if (!fresh.actionable || JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun)) { throw new Error('Source no longer satisfies ordered recovery predicate') } + if (!fresh.actionable || JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun)) { + throw new Error('Source no longer satisfies ordered recovery predicate') + } const workspace = await prepareWorkspace(to, auditDir, sourceOwner.directories) const {target,audit} = workspace const rootOwner = await acquireOwnership([target],{purpose:'recovery',auditDir:audit}) @@ -111,14 +145,20 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper const journal = path.join(audit,'attempts.jsonl') await appendRecord(journal, JSON.stringify({event:'authorized',planHash,operator,reason,rerun:plan.rerun,config,attestation:plan.attestation})) const inspection = await inspectStore(config) - if (inspection.errors.length) { throw new Error(inspection.errors.join('; ')) } + if (inspection.errors.length) { + throw new Error(inspection.errors.join('; ')) + } const parser = new Parser() let data - for (const bytes of inspection.buffers) { data = parser.parse(bytes) } + for (const bytes of inspection.buffers) { + data = parser.parse(bytes) + } // Serialize committed prefix, rebuilding parser indexes without custom hooks. let meta = {index:{id:new Map()}, data:path.dirname(config.datafile), parts:inspection.committed.length} const buffers = [serialize(data,{meta})] - if (config.schemaFile) { meta.schema = JSONTag.parse(await fs.readFile(config.schemaFile,'utf8')) } + if (config.schemaFile) { + meta.schema = JSONTag.parse(await fs.readFile(config.schemaFile,'utf8')) + } const expectedManifest = plan.files[plan.config.integrityFile] != null ? await loadIntegrityManifest(plan.config.integrityFile) : null for (const id of plan.rerun) { await assertFresh(plan) @@ -134,7 +174,9 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper await appendRecord(journal, JSON.stringify({event:'stopped',id,result:result && {status:result.status,message:result.message}})) throw new Error(`Recovery attempt ${id} uncertain or failed; inspect before approving another attempt`) } - for (const file of config.requiredFiles) { await syncFile(file) } + for (const file of config.requiredFiles) { + await syncFile(file) + } if (expectedManifest && command.status === 'done') { verifyIntegrity(expectedManifest, plan.config.integrityFile, command.file, result.data, {required:true}) } @@ -148,66 +190,98 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper await appendRecord(journal, JSON.stringify({event:'done',id,attempt:active.attempt})) } const complete = await inspectStore(config) - if (!complete.ready) { throw new Error('Recovered candidate is not ready: '+JSON.stringify({errors:complete.errors,commands:complete.commands.filter(c=>c.problem)})) } + if (!complete.ready) { + throw new Error('Recovered candidate is not ready: '+JSON.stringify({errors:complete.errors,commands:complete.commands.filter(c=>c.problem)})) + } const report = {kind:'simplystore-recovery-complete',planHash,config,code:plan.code,codeHashes:plan.codeHashes,files:complete.files, fingerprint:complete.fingerprint,committed:complete.committed,operator,reason,warnings:complete.warnings} await publishFile(path.join(audit,'complete.json'), JSON.stringify(report,null,2)) await candidateOwner.release(); candidateOwner = null await rootOwner.release() return report - } finally { + } + finally { // On failure retain candidate locks and its audit. The unchanged source can be released. await sourceOwner.release() } } export async function backupStore(options, {to, quiescent = false} = {}) { - if (!quiescent) { throw new Error('Backup requires a stopped/quiescent source') } + if (!quiescent) { + throw new Error('Backup requires a stopped/quiescent source') + } const config = storePaths(options), owner = await acquireOwnership(mutableDirectories(config)) try { const report = await inspectStore(config) - if (!report.ready) { throw new Error('Backup source is not a complete validated store') } + if (!report.ready) { + throw new Error('Backup source is not a complete validated store') + } const target = await canonicalNew(to); outside(target, owner.directories) await durableMkdir(target) const backupOwner = await acquireOwnership([target]) const copied = await copyStore(config, report.files, path.join(target,'store')) const check = await inspectStore(copied) - if (!check.ready) { throw new Error('Backup reconstruction failed') } - if (hash(JSON.stringify(await inventory(config))) !== report.fingerprint) { throw new Error('Backup source changed') } + if (!check.ready) { + throw new Error('Backup reconstruction failed') + } + if (hash(JSON.stringify(await inventory(config))) !== report.fingerprint) { + throw new Error('Backup source changed') + } const manifest = {kind:'simplystore-backup-complete',root:target,config:copied,files:check.files,fingerprint:check.fingerprint, committed:check.committed,sourceFingerprint:report.fingerprint,warnings:report.warnings} await publishFile(path.join(target,'complete.json'),JSON.stringify(manifest,null,2)) await backupOwner.release() return manifest - } finally { await owner.release() } + } + finally { + await owner.release() + } } export async function restoreBackup(backupDirectory, {to, auditDir, source, sourceQuiescent=false} = {}) { const backup = await fs.realpath(backupDirectory) const manifest = JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) - if (manifest.kind !== 'simplystore-backup-complete') { throw new Error('Backup is incomplete or unrecognized') } + if (manifest.kind !== 'simplystore-backup-complete') { + throw new Error('Backup is incomplete or unrecognized') + } const oldRoot=manifest.root - if(typeof oldRoot!=='string' || !path.isAbsolute(oldRoot) || manifest.fingerprint!==hash(JSON.stringify(manifest.files))){ throw new Error('Invalid backup inventory') } + if(typeof oldRoot!=='string' || !path.isAbsolute(oldRoot) || manifest.fingerprint!==hash(JSON.stringify(manifest.files))){ + throw new Error('Invalid backup inventory') + } const rebase=file=>{ - if(typeof file!=='string' || file!==path.resolve(file) || !file.startsWith(oldRoot+path.sep)){ throw new Error('Backup manifest points outside backup') } + if(typeof file!=='string' || file!==path.resolve(file) || !file.startsWith(oldRoot+path.sep)){ + throw new Error('Backup manifest points outside backup') + } return path.join(backup,path.relative(oldRoot,file)) } manifest.files=Object.fromEntries(Object.entries(manifest.files).map(([file,digest])=>[rebase(file),digest])) manifest.fingerprint=hash(JSON.stringify(manifest.files)) - for(const key of ['datafile','commandLog','commandStatus','integrityFile','schemaFile']){ if(manifest.config[key]){ manifest.config[key]=rebase(manifest.config[key]) } } + for(const key of ['datafile','commandLog','commandStatus','integrityFile','schemaFile']){ + if(manifest.config[key]){ + manifest.config[key]=rebase(manifest.config[key]) + } + } manifest.config.requiredFiles=manifest.config.requiredFiles.map(rebase) - if (mutableDirectories(manifest.config).some(dir=>dir!==backup && !dir.startsWith(backup+path.sep))) { throw new Error('Backup configuration points outside backup') } - if (source && !sourceQuiescent) { throw new Error('Source comparison requires a stopped/quiescent source') } + if (mutableDirectories(manifest.config).some(dir=>dir!==backup && !dir.startsWith(backup+path.sep))) { + throw new Error('Backup configuration points outside backup') + } + if (source && !sourceQuiescent) { + throw new Error('Source comparison requires a stopped/quiescent source') + } const owner = await acquireOwnership([...mutableDirectories(manifest.config),...(source ? mutableDirectories(storePaths(source)) : [])]) try { const report = await inspectStore(manifest.config) - if (!report.ready || report.fingerprint !== manifest.fingerprint) { throw new Error('Backup contents differ from completed manifest') } + if (!report.ready || report.fingerprint !== manifest.fingerprint) { + throw new Error('Backup contents differ from completed manifest') + } const {target,audit} = await prepareWorkspace(to,auditDir,[backup,...owner.directories]) const rootOwner = await acquireOwnership([target],{purpose:'restore',auditDir:audit}) const config = await copyStore(manifest.config,manifest.files,target) const candidate = await acquireOwnership(mutableDirectories(config).filter(dir=>dir!==target),{ancestorToken:rootOwner.token}) const check = await inspectStore(config) - if (!check.ready) { throw new Error('Restored candidate failed validation') } + if (!check.ready) { + throw new Error('Restored candidate failed validation') + } let missingFromBackup = null, sameBase = null if (source) { const retained = await inspectStore(source) @@ -220,33 +294,63 @@ export async function restoreBackup(backupDirectory, {to, auditDir, source, sour await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) await candidate.release(); await rootOwner.release() return complete - } finally { await owner.release() } + } + finally { + await owner.release() + } } // Explicit offline operation: no PID-age or liveness heuristic can authorize it. export async function releaseOfflineLocks(options, {operator, reason, confirmedStopped = false} = {}) { - if (!confirmedStopped || !operator || !reason) { throw new Error('Offline release requires confirmed stopped writer, operator, and reason') } + if (!confirmedStopped || !operator || !reason) { + throw new Error('Offline release requires confirmed stopped writer, operator, and reason') + } const config = storePaths(options), report = await inspectStore(config) const removed = [] const lockDirectories = new Set(await Promise.all(mutableDirectories(config).map(dir=>fs.realpath(dir)))) for (const directory of [...lockDirectories]) { for (let parent=path.dirname(directory); path.dirname(parent)!==parent; parent=path.dirname(parent)) { - try { await fs.access(path.join(parent,'.simplystore-lock')); lockDirectories.add(parent) } catch(error) { if(error.code!=='ENOENT'){ throw error } } + try { + await fs.access(path.join(parent,'.simplystore-lock')); lockDirectories.add(parent) + } + catch(error) { + if(error.code!=='ENOENT'){ + throw error + } + } } } for (const directory of [...lockDirectories].sort().reverse()) { const lock = path.join(directory,'.simplystore-lock') let entries - try { entries = await fs.readdir(lock) } catch (error) { if (error.code === 'ENOENT') { continue; } throw error } - if (entries.some(entry => entry !== 'owner.json' && !entry.endsWith('.tmp'))) { throw new Error(`Unknown lock contents: ${lock}`) } + try { + entries = await fs.readdir(lock) + } + catch (error) { + if (error.code === 'ENOENT') { + continue; + } throw error + } + if (entries.some(entry => entry !== 'owner.json' && !entry.endsWith('.tmp'))) { + throw new Error(`Unknown lock contents: ${lock}`) + } const saved = [] - for (const entry of entries) { saved.push({name:entry,bytes:(await fs.readFile(path.join(lock,entry))).toString('base64')}) } + for (const entry of entries) { + saved.push({name:entry,bytes:(await fs.readFile(path.join(lock,entry))).toString('base64')}) + } const ownerRecord=saved.find(item=>item.name==='owner.json') const owner=ownerRecord ? JSON.parse(Buffer.from(ownerRecord.bytes,'base64').toString()) : null if (owner?.purpose==='recovery' && report.ready) { let completed - try { completed=JSON.parse(await fs.readFile(path.join(owner.auditDir,'complete.json'),'utf8')) } catch { /* Missing report is not completion evidence. */ } - if(completed?.fingerprint!==report.fingerprint) { throw new Error('Completed-looking recovery candidate requires finish with its retained audit before unlocking') } + try { + completed=JSON.parse(await fs.readFile(path.join(owner.auditDir,'complete.json'),'utf8')) + } + catch { + /* Missing report is not completion evidence. */ + } + if(completed?.fingerprint!==report.fingerprint) { + throw new Error('Completed-looking recovery candidate requires finish with its retained audit before unlocking') + } } removed.push({lock,ownerFiles:saved}) } @@ -255,23 +359,33 @@ export async function releaseOfflineLocks(options, {operator, reason, confirmedS operator,reason,removed,async finish() { for (const {lock,ownerFiles} of removed) { for (const entry of ownerFiles) { - if ((await fs.readFile(path.join(lock,entry.name))).toString('base64') !== entry.bytes) { throw new Error('Lock changed after preview') } + if ((await fs.readFile(path.join(lock,entry.name))).toString('base64') !== entry.bytes) { + throw new Error('Lock changed after preview') + } + } + for (const entry of ownerFiles) { + await fs.unlink(path.join(lock,entry.name)) } - for (const entry of ownerFiles) { await fs.unlink(path.join(lock,entry.name)) } await fs.rmdir(lock); await syncDirectory(path.dirname(lock)) } }} } export async function finishRecovery(auditDirectory, {operator,reason,confirmedStopped=false}={}) { - if(!confirmedStopped || !operator || !reason){ throw new Error('Finishing requires confirmed stopped writer and administrator assessment') } + if(!confirmedStopped || !operator || !reason){ + throw new Error('Finishing requires confirmed stopped writer and administrator assessment') + } const audit=await fs.realpath(auditDirectory) const plan=JSON.parse(await fs.readFile(path.join(audit,'plan.json'),'utf8')) const authorization=JSON.parse((await fs.readFile(path.join(audit,'attempts.jsonl'),'utf8')).split('\n')[0]) - if(authorization.planHash!==hash(JSON.stringify(plan))){ throw new Error('Audit authorization does not match retained plan') } + if(authorization.planHash!==hash(JSON.stringify(plan))){ + throw new Error('Audit authorization does not match retained plan') + } await assertFresh(plan) const report=await inspectStore(authorization.config) - if(!report.ready || plan.rerun.some(id=>!report.committed.includes(id))){ throw new Error('Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun') } + if(!report.ready || plan.rerun.some(id=>!report.committed.includes(id))){ + throw new Error('Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun') + } const complete={kind:'simplystore-recovery-complete',planHash:authorization.planHash,config:report.config, code:plan.code,codeHashes:plan.codeHashes,files:report.files,fingerprint:report.fingerprint,committed:report.committed,operator,reason,warnings:report.warnings} await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) @@ -282,9 +396,15 @@ export async function finishRecovery(auditDirectory, {operator,reason,confirmedS } export async function verifyCandidate(report) { - if (!['simplystore-recovery-complete','simplystore-restore-complete'].includes(report.kind)) { throw new Error('Expected completed recovery/restore report') } + if (!['simplystore-recovery-complete','simplystore-restore-complete'].includes(report.kind)) { + throw new Error('Expected completed recovery/restore report') + } const current = await inspectStore(report.config) - if (!current.ready || current.fingerprint !== report.fingerprint) { throw new Error('Candidate changed or is incomplete') } - if (report.code && JSON.stringify(await codeHashes(report.code)) !== JSON.stringify(report.codeHashes)) { throw new Error('Selected code changed after recovery') } + if (!current.ready || current.fingerprint !== report.fingerprint) { + throw new Error('Candidate changed or is incomplete') + } + if (report.code && JSON.stringify(await codeHashes(report.code)) !== JSON.stringify(report.codeHashes)) { + throw new Error('Selected code changed after recovery') + } return current } diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index c5d4173..399087d 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -34,10 +34,12 @@ export const metaIdProxy = { if (!parser.meta.index.id.has(id)) { if (ref[getIndex]) { parser.meta.index.id.set(id, ref[getIndex]) - } else { + } + else { throw new Error('cannot set index.id for non-proxy') } - } else { + } + else { let line = parser.meta.index.id.get(id) resultArr[line] = ref } @@ -96,13 +98,17 @@ export default async function runCommand(commandStr) { let publishing = false try { let task = JSONTag.parse(commandStr, null, metaReadProxy) - if (!task.id) { throw new Error('missing command id') } - if (!task.name) { throw new Error('missing command name parameter') } + if (!task.id) { + throw new Error('missing command id') + } + if (!task.name) { + throw new Error('missing command name parameter') + } if (commands[task.name]) { let time = Date.now() await commands[task.name](dataspace, task, undefined, metaProxy) //TODO: if command/task makes no changes, skip updating data.jsontag and writing it, skip response.data - + const changes = meta.resultArray.filter(e => e[isChanged]) //FIXME: new entities should also report isChanged = true if (changes.length) { @@ -132,14 +138,16 @@ export default async function runCommand(commandStr) { response.meta.parts = meta.parts let end = Date.now() console.log('task time',end-time) - } else { + } + else { console.error('Command not found', task.name) throw { code: 404, message: "Command "+task.name+" not found" } } - } catch(err) { + } + catch(err) { console.error('task error', err) throw publishing ? storageError(err) : err } diff --git a/src/command-worker.mjs b/src/command-worker.mjs index 67248cf..6827b9e 100644 --- a/src/command-worker.mjs +++ b/src/command-worker.mjs @@ -1,12 +1,13 @@ import { parentPort } from 'node:worker_threads' -import runCommand, { initialize } from '../src/command-worker-module.mjs' +import runCommand, { initialize } from '../src/command-worker-module.mjs' parentPort.on('message', async data => { let result await initialize(data) try { result = await runCommand(data.command) - } catch (error) { + } + catch (error) { result = {status: 'failed', code: 500, message: error.message, storageFailure: Boolean(error.storageFailure)} } diff --git a/src/execute-worker.mjs b/src/execute-worker.mjs index 2936ade..35ceaee 100644 --- a/src/execute-worker.mjs +++ b/src/execute-worker.mjs @@ -5,19 +5,41 @@ export function executeWorker(filename, task, timeout = 30000) { const worker = new Worker(filename) let settled = false, timer const finish = async (settle, result) => { - if (settled) { return } + if (settled) { + return + } settled = true clearTimeout(timer) - try { await worker.terminate() } catch (error) { reject(error); return } + try { + await worker.terminate() + } + catch (error) { + reject(error); return + } settle(result) } - worker.on('message', result => { void finish(resolve, result) }) - worker.on('error', error => { void finish(reject, error) }) + worker.on('message', result => { + void finish(resolve, result) + }) + worker.on('error', error => { + void finish(reject, error) + }) worker.on('exit', code => { - if (!settled) { void finish(reject, new Error(`Command worker exited without a result (${code})`)) } + if (!settled) { + void finish(reject, new Error(`Command worker exited without a result (${code})`)) + } }) - if (timeout) { timer = setTimeout(() => { void finish(resolve, - {status: 'unsafe', code: 504, message: `command worker timed out after ${timeout}ms`}) }, timeout) } - try { worker.postMessage(task) } catch (error) { void finish(reject, error) } + if (timeout) { + timer = setTimeout(() => { + void finish(resolve, + {status: 'unsafe', code: 504, message: `command worker timed out after ${timeout}ms`}) + }, timeout) + } + try { + worker.postMessage(task) + } + catch (error) { + void finish(reject, error) + } }) } diff --git a/src/index.id.mjs b/src/index.id.mjs index cc91bc9..ada8552 100644 --- a/src/index.id.mjs +++ b/src/index.id.mjs @@ -38,7 +38,8 @@ export default { let filename if (!uuid) { filename = 'index.id.json' - } else { + } + else { filename = 'index.id.'+filename+'.json' } return JSON.parse(fs.readFileSync(meta.data+filename)) diff --git a/src/index.offset.mjs b/src/index.offset.mjs index 250fe75..3fabbff 100644 --- a/src/index.offset.mjs +++ b/src/index.offset.mjs @@ -44,7 +44,8 @@ export default { let filename if (!uuid) { filename = 'index.offset.json' - } else { + } + else { filename = 'index.offset.'+filename+'.json' } return JSON.parse(fs.readFileSync(meta.data+filename)) diff --git a/src/recovery.mjs b/src/recovery.mjs index a3093c1..895c500 100644 --- a/src/recovery.mjs +++ b/src/recovery.mjs @@ -27,7 +27,8 @@ function parseDurableRecord(file, line, lineNumber, recordKind) { throw new Error(`${recordKind} record must be an object`) } return record - } catch (cause) { + } + catch (cause) { throw new RecoveryIntegrityError(`Invalid ${recordKind} record`, { file, lineNumber, @@ -62,10 +63,12 @@ export function loadCommandStatus(commandStatusFile, logger = console) { assertStringField(command, 'status', commandStatusFile, index + 1, 'command status') status.set(command.command, command) } - } else { + } + else { logger.error('Could not open command status', commandStatusFile) } - } else { + } + else { logger.log('no command status', commandStatusFile) } return status @@ -115,10 +118,10 @@ function readAttempt(record, fallback = 1) { } export async function recoverActiveCommands(status) { - if ([...status.values()].some(command => command?.status === activeCommandStatus)) { - throw new RecoveryIntegrityError('Active command requires administrator assessment; automatic rerun is disabled') - } - return status + if ([...status.values()].some(command => command?.status === activeCommandStatus)) { + throw new RecoveryIntegrityError('Active command requires administrator assessment; automatic rerun is disabled') + } + return status } export function nextActiveCommandStatus(commandId, currentStatus) { diff --git a/src/server.mjs b/src/server.mjs index 358d558..e0371ce 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -50,10 +50,10 @@ async function main(options) { assertRuntimeEnvironmentConfiguration() if (!options) { options = {} - } + } const port = options.port || 3000 const datafile = options.datafile || './data.od-jsontag' - const schemaFile = options.schemaFile || null + const schemaFile = options.schemaFile || null const wwwroot = options.wwwroot || __dirname+'/www' const maxWorkers = options.maxWorkers || 8 const queryWorker = options.queryWorker || __dirname+'/src/query-worker.mjs' @@ -85,15 +85,22 @@ async function main(options) { let inspection try { inspection = await inspectStore(config) - if (!inspection.ready) { throw new Error('Administrative recovery required: ' + + if (!inspection.ready) { + throw new Error('Administrative recovery required: ' + [...inspection.errors, ...inspection.commands.filter(c => c.problem).map(c => c.problem), ...inspection.commands.filter(c => c.status === 'accepted' || c.status === 'active').map(c => `${c.id}: ${c.status}; administrator assessment required`), - ...inspection.commands.filter(c => c.present && c.status !== 'done').map(c => `${c.id}: uncommitted dataset`)].join('; ')) } - for (const [file, digest] of Object.entries(inspection.files)) { if (digest !== null) { await syncFile(file) } } + ...inspection.commands.filter(c => c.present && c.status !== 'done').map(c => `${c.id}: uncommitted dataset`)].join('; ')) + } + for (const [file, digest] of Object.entries(inspection.files)) { + if (digest !== null) { + await syncFile(file) + } + } const data = await loadData(inspection.committed) jsontagBuffers = [data.data] meta = data.meta - } catch (error) { + } + catch (error) { await ownership.release() throw error } @@ -103,14 +110,16 @@ async function main(options) { let storageFailed = false, closing = false let runner = Promise.resolve() function failStorage(error) { - if (storageFailed) { return } + if (storageFailed) { + return + } storageFailed = true console.error('Storage outcome uncertain; stopping mutation. Administrator recovery required:', error) // Retain ownership evidence; a process exit must never imply a rollback. setImmediate(() => process.exit(1)) } - const queryWorkerInitTask = () => { + const queryWorkerInitTask = () => { return { name: 'init', req: { @@ -148,18 +157,29 @@ async function main(options) { const listener = server.listen(port, () => { console.log('SimplyStore listening on port '+port) }) - listener.on('error', error => { failStorage(error) }) + listener.on('error', error => { + failStorage(error) + }) async function shutdown() { - if (closing) { return } + if (closing) { + return + } closing = true listener.close() try { - await acceptSerial(async () => {}) + await acceptSerial(async () => { + + }) await runner queryWorkerPool.close(); slowQueryWorkerPool.close() - if (!storageFailed) { await ownership.release() } + if (!storageFailed) { + await ownership.release() + } process.exit(storageFailed ? 1 : 0) - } catch (error) { failStorage(error) } + } + catch (error) { + failStorage(error) + } } process.once('SIGTERM', shutdown) process.once('SIGINT', shutdown) @@ -208,7 +228,8 @@ async function main(options) { integrityFile: integrityEnabled ? integrityFile : null, integrityRequired: integrityEnabled }) - } catch (error) { + } + catch (error) { void finish(reject, error) } }) @@ -220,7 +241,7 @@ async function main(options) { pool = queryWorkerPool } if ( !accept(req,res, - ['application/jsontag','application/json','text/html','text/javascript','image/*'], + ['application/jsontag','application/json','text/html','text/javascript','image/*'], function(req, res, accept) { let result = true switch(accept) { @@ -229,7 +250,7 @@ async function main(options) { case 'text/javascript': handleWebRequest(req,res,{root:wwwroot}); result = false - break + break } return result } @@ -251,7 +272,8 @@ async function main(options) { try { let result = await pool.run('query', request, { timeout }) sendResponse(result, res) - } catch(error) { + } + catch(error) { sendError(error, res) } let end = Date.now() @@ -271,7 +293,7 @@ async function main(options) { } let start = Date.now() if ( !accept(req,res, - ['application/jsontag','application/json']) + ['application/jsontag','application/json']) ) { sendError({code:406, message:'Not Acceptable',accept:['application/json','application/jsontag']},res) return @@ -290,12 +312,13 @@ async function main(options) { try { let result = await pool.run('query', request, {slowTimeout}) sendResponse(result, res) - } catch(error) { + } + catch(error) { sendError(error, res) } let end = Date.now() console.log(path, (end-start), process.memoryUsage()) -// queryWorkerPool.memoryUsage() + // queryWorkerPool.memoryUsage() } async function handleSlowPostQuery(req, res) { @@ -303,20 +326,33 @@ async function main(options) { } async function handlePostCommand(req, res) { - if (storageFailed || closing) { return sendResponse({code:503, body: JSON.stringify({message:'Store is unavailable'})}, res) } + if (storageFailed || closing) { + return sendResponse({code:503, body: JSON.stringify({message:'Store is unavailable'})}, res) + } try { await acceptSerial(async () => { - if (storageFailed) { throw new Error('Store is unavailable') } - if (closing) { return sendResponse({code:503,body:JSON.stringify({message:'Store is closing'})},res) } + if (storageFailed) { + throw new Error('Store is unavailable') + } + if (closing) { + return sendResponse({code:503,body:JSON.stringify({message:'Store is closing'})},res) + } const accepted = await checkCommand(req, res) - if (!accepted) { return } - if (storageFailed) { throw new Error('Store failed during acceptance') } + if (!accepted) { + return + } + if (storageFailed) { + throw new Error('Store failed during acceptance') + } commandQueue.push({id: accepted.id, command: accepted.line}) sendResponse({code:202, body:JSON.stringify(status.get(accepted.id))}, res) }) void drainCommandQueue() - } catch (error) { - if (!res.headersSent) { sendResponse({code:500, body:JSON.stringify({message:'Storage outcome uncertain'})}, res) } + } + catch (error) { + if (!res.headersSent) { + sendResponse({code:500, body:JSON.stringify({message:'Storage outcome uncertain'})}, res) + } failStorage(error) } } @@ -328,7 +364,8 @@ async function main(options) { jsontag: false, body: JSON.stringify(result) },res) - } else { + } + else { sendResponse({ code: 404, jsontag: false, @@ -338,7 +375,9 @@ async function main(options) { } function drainCommandQueue() { - if (commandRunnerActive || storageFailed) { return runner } + if (commandRunnerActive || storageFailed) { + return runner + } commandRunnerActive = true runner = (async () => { try { @@ -347,15 +386,21 @@ async function main(options) { console.log('starting command', command.id) const active = nextActiveCommandStatus(command.id, status.get(command.id)) await appendFile(commandStatus, JSONTag.stringify(active)) - if (storageFailed) { throw new Error('Store failed before execution') } + if (storageFailed) { + throw new Error('Store failed before execution') + } status.set(command.id, active) await faultPoint('after-active-status-before-command-worker') const result = await executeWorker(commandWorker, {...command, meta, data: jsontagBuffers, commandsFile, indexFile, datafile, integrityFile: integrityEnabled ? integrityFile : null, integrityRequired: integrityEnabled}, commandTimeout) - if (storageFailed) { throw new Error('Store failed during execution') } - if (result?.storageFailure) { throw new Error(result.message || 'Worker persistence failure') } + if (storageFailed) { + throw new Error('Store failed during execution') + } + if (result?.storageFailure) { + throw new Error(result.message || 'Worker persistence failure') + } if (!result || result.status === 'failed' || result.status === 'unsafe' || result.code >= 300) { const terminal = {command: command.id, status: result?.status === 'unsafe' ? 'unsafe' : 'failed', code: result?.code || 500, message: result?.message || 'Command failed', attempt: active.attempt} @@ -363,7 +408,9 @@ async function main(options) { status.set(command.id, terminal) continue } - for (const file of config.requiredFiles) { await syncFile(file) } + for (const file of config.requiredFiles) { + await syncFile(file) + } await faultPoint('before-command-done-status') const done = {command: command.id, code:200, status:'done'} await appendFile(commandStatus, JSONTag.stringify(done)) @@ -376,8 +423,13 @@ async function main(options) { queryWorkerPool.update(task); slowQueryWorkerPool.update(task) } } - } catch (error) { failStorage(error) } - finally { commandRunnerActive = false } + } + catch (error) { + failStorage(error) + } + finally { + commandRunnerActive = false + } })() return runner } @@ -393,7 +445,8 @@ async function main(options) { code: 202, status: 'accepted' } - } catch(err) { + } + catch(err) { error = { code: 400, message: "Bad request", @@ -411,24 +464,26 @@ async function main(options) { } sendResponse({code: 422, body: JSON.stringify(error)}, res) return false - } else if (status.has(command.id)) { + } + else if (status.has(command.id)) { const currentStatus = Object.assign({command: command.id}, status.get(command.id)) sendResponse({body: JSON.stringify(currentStatus)}, res) return false - } else if (!command.name) { + } + else if (!command.name) { error = { code: 422, message: "Command has no name", details: command } sendResponse({code:422, body: JSON.stringify(error)}, res) - return false + return false } const line = JSONTag.stringify(command) await appendFile(commandLog, line) await faultPoint('after-command-log-before-accepted-status') await appendFile(commandStatus, JSONTag.stringify(commandOK)) - status.set(command.id, commandOK) + status.set(command.id, commandOK) await faultPoint('after-command-accepted-status-before-response') return {id:command.id, line} } @@ -440,7 +495,8 @@ function sendResponse(response, res) { } if (response.jsontag) { res.setHeader('content-type','application/jsontag') - } else { + } + else { res.setHeader('content-type','application/json') } res.send(response.body)+"\n" @@ -450,7 +506,8 @@ function sendError(error, res) { console.error(error) if (error.code && httpStatusCodes[error.code]) { res.status(error.code) - } else { + } + else { res.status(500) } res.setHeader('content-type','application/json') @@ -473,8 +530,7 @@ function accept(req, res, mimetypes, handler) { return true } -function handleWebRequest(req,res,options) -{ +function handleWebRequest(req,res,options) { let path = req.path; path = path.replace(/[^a-z0-9_.\-/]*/gi, '') // whitelist acceptable file paths path = path.replace(/\.+/g, '.') // blacklist '..' @@ -489,7 +545,8 @@ function handleWebRequest(req,res,options) } if (fs.existsSync(fileOptions.root+path)) { res.sendFile(path, fileOptions) - } else { + } + else { res.sendFile('/index.html', fileOptions) } } diff --git a/src/storage.mjs b/src/storage.mjs index 88e1965..1aaed60 100644 --- a/src/storage.mjs +++ b/src/storage.mjs @@ -5,7 +5,9 @@ import {randomUUID} from 'node:crypto' import {Buffer} from 'node:buffer' export function storageError(error) { - if (!(error instanceof Error)) { error = new Error(String(error)) } + if (!(error instanceof Error)) { + error = new Error(String(error)) + } error.storageFailure = true return error } @@ -16,7 +18,9 @@ export function serialWriter() { return operation => { const result = tail.then(operation) tail = result - void result.catch(() => {}) + void result.catch(() => { + + }) return result } } @@ -26,11 +30,21 @@ async function withHandle(file, flags, operation, mode) { try { handle = await fsp.open(file, flags, mode) result = await operation(handle) - } catch (error) { failure = error } + } + catch (error) { + failure = error + } if (handle) { - try { await handle.close() } catch (error) { failure ??= error } + try { + await handle.close() + } + catch (error) { + failure ??= error + } + } + if (failure) { + throw storageError(failure) } - if (failure) { throw storageError(failure) } return result } @@ -55,7 +69,9 @@ export async function syncAncestors(directory) { for (let current = directory; ; current = path.dirname(current)) { await syncDirectory(current) const parent = path.dirname(current) - if (parent === current || (await fsp.stat(parent)).dev !== device) { break } + if (parent === current || (await fsp.stat(parent)).dev !== device) { + break + } } } @@ -65,11 +81,17 @@ export async function syncFile(file) { } export async function durableMkdir(directory) { - try { await fsp.mkdir(directory) } catch (error) { + try { + await fsp.mkdir(directory) + } + catch (error) { if (error.code === 'ENOENT') { await durableMkdir(path.dirname(directory)) await fsp.mkdir(directory) - } else if (error.code !== 'EEXIST') { throw storageError(error) } + } + else if (error.code !== 'EEXIST') { + throw storageError(error) + } } await syncDirectory(directory) await syncDirectory(path.dirname(directory)) @@ -81,19 +103,33 @@ export async function publishFile(file, data, {mode} = {}) { try { try { const stat = await fsp.lstat(file) - if (stat.isSymbolicLink()) { throw new Error('Refusing to publish over a symlink') } + if (stat.isSymbolicLink()) { + throw new Error('Refusing to publish over a symlink') + } mode ??= stat.mode & 0o777 - } catch (error) { if (error.code !== 'ENOENT') { throw error } } + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } await withHandle(temporary, 'wx', async handle => { - if (mode !== undefined) { await handle.chmod(mode) } + if (mode !== undefined) { + await handle.chmod(mode) + } await writeAll(handle, data) await handle.sync() }, mode) await fsp.rename(temporary, file) renamed = true await syncDirectory(path.dirname(file)) - } catch (error) { - if (!renamed) { await fsp.unlink(temporary).catch(() => {}) } + } + catch (error) { + if (!renamed) { + await fsp.unlink(temporary).catch(() => { + + }) + } throw storageError(error) } } @@ -101,7 +137,9 @@ export async function publishFile(file, data, {mode} = {}) { const appenders = new Map() export function appendRecord(file, record) { const key = path.resolve(file) - if (!appenders.has(key)) { appenders.set(key, serialWriter()) } + if (!appenders.has(key)) { + appenders.set(key, serialWriter()) + } return appenders.get(key)(async () => { await withHandle(file, 'a', async handle => { await writeAll(handle, record + '\n') @@ -120,33 +158,74 @@ export function publishFileSync(file, data) { let mode try { const stat = fs.lstatSync(file) - if (stat.isSymbolicLink()) { throw new Error('Refusing to publish over a symlink') } + if (stat.isSymbolicLink()) { + throw new Error('Refusing to publish over a symlink') + } mode = stat.mode & 0o777 - } catch (error) { if (error.code !== 'ENOENT') { throw error } } + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } fd = fs.openSync(temporary, 'wx', mode) - if (mode !== undefined) { fs.fchmodSync(fd,mode) } + if (mode !== undefined) { + fs.fchmodSync(fd,mode) + } let offset = 0 while (offset < bytes.length) { const count = fs.writeSync(fd, bytes, offset, bytes.length - offset) - if (!Number.isInteger(count) || count <= 0 || count > bytes.length - offset) { throw new Error('Invalid or zero-progress write') } + if (!Number.isInteger(count) || count <= 0 || count > bytes.length - offset) { + throw new Error('Invalid or zero-progress write') + } offset += count } fs.fsyncSync(fd) - } catch (error) { failure = error } + } + catch (error) { + failure = error + } if (fd !== undefined) { - try { fs.closeSync(fd) } catch (error) { failure ??= error } + try { + fs.closeSync(fd) + } + catch (error) { + failure ??= error + } } try { - if (failure) { throw failure } + if (failure) { + throw failure + } fs.renameSync(temporary, file) renamed = true fd = fs.openSync(path.dirname(file), 'r') failure = undefined - try { fs.fsyncSync(fd) } catch (error) { failure = error } - try { fs.closeSync(fd) } catch (error) { failure ??= error } - if (failure) { throw failure } - } catch (error) { - if (!renamed) { try { fs.unlinkSync(temporary) } catch { /* Preserve the primary failure. */ } } + try { + fs.fsyncSync(fd) + } + catch (error) { + failure = error + } + try { + fs.closeSync(fd) + } + catch (error) { + failure ??= error + } + if (failure) { + throw failure + } + } + catch (error) { + if (!renamed) { + try { + fs.unlinkSync(temporary) + } + catch { + /* Preserve the primary failure. */ + } + } throw storageError(error) } } diff --git a/src/store-inspection.mjs b/src/store-inspection.mjs index 2eda147..c89a3eb 100644 --- a/src/store-inspection.mjs +++ b/src/store-inspection.mjs @@ -28,22 +28,35 @@ export async function inventory(config) { for (const directory of mutableDirectories(config)) { const entries = await fs.readdir(directory, {withFileTypes: true}) for (const entry of entries.sort((a,b) => a.name.localeCompare(b.name))) { - if (entry.name === '.simplystore-lock') { continue } + if (entry.name === '.simplystore-lock') { + continue + } const file = path.join(directory, entry.name) const dataArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith(stem) && (entry.name.endsWith(extension) || entry.name.endsWith('.tmp')) const indexArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith('index.') const temporary = [...expected].some(name=>file.startsWith(name+'.') && file.endsWith('.tmp')) - if (!expected.has(file) && !dataArtifact && !indexArtifact && !temporary) { continue } - if (entry.isSymbolicLink()) { throw new Error(`Symlink in store artifact: ${file}`) } - if (entry.isFile()) { files[file] = hash(await fs.readFile(file)) } + if (!expected.has(file) && !dataArtifact && !indexArtifact && !temporary) { + continue + } + if (entry.isSymbolicLink()) { + throw new Error(`Symlink in store artifact: ${file}`) + } + if (entry.isFile()) { + files[file] = hash(await fs.readFile(file)) + } } } for (const file of [config.datafile, config.commandLog, config.commandStatus, ...(config.integrity ? [config.integrityFile] : []), ...config.requiredFiles, ...(config.schemaFile ? [config.schemaFile] : [])]) { if (!(file in files)) { - try { files[file] = hash(await fs.readFile(file)) } catch (error) { - if (error.code !== 'ENOENT') { throw error } + try { + files[file] = hash(await fs.readFile(file)) + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } files[file] = null } } @@ -53,18 +66,30 @@ export async function inventory(config) { async function records(file, errors, kind) { let text - try { text = await fs.readFile(file, 'utf8') } catch (error) { + try { + text = await fs.readFile(file, 'utf8') + } + catch (error) { errors.push(`${kind}: ${error.message}`); return [] } - if (text && !text.endsWith('\n')) { errors.push(`${kind}: incomplete final record in ${file}`) } + if (text && !text.endsWith('\n')) { + errors.push(`${kind}: incomplete final record in ${file}`) + } const result = [] for (const [index, line] of text.split('\n').entries()) { - if (!line) { continue } + if (!line) { + continue + } try { const value = JSONTag.parse(line) - if (!value || typeof value !== 'object' || Array.isArray(value)) { throw new Error('record is not an object') } + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new Error('record is not an object') + } result.push({value, line, lineNumber: index + 1}) - } catch (error) { errors.push(`${kind} ${file}:${index + 1}: ${error.message}`) } + } + catch (error) { + errors.push(`${kind} ${file}:${index + 1}: ${error.message}`) + } } return result } @@ -74,7 +99,9 @@ export async function inspectStore(options = {}) { const config = storePaths(options) const errors = [], warnings = [] const canonicalPaths = await Promise.all([config.datafile, config.commandLog, config.commandStatus, config.integrityFile].map(async file => path.join(await fs.realpath(path.dirname(file)),path.basename(file)))) - if (new Set(canonicalPaths).size !== canonicalPaths.length) { errors.push('Configured canonical artifact paths overlap') } + if (new Set(canonicalPaths).size !== canonicalPaths.length) { + errors.push('Configured canonical artifact paths overlap') + } const before = await inventory(config) const log = await records(config.commandLog, errors, 'command log') const statuses = await records(config.commandStatus, errors, 'command status') @@ -85,7 +112,9 @@ export async function inspectStore(options = {}) { errors.push(`Invalid command at log line ${record.lineNumber}`); continue } if (byId.has(id)) { - if (JSONTag.stringify(byId.get(id).command) !== JSONTag.stringify(record.value)) { errors.push(`Conflicting command ID ${id}`) } + if (JSONTag.stringify(byId.get(id).command) !== JSONTag.stringify(record.value)) { + errors.push(`Conflicting command ID ${id}`) + } continue } const command = {id, position: commands.length, command: record.value, line: record.line, @@ -95,12 +124,18 @@ export async function inspectStore(options = {}) { const doneOrder = new Map() for (const {value, lineNumber} of statuses) { const command = byId.get(value.command) - if (!command) { errors.push(`Status line ${lineNumber} has no logged command: ${value.command}`); continue } + if (!command) { + errors.push(`Status line ${lineNumber} has no logged command: ${value.command}`); continue + } if (!['accepted', 'active', 'done', 'failed', 'unsafe'].includes(value.status)) { errors.push(`Unknown status for ${command.id}: ${value.status}`); continue } - if (value.status === 'accepted') { command.accepted = true } - if (value.status !== 'accepted' && !command.accepted) { errors.push(`No acceptance evidence for ${command.id}`) } + if (value.status === 'accepted') { + command.accepted = true + } + if (value.status !== 'accepted' && !command.accepted) { + errors.push(`No acceptance evidence for ${command.id}`) + } if (value.status === 'done') { doneOrder.delete(command.id); doneOrder.set(command.id, command.position) } @@ -108,8 +143,12 @@ export async function inspectStore(options = {}) { } let lastDonePosition = -1 for (const [id, position] of doneOrder) { - if (byId.get(id).status !== 'done') { continue } - if (position < lastDonePosition) { errors.push(`Committed execution differs from command-log order at ${id}`) } + if (byId.get(id).status !== 'done') { + continue + } + if (position < lastDonePosition) { + errors.push(`Committed execution differs from command-log order at ${id}`) + } lastDonePosition = position } let manifest @@ -117,31 +156,48 @@ export async function inspectStore(options = {}) { // Preserve optional existence-based compatibility; required integrity is explicit. const enabled = config.integrity || before[config.integrityFile] != null if (enabled) { - if (before[config.integrityFile] == null) { throw new Error(`Missing integrity manifest ${config.integrityFile}`) } + if (before[config.integrityFile] == null) { + throw new Error(`Missing integrity manifest ${config.integrityFile}`) + } await records(config.integrityFile, errors, 'integrity manifest') manifest = await loadIntegrityManifest(config.integrityFile) } - } catch (error) { errors.push(error.message) } + } + catch (error) { + errors.push(error.message) + } const parser = new Parser() const buffers = [], committed = [] let data, prefixValid = errors.length === 0 async function validateData(file) { const bytes = await fs.readFile(file) assertOdJsonTagFraming(bytes, file) - if (manifest) { verifyIntegrity(manifest, config.integrityFile, file, bytes, {required: true}) } + if (manifest) { + verifyIntegrity(manifest, config.integrityFile, file, bytes, {required: true}) + } return bytes } try { const bytes = await validateData(config.datafile) - if (!bytes.length) { throw new Error('Empty base dataset') } + if (!bytes.length) { + throw new Error('Empty base dataset') + } data = parser.parse(bytes); buffers.push(bytes) - } catch (error) { errors.push(error.message); prefixValid = false } + } + catch (error) { + errors.push(error.message); prefixValid = false + } for (const command of commands) { command.present = before[command.file] != null command.condition = command.present ? 'present' : 'missing' - if (!command.accepted) { errors.push(`Orphan command without acceptance: ${command.id}`) } + if (!command.accepted) { + errors.push(`Orphan command without acceptance: ${command.id}`) + } if (command.present) { - try { await validateData(command.file) } catch (error) { + try { + await validateData(command.file) + } + catch (error) { command.condition = 'corrupt'; command.problem = error.message } } @@ -149,13 +205,18 @@ export async function inspectStore(options = {}) { if (!command.present || command.condition === 'corrupt') { command.problem ||= `Missing changeset for committed command ${command.id}: ${command.file}` prefixValid = false - } else if (prefixValid) { + } + else if (prefixValid) { try { const bytes = await validateData(command.file) data = parser.parse(bytes); buffers.push(bytes); committed.push(command.id) - } catch (error) { command.problem = error.message; prefixValid = false } + } + catch (error) { + command.problem = error.message; prefixValid = false + } } - } else if (command.status === 'accepted' || command.status === 'active') { + } + else if (command.status === 'accepted' || command.status === 'active') { prefixValid = false } command.laterDatasets = commands.filter(later => later.position > command.position && later.accepted && before[later.file] != null).map(later => later.id) @@ -167,9 +228,15 @@ export async function inspectStore(options = {}) { errors.push(`Unexplained changeset: ${file}`) } } - for (const file of config.requiredFiles) { if (before[file] == null) { errors.push(`Missing required artifact: ${file}`) } } + for (const file of config.requiredFiles) { + if (before[file] == null) { + errors.push(`Missing required artifact: ${file}`) + } + } const after = await inventory(config) - if (JSON.stringify(before) !== JSON.stringify(after)) { errors.push('Store changed during inspection') } + if (JSON.stringify(before) !== JSON.stringify(after)) { + errors.push('Store changed during inspection') + } const pending = commands.filter(c => c.status === 'accepted' || c.status === 'active') const damaged = commands.filter(c => c.problem || (c.present && c.status !== 'done')) const ready = errors.length === 0 && pending.length === 0 && damaged.length === 0 diff --git a/src/store-ownership.mjs b/src/store-ownership.mjs index c510a63..07ebf1e 100644 --- a/src/store-ownership.mjs +++ b/src/store-ownership.mjs @@ -14,34 +14,58 @@ export async function acquireOwnership(directories, {ancestorToken, purpose, aud try { await fs.access(lock) let owner - try { owner = JSON.parse(await fs.readFile(path.join(lock,'owner.json'),'utf8')) } catch { /* An incomplete lock remains exclusive. */ } - if (!ancestorToken || owner?.token !== ancestorToken) { throw new Error(`Store is locked by unfinished parent operation: ${lock}`) } - } catch (error) { if (error.code !== 'ENOENT') { throw error } } + try { + owner = JSON.parse(await fs.readFile(path.join(lock,'owner.json'),'utf8')) + } + catch { + /* An incomplete lock remains exclusive. */ + } + if (!ancestorToken || owner?.token !== ancestorToken) { + throw new Error(`Store is locked by unfinished parent operation: ${lock}`) + } + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } + } + if (path.dirname(parent) === parent) { + break } - if (path.dirname(parent) === parent) { break } } } const token = randomUUID(), held = [] try { for (const directory of canonical) { const lock = path.join(directory, '.simplystore-lock') - try { await fs.mkdir(lock) } catch (error) { - if (error.code === 'EEXIST') { throw new Error(`Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.`) } + try { + await fs.mkdir(lock) + } + catch (error) { + if (error.code === 'EEXIST') { + throw new Error(`Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.`) + } throw error } held.push(lock) await publishFile(path.join(lock, 'owner.json'), JSON.stringify({token, purpose, auditDir, pid: process.pid, host: os.hostname(), started: new Date().toISOString()})) await syncAncestors(directory) } - } catch (error) { + } + catch (error) { // No store writes have begun, so only our own acquired locks can be released. - for (const lock of held.reverse()) { await fs.rm(lock, {recursive: true}); await syncDirectory(path.dirname(lock)) } + for (const lock of held.reverse()) { + await fs.rm(lock, {recursive: true}); await syncDirectory(path.dirname(lock)) + } throw error } return {token, directories: canonical, async release() { for (const lock of [...held].reverse()) { const owner = JSON.parse(await fs.readFile(path.join(lock, 'owner.json'), 'utf8')) - if (owner.token !== token) { throw new Error(`Ownership changed: ${lock}`) } + if (owner.token !== token) { + throw new Error(`Ownership changed: ${lock}`) + } await fs.unlink(path.join(lock, 'owner.json')) await fs.rmdir(lock) await syncDirectory(path.dirname(lock)) diff --git a/src/util.mjs b/src/util.mjs index 527557c..aa3bcea 100644 --- a/src/util.mjs +++ b/src/util.mjs @@ -2,17 +2,16 @@ import JSONTag from '@muze-nl/jsontag' import {appendRecord} from './storage.mjs' export function deepFreeze(obj) { - Object.freeze(obj) - Object.keys(obj).forEach(prop => { - if (typeof obj[prop] === 'object' && !Object.isFrozen(obj[prop])) { - deepFreeze(obj[prop]) - } - }) - return obj + Object.freeze(obj) + Object.keys(obj).forEach(prop => { + if (typeof obj[prop] === 'object' && !Object.isFrozen(obj[prop])) { + deepFreeze(obj[prop]) + } + }) + return obj } -export function isString(s) -{ +export function isString(s) { return typeof s === 'string' || s instanceof String } @@ -20,7 +19,8 @@ export function joinArgs(args) { return args = args.map(arg => { if (isString(arg)) { return arg - } else { + } + else { return JSONTag.stringify(arg) } }).join(' ') diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs index ff7c675..c3d6188 100644 --- a/test/admin-recovery.mjs +++ b/test/admin-recovery.mjs @@ -34,13 +34,15 @@ test('administrator recovers trailing missing/waiting commands in log order on a assert.ok((await fs.readFile(result.config.commandStatus)).subarray(0,before.length).equals(before)) assert.deepEqual((await fs.readFile(result.config.commandLog)).toString(),(await fs.readFile(store.commandLog)).toString()) }) -for (const state of ['accepted','done','unsafe']) { test(`later ${state} dataset blocks rerun, including zero-byte no-op`,async t=>{ - const {store}=await setup(t,[['A','done'],['B',state]]) - await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') - const plan=await planRecovery(store,{quiescent:true}) - assert.equal(plan.actionable,false) - assert.ok(plan.blocks.some(s=>s.includes('later accepted datasets'))) -}) } +for (const state of ['accepted','done','unsafe']) { + test(`later ${state} dataset blocks rerun, including zero-byte no-op`,async t=>{ + const {store}=await setup(t,[['A','done'],['B',state]]) + await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') + const plan=await planRecovery(store,{quiescent:true}) + assert.equal(plan.actionable,false) + assert.ok(plan.blocks.some(s=>s.includes('later accepted datasets'))) + }) +} test('new later file makes an approved plan stale before executing a handler',async t=>{ const {store,outputs}=await setup(t,[['A','done'],['B','accepted']]) const plan=await planRecovery(store,{quiescent:true}) @@ -61,7 +63,10 @@ test('cooperative ownership excludes competing writer and path aliases',async t= try { await fs.symlink(store.dir,path.join(outputs,'alias')) await assert.rejects(acquireOwnership([path.join(outputs,'alias')]),/locked/) - } finally { await owner.release() } + } + finally { + await owner.release() + } }) test('backup and restore validate saved bytes without executing commands',async t=>{ const {store,outputs}=await setup(t,[['A','done']]) @@ -214,7 +219,9 @@ test('verified A prefix permits only missing B, waiting C and missing D in log o const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted'],['C','accepted'],['D','accepted']]) const complete=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) const source={...complete.config,commandsFile:store.commandsFile,indexFile:store.indexFile} - for(const id of ['B','C','D']){ await fs.unlink(path.join(path.dirname(source.datafile),`data.${id}.jsontag`)) } + for(const id of ['B','C','D']){ + await fs.unlink(path.join(path.dirname(source.datafile),`data.${id}.jsontag`)) + } await fs.appendFile(source.commandStatus,JSONTag.stringify({command:'C',status:'accepted',code:202})+'\n') const prefix=await fs.readFile(path.join(path.dirname(source.datafile),'data.A.jsontag')) const plan=await planRecovery(source,{quiescent:true}) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs index 00bde87..2621429 100644 --- a/test/conversion-offsets.mjs +++ b/test/conversion-offsets.mjs @@ -127,14 +127,14 @@ test('conversion propagates rejection from custom finalization', async t => { }) test('conversion initializes existing-format logs and optional base integrity without overwriting a store',async t=>{ - const files=await fixture(t) - await run(process.execPath,[converter,files.input,files.output,'--integrity'],{timeout:10000}) - const {loadIntegrityManifest,verifyIntegrity}=await import('../src/integrity.mjs') - const integrity=path.join(files.dir,'data.integrity.jsontag') - assert.equal(verifyIntegrity(await loadIntegrityManifest(integrity),integrity,files.output,await fs.readFile(files.output),{required:true}),true) - assert.equal(await fs.readFile(path.join(files.dir,'command-log.jsontag'),'utf8'),'') - assert.equal(await fs.readFile(path.join(files.dir,'command-status.jsontag'),'utf8'),'') - const before=await fs.readFile(files.output) - await assert.rejects(convert(files),/existing store is not overwritten/) - assert.deepEqual(await fs.readFile(files.output),before) + const files=await fixture(t) + await run(process.execPath,[converter,files.input,files.output,'--integrity'],{timeout:10000}) + const {loadIntegrityManifest,verifyIntegrity}=await import('../src/integrity.mjs') + const integrity=path.join(files.dir,'data.integrity.jsontag') + assert.equal(verifyIntegrity(await loadIntegrityManifest(integrity),integrity,files.output,await fs.readFile(files.output),{required:true}),true) + assert.equal(await fs.readFile(path.join(files.dir,'command-log.jsontag'),'utf8'),'') + assert.equal(await fs.readFile(path.join(files.dir,'command-status.jsontag'),'utf8'),'') + const before=await fs.readFile(files.output) + await assert.rejects(convert(files),/existing store is not overwritten/) + assert.deepEqual(await fs.readFile(files.output),before) }) diff --git a/test/durability-crash.mjs b/test/durability-crash.mjs index be13b1d..598f10a 100644 --- a/test/durability-crash.mjs +++ b/test/durability-crash.mjs @@ -26,7 +26,8 @@ async function postCommandExpectingCrash(port, command) { try { const response = await postCommand(port, command) assert.equal(response.status, 202) - } catch (error) { + } + catch (error) { assert.match(error.message, /fetch failed|terminated|socket|other side closed|aborted/i) } } @@ -84,27 +85,29 @@ test('fault points are inert outside test environment', async () => { }) for (const faultPointName of [ - 'after-command-log-before-accepted-status', - 'after-command-accepted-status-before-response', - 'after-active-status-before-command-worker', - 'before-command-changeset-write', - 'after-command-changeset-write', - 'before-command-done-status' -]) { test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { - const fixture=await makeServerFixture(t) - const port=await getOpenPort() - const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:faultPointName}) - await waitForServer(first.child,first.getOutput,port) - await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) - assert.equal((await waitForExit(first.child)).signal,'SIGKILL') - const before=await fs.readFile(fixture.commandStatus) - await unlockStoppedFixture(t,fixture) - const second=startServer(t,fixture,{port}) - assert.equal((await waitForExit(second.child)).code,1) - assert.match(second.getOutput(),/Administrative recovery required/) - assert.deepEqual(await fs.readFile(fixture.commandStatus),before) - assert.equal((await readCommandLogRecords(fixture))[0].id,'A') -}) } + 'after-command-log-before-accepted-status', + 'after-command-accepted-status-before-response', + 'after-active-status-before-command-worker', + 'before-command-changeset-write', + 'after-command-changeset-write', + 'before-command-done-status' +]) { + test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { + const fixture=await makeServerFixture(t) + const port=await getOpenPort() + const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:faultPointName}) + await waitForServer(first.child,first.getOutput,port) + await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) + assert.equal((await waitForExit(first.child)).signal,'SIGKILL') + const before=await fs.readFile(fixture.commandStatus) + await unlockStoppedFixture(t,fixture) + const second=startServer(t,fixture,{port}) + assert.equal((await waitForExit(second.child)).code,1) + assert.match(second.getOutput(),/Administrative recovery required/) + assert.deepEqual(await fs.readFile(fixture.commandStatus),before) + assert.equal((await readCommandLogRecords(fixture))[0].id,'A') + }) +} test('crash after done status but before query update recovers committed state without replay', async t => { const fixture = await makeServerFixture(t) @@ -145,15 +148,15 @@ test('crash after done status but before query update recovers committed state w }) test('crash ownership cannot be stolen automatically',async t=>{ - const fixture=await makeServerFixture(t),port=await getOpenPort() - const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:'before-command-done-status'}) - await waitForServer(first.child,first.getOutput,port) - await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) - await waitForExit(first.child) - const second=startServer(t,fixture,{port}) - assert.equal((await waitForExit(second.child)).code,1) - assert.match(second.getOutput(),/Store is locked/) - assert.equal((await readCommandStatusRecords(fixture)).filter(s=>s.status==='active').length,1) + const fixture=await makeServerFixture(t),port=await getOpenPort() + const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:'before-command-done-status'}) + await waitForServer(first.child,first.getOutput,port) + await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) + await waitForExit(first.child) + const second=startServer(t,fixture,{port}) + assert.equal((await waitForExit(second.child)).code,1) + assert.match(second.getOutput(),/Store is locked/) + assert.equal((await readCommandStatusRecords(fixture)).filter(s=>s.status==='active').length,1) }) test('hanging command times out unsafe and later accepted command commits', async t => { diff --git a/test/durability-helpers.mjs b/test/durability-helpers.mjs index 0eb436a..be32b36 100644 --- a/test/durability-helpers.mjs +++ b/test/durability-helpers.mjs @@ -115,7 +115,9 @@ export async function waitForServer(child, getOutput, port) { reject(new Error(`Server exited before startup (${code || signal}):\n${getOutput()}`)) } const checkReady = () => { - if (finished) { return } + if (finished) { + return + } if (getOutput().includes(`SimplyStore listening on port ${port}`)) { finished = true clearTimeout(timeout) @@ -192,7 +194,8 @@ async function readJsonTagLines(file) { let text try { text = await fs.readFile(file, 'utf8') - } catch (error) { + } + catch (error) { if (error.code === 'ENOENT') { return [] } @@ -240,11 +243,11 @@ export async function reconstructCommittedPersonNames(fixture) { // Explicit administrator action in disposable fixtures whose writer has exited. export async function unlockStoppedFixture(t, fixture) { - const {releaseOfflineLocks} = await import('../src/admin-recovery.mjs') - const {publishFile} = await import('../src/storage.mjs') - const audit = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-unlock-audit-')) - t.after(() => fs.rm(audit, {recursive:true,force:true})) - const release = await releaseOfflineLocks(fixture, {operator:'test administrator',reason:'Fixture writer has exited; inspect before startup',confirmedStopped:true}) - await publishFile(path.join(audit,'release.json'),JSON.stringify(release)) - await release.finish() + const {releaseOfflineLocks} = await import('../src/admin-recovery.mjs') + const {publishFile} = await import('../src/storage.mjs') + const audit = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-unlock-audit-')) + t.after(() => fs.rm(audit, {recursive:true,force:true})) + const release = await releaseOfflineLocks(fixture, {operator:'test administrator',reason:'Fixture writer has exited; inspect before startup',confirmedStopped:true}) + await publishFile(path.join(audit,'release.json'),JSON.stringify(release)) + await release.finish() } diff --git a/test/durability-latency.mjs b/test/durability-latency.mjs index 0c2e331..2284432 100644 --- a/test/durability-latency.mjs +++ b/test/durability-latency.mjs @@ -14,9 +14,15 @@ test('measure local startup, acceptance, and completion latency',async t=>{ const start=performance.now(),id=`latency-${i}` assert.equal((await postCommand(port,{id,name:'addPerson',value:{name:id}})).status,202) accepted.push(performance.now()-start) - for(;;){ const status=await getCommandStatus(port,id);if(status.status==='done'){ break; }assert.equal(['accepted','active'].includes(status.status),true);await new Promise(r=>setTimeout(r,2)) } + for(;;){ + const status=await getCommandStatus(port,id);if(status.status==='done'){ + break; + }assert.equal(['accepted','active'].includes(status.status),true);await new Promise(r=>setTimeout(r,2)) + } completed.push(performance.now()-start) } - const stats=values=>{ const v=[...values].sort((a,b)=>a-b);return {p50Ms:v[Math.floor(v.length*.5)],p95Ms:v[Math.floor(v.length*.95)],maxMs:v.at(-1)} } + const stats=values=>{ + const v=[...values].sort((a,b)=>a-b);return {p50Ms:v[Math.floor(v.length*.5)],p95Ms:v[Math.floor(v.length*.95)],maxMs:v.at(-1)} + } t.diagnostic(JSON.stringify({samples:30,startupMs,acceptance:stats(accepted),completion:stats(completed),filesystem:execFileSync('stat',['-f','-c','%T',store.dir],{encoding:'utf8'}).trim(),node:process.version,notes:'Single localhost client; completion includes a fresh worker and polling; no production performance claim'})) }) diff --git a/test/durability-startup.mjs b/test/durability-startup.mjs index c4aaf01..89e9055 100644 --- a/test/durability-startup.mjs +++ b/test/durability-startup.mjs @@ -253,16 +253,18 @@ test('durable command log replays only accepted commands', async t => { assert.equal(JSONTag.parse(commands[0].command).name, 'addPerson') }) -for (const attempt of [1, 2]) { test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { - const fixture = await makeFixture(t) - const file = path.join(fixture.dir, 'status.jsontag') - await writeJsonTagLines(file, [{command:'A',status:'active',attempt}]) - const before = await fs.readFile(file) - const status = loadCommandStatus(file) - await assert.rejects(recoverActiveCommands(status,file,{maxCrashAttempts:2}), /administrator assessment/) - assert.equal(status.get('A').status,'active') - assert.deepEqual(await fs.readFile(file),before) -}) } +for (const attempt of [1, 2]) { + test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { + const fixture = await makeFixture(t) + const file = path.join(fixture.dir, 'status.jsontag') + await writeJsonTagLines(file, [{command:'A',status:'active',attempt}]) + const before = await fs.readFile(file) + const status = loadCommandStatus(file) + await assert.rejects(recoverActiveCommands(status,file,{maxCrashAttempts:2}), /administrator assessment/) + assert.equal(status.get('A').status,'active') + assert.deepEqual(await fs.readFile(file),before) + }) +} test('malformed durable status record refuses recovery with explicit integrity error', async t => { const fixture = await makeFixture(t) @@ -272,7 +274,8 @@ test('malformed durable status record refuses recovery with explicit integrity e let error try { loadCommandStatus(statusFile) - } catch (caught) { + } + catch (caught) { error = caught } @@ -293,7 +296,8 @@ test('structurally invalid durable status record refuses recovery', async t => { let error try { loadCommandStatus(statusFile) - } catch (caught) { + } + catch (caught) { error = caught } @@ -313,7 +317,8 @@ test('malformed durable command log record refuses recovery with explicit integr let error try { loadCommandLog(status, commandLog) - } catch (caught) { + } + catch (caught) { error = caught } @@ -335,7 +340,8 @@ test('structurally invalid durable command log record refuses recovery', async t let error try { loadCommandLog(status, commandLog) - } catch (caught) { + } + catch (caught) { error = caught } @@ -352,7 +358,8 @@ test('OD-JSONTag framing validation catches truncated lazy records', () => { let error try { assertOdJsonTagFraming(buffer, 'data.truncated.jsontag', 'changeset OD-JSONTag data') - } catch (caught) { + } + catch (caught) { error = caught } diff --git a/test/power-loss-baseline.mjs b/test/power-loss-baseline.mjs index 78f12e9..6a88db0 100644 --- a/test/power-loss-baseline.mjs +++ b/test/power-loss-baseline.mjs @@ -55,18 +55,29 @@ for (const [name,op,file,contains] of [ ['changeset sync','sync','data.A.jsontag.'], ['changeset rename','rename','data.A.jsontag'], ['done sync','datasync','command-status.jsontag','"status":"done"'] -]) { test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`,async t=>{ - const store=await fixture(t),server=await ready(t,store,{fault:{op,file,contains,action:'error'}}) - try { await submit(store,server.port,command('A')) } catch(error){ assert.match(error.message,/fetch failed|terminated|socket/) } - assert.equal((await waitForExit(server.child)).code,1) - assert.match(server.getOutput(),/Storage outcome uncertain/) - assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) - const status=await readCommandStatusRecords(store) - // Failed sync can leave a complete but uncertain done record; never append failed after it. - const done=status.findIndex(s=>s.status==='done') - if(done>=0){ assert.equal(status.slice(done+1).some(s=>s.status==='failed'),false) } - else { assert.equal(status.some(s=>s.status==='done'),false) } -}) } +]) { + test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`,async t=>{ + const store=await fixture(t),server=await ready(t,store,{fault:{op,file,contains,action:'error'}}) + try { + await submit(store,server.port,command('A')) + } + catch(error){ + assert.match(error.message,/fetch failed|terminated|socket/) + } + assert.equal((await waitForExit(server.child)).code,1) + assert.match(server.getOutput(),/Storage outcome uncertain/) + assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) + const status=await readCommandStatusRecords(store) + // Failed sync can leave a complete but uncertain done record; never append failed after it. + const done=status.findIndex(s=>s.status==='done') + if(done>=0){ + assert.equal(status.slice(done+1).some(s=>s.status==='failed'),false) + } + else { + assert.equal(status.some(s=>s.status==='done'),false) + } + }) +} test('PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip',async t=>{ const store=await fixture(t,{commandsSource:`export default {addPerson(data,command,request,meta){ if(request!==undefined)throw new Error('unlogged input leaked') @@ -84,37 +95,43 @@ parentPort.on('message',async task=>{if('request' in task)throw new Error('reque assert.equal(JSONTag.stringify((await readCommandLogRecords(store))[0]),JSONTag.stringify(input)) assert.equal((await queryPersons(server.port))[0].name,input.value.name) }) -for(const duplicate of [false,true]){ test(`PL16 delayed append cannot be overtaken${duplicate?' by duplicate ID':''}`,async t=>{ - const store=await fixture(t),server=await ready(t,store,{fault:{op:'datasync:after',file:'command-log.jsontag',contains:'"id":"A"',action:'pause'}}) - const a=submit(store,server.port,command('A')) - await eventually(async()=> (await rows(store.trace)).some(e=>e.op==='injected'),'A append gate') - const b=submit(store,server.port,command(duplicate?'A':'B')) - await new Promise(resolve=>setTimeout(resolve,30)) - assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),['A']) - await fs.writeFile(store.release,'') - assert.equal((await a).http,202);assert.ok([200,202].includes((await b).http)) - await completion(store,server.port,duplicate?'A':'B') - const expected=duplicate?['A']:['A','B'] - assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),expected) - assert.deepEqual((await readCommandStatusRecords(store)).filter(s=>s.status==='active').map(s=>s.command),expected) - assert.deepEqual((await queryPersons(server.port)).map(p=>p.name),expected) -}) } -for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.jsontag']){ test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`,async t=>{ - const store=await fixture(t),server=await ready(t,store);await commit(store,server) - await crash(server.child);await unlockStoppedFixture(t,store) - await fs.unlink(path.join(store.dir,artifact)) - const before=await fs.readdir(store.dir) - const restarted=await launch(t,store,await getOpenPort()) - assert.equal((await waitForExit(restarted.child)).code,1) - assert.match(restarted.getOutput(),/Administrative recovery required/) - assert.deepEqual((await fs.readdir(store.dir)).sort(),before.sort()) -}) } +for(const duplicate of [false,true]){ + test(`PL16 delayed append cannot be overtaken${duplicate?' by duplicate ID':''}`,async t=>{ + const store=await fixture(t),server=await ready(t,store,{fault:{op:'datasync:after',file:'command-log.jsontag',contains:'"id":"A"',action:'pause'}}) + const a=submit(store,server.port,command('A')) + await eventually(async()=> (await rows(store.trace)).some(e=>e.op==='injected'),'A append gate') + const b=submit(store,server.port,command(duplicate?'A':'B')) + await new Promise(resolve=>setTimeout(resolve,30)) + assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),['A']) + await fs.writeFile(store.release,'') + assert.equal((await a).http,202);assert.ok([200,202].includes((await b).http)) + await completion(store,server.port,duplicate?'A':'B') + const expected=duplicate?['A']:['A','B'] + assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),expected) + assert.deepEqual((await readCommandStatusRecords(store)).filter(s=>s.status==='active').map(s=>s.command),expected) + assert.deepEqual((await queryPersons(server.port)).map(p=>p.name),expected) + }) +} +for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.jsontag']){ + test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`,async t=>{ + const store=await fixture(t),server=await ready(t,store);await commit(store,server) + await crash(server.child);await unlockStoppedFixture(t,store) + await fs.unlink(path.join(store.dir,artifact)) + const before=await fs.readdir(store.dir) + const restarted=await launch(t,store,await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code,1) + assert.match(restarted.getOutput(),/Administrative recovery required/) + assert.deepEqual((await fs.readdir(store.dir)).sort(),before.sort()) + }) +} test('PL12/17 startup preserves uncertain active work without repeating external effects',async t=>{ const store=await fixture(t,{commandsSource:`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify('/tmp/placeholder-effect')},'') ;data.persons.push(command.value)}}`}) const witness=path.join(store.audit,'effects') await fs.writeFile(store.commandsFile,`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify(witness)},command.id+'\\n');data.persons.push(command.value)}}`) const server=await ready(t,store,{fault:{op:'write',file:'command-status.jsontag',contains:'"status":"done"',action:'error'}}) - await submit(store,server.port,command('A'));await waitForExit(server.child).catch(error=>{ throw new Error(server.getOutput(),{cause:error}) }) + await submit(store,server.port,command('A'));await waitForExit(server.child).catch(error=>{ + throw new Error(server.getOutput(),{cause:error}) + }) const before=await fs.readFile(store.commandStatus) await unlockStoppedFixture(t,store) const restarted=await launch(t,store,await getOpenPort()) @@ -123,22 +140,26 @@ test('PL12/17 startup preserves uncertain active work without repeating external assert.equal(await fs.readFile(witness,'utf8'),'A\n') }) -for(const operation of ['sync','close:after']){ test(`directory ${operation} failure after rename prevents done`,async t=>{ - const store=await fixture(t) - const server=await ready(t,store,{fault:{op:operation,file:path.basename(store.dir),action:'error',after:{op:'rename',file:'data.A.jsontag'}}}) - await submit(store,server.port,command('A')) - assert.equal((await waitForExit(server.child)).code,1) - assert.ok((await rows(store.trace)).some(e=>e.op==='injected'&&e.file===store.dir)) - assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) -}) } -for(const operation of ['open','write','close:after']){ test(`changeset ${operation} error is not masked or acknowledged`,async t=>{ - const store=await fixture(t) - const server=await ready(t,store,{fault:{op:operation,file:'data.A.jsontag.',action:'error',code:operation==='write'?'ENOSPC':'EIO'}}) - await submit(store,server.port,command('A')) - assert.equal((await waitForExit(server.child)).code,1) - assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) - assert.match(server.getOutput(),/Storage outcome uncertain/) -}) } +for(const operation of ['sync','close:after']){ + test(`directory ${operation} failure after rename prevents done`,async t=>{ + const store=await fixture(t) + const server=await ready(t,store,{fault:{op:operation,file:path.basename(store.dir),action:'error',after:{op:'rename',file:'data.A.jsontag'}}}) + await submit(store,server.port,command('A')) + assert.equal((await waitForExit(server.child)).code,1) + assert.ok((await rows(store.trace)).some(e=>e.op==='injected'&&e.file===store.dir)) + assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) + }) +} +for(const operation of ['open','write','close:after']){ + test(`changeset ${operation} error is not masked or acknowledged`,async t=>{ + const store=await fixture(t) + const server=await ready(t,store,{fault:{op:operation,file:'data.A.jsontag.',action:'error',code:operation==='write'?'ENOSPC':'EIO'}}) + await submit(store,server.port,command('A')) + assert.equal((await waitForExit(server.child)).code,1) + assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) + assert.match(server.getOutput(),/Storage outcome uncertain/) + }) +} test('short command-log append preserves its entire JSONTag record',async t=>{ const store=await fixture(t) const server=await ready(t,store,{fault:{op:'write',file:'command-log.jsontag',contains:'"id":"A"',action:'short-write'}}) diff --git a/test/power-loss-helpers.mjs b/test/power-loss-helpers.mjs index f53d201..0678925 100644 --- a/test/power-loss-helpers.mjs +++ b/test/power-loss-helpers.mjs @@ -36,7 +36,12 @@ export async function submit(store, port, command, query = '') { }) const text = await response.text() let body - try { body = JSON.parse(text) } catch { body = {raw: text} } + try { + body = JSON.parse(text) + } + catch { + body = {raw: text} + } await observe(store, {event: 'response', id: command.id, http: response.status, body}) return {http: response.status, body} } @@ -45,7 +50,9 @@ export async function eventually(fn, description, timeout = 6000) { const end = Date.now() + timeout while (Date.now() < end) { const result = await fn() - if (result) { return result } + if (result) { + return result + } await new Promise(resolve => setTimeout(resolve, 10)) } assert.fail(`Timed out: ${description}`) @@ -77,8 +84,12 @@ export async function launch(t, store, port, options = {}) { stdio: ['ignore', 'pipe', 'pipe'] }) let output = '' - child.stdout.on('data', data => { output += data }) - child.stderr.on('data', data => { output += data }) + child.stdout.on('data', data => { + output += data + }) + child.stderr.on('data', data => { + output += data + }) t.after(() => stopServer(child)) return {child, getOutput: () => output} } @@ -107,7 +118,9 @@ export async function loseUnpublishedName(store, image, original) { assert.ok(publication >= 0, `trace contains publication of ${original}`) const durable = trace.slice(publication + 1).some(e => e.file === path.dirname(original) && e.op === 'sync') - if (!durable) { await fs.rm(path.join(image.dir, path.basename(original))) } + if (!durable) { + await fs.rm(path.join(image.dir, path.basename(original))) + } return !durable } diff --git a/test/power-loss-preload.mjs b/test/power-loss-preload.mjs index f76b813..8f16071 100644 --- a/test/power-loss-preload.mjs +++ b/test/power-loss-preload.mjs @@ -20,8 +20,10 @@ if (configFile) { const scoped = file => typeof file === 'string' && (file === root || file.startsWith(root + path.sep)) function record(op, file, extra = {}) { - if (scoped(file)) { raw.appendFileSync(config.trace, - JSON.stringify({op, file, threadId, ...extra}) + '\n') } + if (scoped(file)) { + raw.appendFileSync(config.trace, + JSON.stringify({op, file, threadId, ...extra}) + '\n') + } } function snapshot(file) { return raw.statSync(file).isDirectory() ? {} : {bytes: raw.readFileSync(file).toString('base64')} @@ -30,19 +32,27 @@ if (configFile) { const rule = config.fault if (injected || !rule || rule.op !== op || !scoped(file) || !path.basename(file).startsWith(rule.file) || - (rule.contains && !content.includes(rule.contains))) { return false } - if (rule.after && !raw.readFileSync(config.trace,'utf8').split('\n').filter(Boolean).map(JSON.parse).some(event => event.op === rule.after.op && path.basename(event.file) === rule.after.file)) { return false } + (rule.contains && !content.includes(rule.contains))) { + return false + } + if (rule.after && !raw.readFileSync(config.trace,'utf8').split('\n').filter(Boolean).map(JSON.parse).some(event => event.op === rule.after.op && path.basename(event.file) === rule.after.file)) { + return false + } injected = true record('injected', file, {operation: op, action: rule.action}) if (rule.action === 'pause') { const deadline = Date.now() + 15000 while (!raw.existsSync(config.release)) { - if (Date.now() > deadline) { throw new Error('Baseline gate was not released') } + if (Date.now() > deadline) { + throw new Error('Baseline gate was not released') + } await new Promise(resolve => setTimeout(resolve, 5)) } - } else if (rule.action === 'short-write') { + } + else if (rule.action === 'short-write') { return true - } else { + } + else { throw Object.assign(new Error(`Injected ${rule.code || 'EIO'} at ${op}`), {code: rule.code || 'EIO'}) } return false @@ -59,16 +69,24 @@ if (configFile) { const existed = raw.existsSync(absolute) await fault('open', absolute) const handle = await open(file, ...args) - if (!scoped(absolute)) { return handle } + if (!scoped(absolute)) { + return handle + } record('open', absolute, {existed, flags: args[0]}) let content = '' for (const method of ['appendFile', 'write', 'datasync', 'sync', 'close']) { const original = handle[method].bind(handle) handle[method] = async (...params) => { - if (method === 'appendFile') { content = String(params[0]) } - if (method === 'write') { content = String(params[0].subarray(params[1], params[1]+params[2])) } + if (method === 'appendFile') { + content = String(params[0]) + } + if (method === 'write') { + content = String(params[0].subarray(params[1], params[1]+params[2])) + } const short = await fault(method, absolute, content) - if (short && method === 'write') { params[2] = Math.max(1, Math.floor(params[2]/2)) } + if (short && method === 'write') { + params[2] = Math.max(1, Math.floor(params[2]/2)) + } const result = await original(...params) record(method, absolute, method === 'datasync' || method === 'sync' ? snapshot(absolute) : method === 'write' ? {bytesWritten:result.bytesWritten} : {}) @@ -102,7 +120,9 @@ if (configFile) { args[2] = Math.max(1, Math.floor(args[2] / 2)) } raw.write(fd, ...args, (error, bytes, buffer) => { - if (!error) { record('write', file, {bytesWritten: bytes}) } + if (!error) { + record('write', file, {bytesWritten: bytes}) + } callback(error, bytes, buffer) }) }).catch(callback) @@ -110,7 +130,9 @@ if (configFile) { fs.fsync = function(fd, callback) { const file = handles.get(fd) void fault('fsync', file).then(() => raw.fsync(fd, error => { - if (!error && scoped(file)) { record('sync', file, snapshot(file)) } + if (!error && scoped(file)) { + record('sync', file, snapshot(file)) + } callback(error) })).catch(callback) } @@ -127,7 +149,9 @@ if (configFile) { fs.rename = function(from, to, callback) { const file = path.resolve(String(to)) void fault('rename', file).then(() => raw.rename(from, to, error => { - if (!error) { record('rename', file, {from: path.resolve(String(from))}) } + if (!error) { + record('rename', file, {from: path.resolve(String(from))}) + } callback(error) })).catch(callback) } diff --git a/test/storage.mjs b/test/storage.mjs index 00b4e07..47e2bbc 100644 --- a/test/storage.mjs +++ b/test/storage.mjs @@ -8,15 +8,25 @@ import {writeAll,publishFile,publishFileSync,appendRecord,serialWriter} from '.. test('complete-write loop preserves bytes through repeated short writes and rejects zero progress', async()=>{ const bytes = Buffer.from('cafΓ© ζΌ’ε­— πŸ˜€'), written=[] - await writeAll({async write(buffer,offset,length){ const n=Math.min(3,length);written.push(buffer.subarray(offset,offset+n));return {bytesWritten:n} }},bytes) + await writeAll({async write(buffer,offset,length){ + const n=Math.min(3,length);written.push(buffer.subarray(offset,offset+n));return {bytesWritten:n} + }},bytes) assert.deepEqual(Buffer.concat(written),bytes) - for (const count of [0,-1,NaN,1000]) { await assert.rejects(writeAll({async write(){ return {bytesWritten:count} }},bytes), /progress write/) } + for (const count of [0,-1,NaN,1000]) { + await assert.rejects(writeAll({async write(){ + return {bytesWritten:count} + }},bytes), /progress write/) + } }) test('serialized writer preserves order and stops after an uncertain operation',async()=>{ const run=serialWriter(), seen=[] let release - const gate=new Promise(resolve=>{ release=resolve }) - const first=run(async()=>{ seen.push('A');await gate;throw new Error('uncertain') }) + const gate=new Promise(resolve=>{ + release=resolve + }) + const first=run(async()=>{ + seen.push('A');await gate;throw new Error('uncertain') + }) const second=run(async()=>seen.push('B')) release();await assert.rejects(first,/uncertain/);await assert.rejects(second,/uncertain/) assert.deepEqual(seen,['A']) From 529272489ca03b50271c3555987eb96dd2279ed5 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:07:19 +0200 Subject: [PATCH 28/37] Record JAQT readability refactor request --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 13 +++++++++++++ AGENTS.md | 1 + 2 files changed, 14 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index 3bc6329..db3719d 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -120,3 +120,16 @@ The maintainer clarified the block layout: Expand block bodies, including empty blocks and callbacks, onto separate lines; place `else`, `catch`, and `finally` after the closing-brace line. Object literals and destructuring braces are not statement blocks. This remains formatting-only. + +## Readability and JAQT review β€” 2026-09-19 + +The maintainer identified the dense startup diagnostic expression, requested a +read-only JAQT feasibility check, and approved the example with named diagnostic +groups using the installed JAQT 0.10.6. They then instructed: + +> That is a lot more readable. Please check the rest of the code to see if jaqt could improve it, if so, update it. + +Review the remaining code for clearer query expressions and named intermediate +results; use JAQT where it improves readability. Preserve behavior, ordering and +public result types. This authorizes the refactor within the still-active cycle; +it does not constitute cycle acceptance. diff --git a/AGENTS.md b/AGENTS.md index 8317399..e5c5760 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,6 +60,7 @@ Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifa - Always use braces for control-flow bodies, including single-statement `if`, `else`, and loop bodies. Start block contents on the line after `{`, and put the closing `}` on its own line. This includes empty blocks and callbacks; `else`, `catch`, and `finally` start on the next line after `}`. +- Prefer named intermediate results over dense chains of operations. Use the existing JAQT library where it makes data selection and projection clearer; keep straightforward operations simple. - Treat `master` as the authoritative branch unless the human states otherwise. - For ordinary repository-changing Spiral cycle work, create one dedicated branch from the authoritative branch, normally `spiral/CYC-###-short-goal`. - For new Spiral artifacts after CYC-018, use distributed-safe IDs allocated by `node .spiral-core/bin/spiral.mjs allocate ` instead of scanning for the next legacy number. Existing `SRC-001` / `CYC-017` style IDs remain valid historical artifacts and must not be renamed. From 49fa48c123d0035df087b7a9d4c1460da92b85fa Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:10:36 +0200 Subject: [PATCH 29/37] Clarify recovery queries and diagnostics with JAQT --- .../implementations/IMP-20260919-TTZ7C-26.md | 22 ++++ .../implementations/IMP-20260919-TTZ7C-26.ttl | 10 +- src/admin-recovery.mjs | 111 +++++++++++++++--- src/server.mjs | 32 ++++- src/store-inspection.mjs | 29 ++++- test/admin-recovery.mjs | 50 ++++++++ 6 files changed, 220 insertions(+), 34 deletions(-) diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md index 0f479a1..c2f37db 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.md +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -61,3 +61,25 @@ The final acceptance examples additionally exercise exactly the verified A / mis B / waiting C / missing D suffix, disclosed loss from an older backup, and identical versus conflicting historical duplicate IDs. These add verification coverage without changing the committed runtime or VM exercise. + +## JAQT readability refactor during review + +The maintainer approved using JAQT and named intermediate results where they +clarify the cycle code. Reviewed source/scripts for data-selection expressions; +the useful changes are in server startup diagnostics, inspection's later-dataset +and readiness queries, and administrator recovery/backup comparisons. Existing +simple array operations, sequential I/O and mutation loops remain direct code. + +JAQT 0.10.6 is already a dependency. Query patterns operate on inspected command +records with established string statuses/IDs and boolean flags; truthy diagnostic +fields use Boolean predicates. Query order remains command-log order. Public +plan/report collections and later-dataset IDs are materialized as ordinary arrays +so JAQT proxies do not change cloning or public data contracts. Optional report +properties keep their prior undefined/omitted behavior. + +The backup-coverage predicate now names command/status matching separately from +surviving-byte comparison. Named stages also replace dense canonical-path and +code-hash expressions where JAQT would not improve the operation. Tests cover +backup differences and unchanged plain-data output alongside the existing suite. +This is a behavior-preserving refactor; persistence boundaries, formats, recovery +eligibility and administrator authority are unchanged. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl index 6e88331..128e2d4 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -29,10 +29,6 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-011 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260917-TTZ7C-11 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260917-TTZ7C-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-22 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "049c45642fe60146bd511bb87abb4b032f8df36a" ] ; - sd:implementationChangeKind sd:MixedChange . + sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "529272489ca03b50271c3555987eb96dd2279ed5" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "529272489ca03b50271c3555987eb96dd2279ed5" ] ; + sd:implementationChangeKind sd:BehaviorPreservingChange . diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs index ed7aa2c..e308a99 100644 --- a/src/admin-recovery.mjs +++ b/src/admin-recovery.mjs @@ -3,6 +3,7 @@ import fs from 'node:fs/promises' import path from 'node:path' import {fileURLToPath} from 'node:url' import JSONTag from '@muze-nl/jsontag' +import {from, _, anyOf, not} from '@muze-nl/jaqt' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import {inspectStore, inventory, hash, mutableDirectories, storePaths} from './store-inspection.mjs' @@ -21,7 +22,12 @@ function deployment(options) { commandWorker:path.resolve(options.commandWorker || defaultWorker)} } async function codeHashes(code) { - return Object.fromEntries(await Promise.all(Object.entries(code).map(async ([key,file]) => [key,hash(await fs.readFile(file))]))) + const entries = Object.entries(code) + const hashes = await Promise.all(entries.map(async ([key, file]) => { + const bytes = await fs.readFile(file) + return [key, hash(bytes)] + })) + return Object.fromEntries(hashes) } async function canonicalNew(file) { const absolute = path.resolve(file) @@ -54,23 +60,50 @@ async function assertFresh(plan) { export async function planRecovery(options, {quiescent = false} = {}) { const report = await inspectStore(options) const code = deployment(options) - const candidates = report.commands.filter(c => ['accepted','active'].includes(c.status) || (c.status === 'done' && !c.present)) + const commands = from(report.commands) + const candidates = commands.where(anyOf( + {status: anyOf('accepted', 'active')}, + {status: 'done', present: false} + )) + const uncertain = commands.where(anyOf( + {condition: 'corrupt'}, + {present: Boolean, status: not('done')} + )) + const blockedCandidates = candidates.where({ + laterDatasets: datasets => datasets.length > 0 + }) + const blocks = [...report.errors] - for (const command of report.commands) { - if (command.condition === 'corrupt' || (command.present && command.status !== 'done')) { - blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) - } + for (const command of uncertain) { + blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) } - for (const command of candidates) { - if (command.laterDatasets.length) { - blocks.push(`${command.id}: later accepted datasets exist: ${command.laterDatasets.join(',')}`) - } + for (const command of blockedCandidates) { + const laterIds = command.laterDatasets.join(',') + blocks.push(`${command.id}: later accepted datasets exist: ${laterIds}`) + } + + const rerun = [...candidates.select(_.id)] + const summaries = commands.select(record => { + const summary = {...record} + delete summary.command + return summary + }) + const selectedCodeHashes = await codeHashes(code) + return { + kind: 'simplystore-recovery-plan', + config: report.config, + code, + codeHashes: selectedCodeHashes, + files: report.files, + fingerprint: report.fingerprint, + actionable: quiescent && blocks.length === 0, + blocks, + rerun, + committed: report.committed, + commands: [...summaries], + warnings: report.warnings, + attestation: 'Approval asserts source quiescence, complete authoritative history, complete logged command inputs, original command meaning, and assessed external effects.' } - return {kind:'simplystore-recovery-plan', config:report.config, code, codeHashes:await codeHashes(code), - files:report.files, fingerprint:report.fingerprint, actionable:quiescent && blocks.length === 0, - blocks, rerun:candidates.map(c => c.id), committed:report.committed, - commands:report.commands.map(record => Object.fromEntries(Object.entries(record).filter(([key])=>key!=='command'))), warnings:report.warnings, - attestation:'Approval asserts source quiescence, complete authoritative history, complete logged command inputs, original command meaning, and assessed external effects.'} } export async function writePlan(file, plan) { const target = await canonicalNew(file) @@ -191,7 +224,10 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper } const complete = await inspectStore(config) if (!complete.ready) { - throw new Error('Recovered candidate is not ready: '+JSON.stringify({errors:complete.errors,commands:complete.commands.filter(c=>c.problem)})) + const problems = from(complete.commands).where({problem: Boolean}) + const diagnostic = {errors: complete.errors, commands: [...problems]} + const details = JSON.stringify(diagnostic) + throw new Error(`Recovered candidate is not ready: ${details}`) } const report = {kind:'simplystore-recovery-complete',planHash,config,code:plan.code,codeHashes:plan.codeHashes,files:complete.files, fingerprint:complete.fingerprint,committed:complete.committed,operator,reason,warnings:complete.warnings} @@ -238,6 +274,38 @@ export async function backupStore(options, {to, quiescent = false} = {}) { } } +function backupCoverage(retained, restored) { + const originalBase = retained.files[retained.config.datafile] + const backupBase = restored.files[restored.config.datafile] + const sameBase = originalBase === backupBase + const backupCommands = from(restored.commands) + + function isCovered(command) { + if (!sameBase) { + return false + } + const matching = backupCommands.where({ + id: command.id, + line: command.line, + status: command.status + }) + if (command.status !== 'done' || !command.present) { + return matching.length > 0 + } + return matching.some(backupCommand => { + const originalDigest = retained.files[command.file] + const backupDigest = restored.files[backupCommand.file] + return originalDigest === backupDigest + }) + } + + const missing = from(retained.commands) + .where({accepted: true}) + .where(not(isCovered)) + .select(_.id) + return {sameBase, missingFromBackup: [...missing]} +} + export async function restoreBackup(backupDirectory, {to, auditDir, source, sourceQuiescent=false} = {}) { const backup = await fs.realpath(backupDirectory) const manifest = JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) @@ -285,8 +353,9 @@ export async function restoreBackup(backupDirectory, {to, auditDir, source, sour let missingFromBackup = null, sameBase = null if (source) { const retained = await inspectStore(source) - sameBase=retained.files[retained.config.datafile]===check.files[check.config.datafile] - missingFromBackup = retained.commands.filter(c => c.accepted && (!sameBase || !check.commands.some(b => b.id === c.id && b.line === c.line && b.status===c.status && (c.status!=='done' || !c.present || retained.files[c.file]===check.files[b.file])))).map(c=>c.id) + const coverage = backupCoverage(retained, check) + sameBase = coverage.sameBase + missingFromBackup = coverage.missingFromBackup } const complete = {kind:'simplystore-restore-complete',config,files:check.files,fingerprint:check.fingerprint, committed:check.committed,missingFromBackup,sameBase,warnings:check.warnings, @@ -355,7 +424,11 @@ export async function releaseOfflineLocks(options, {operator, reason, confirmedS removed.push({lock,ownerFiles:saved}) } // Return preview only; the CLI persists its external audit before calling finish. - return {inspection:{ready:report.ready,errors:report.errors,commands:report.commands.map(c=>({id:c.id,status:c.status,problem:c.problem}))}, + const commands = report.commands.map(command => { + return {id: command.id, status: command.status, problem: command.problem} + }) + const inspection = {ready: report.ready, errors: report.errors, commands} + return {inspection, operator,reason,removed,async finish() { for (const {lock,ownerFiles} of removed) { for (const entry of ownerFiles) { diff --git a/src/server.mjs b/src/server.mjs index e0371ce..c7f6f15 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -2,6 +2,7 @@ import {setImmediate} from 'node:timers' import express from 'express' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' +import {from, _, anyOf, not} from '@muze-nl/jaqt' import WorkerPool from './workerPool.mjs' import { Worker } from 'worker_threads' import { fileURLToPath } from 'url' @@ -86,10 +87,33 @@ async function main(options) { try { inspection = await inspectStore(config) if (!inspection.ready) { - throw new Error('Administrative recovery required: ' + - [...inspection.errors, ...inspection.commands.filter(c => c.problem).map(c => c.problem), - ...inspection.commands.filter(c => c.status === 'accepted' || c.status === 'active').map(c => `${c.id}: ${c.status}; administrator assessment required`), - ...inspection.commands.filter(c => c.present && c.status !== 'done').map(c => `${c.id}: uncommitted dataset`)].join('; ')) + const commands = from(inspection.commands) + + const problems = commands + .where({problem: Boolean}) + .select(_.problem) + + const pending = commands + .where({status: anyOf('accepted', 'active')}) + .select(command => { + return `${command.id}: ${command.status}; administrator assessment required` + }) + + const uncommitted = commands + .where({present: Boolean, status: not('done')}) + .select(command => { + return `${command.id}: uncommitted dataset` + }) + + const reasons = [ + ...inspection.errors, + ...problems, + ...pending, + ...uncommitted + ] + + const details = reasons.join('; ') + throw new Error(`Administrative recovery required: ${details}`) } for (const [file, digest] of Object.entries(inspection.files)) { if (digest !== null) { diff --git a/src/store-inspection.mjs b/src/store-inspection.mjs index c89a3eb..092a27a 100644 --- a/src/store-inspection.mjs +++ b/src/store-inspection.mjs @@ -1,6 +1,7 @@ import fs from 'node:fs/promises' import path from 'node:path' import JSONTag from '@muze-nl/jsontag' +import {from, _, anyOf, not} from '@muze-nl/jaqt' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import {createHash} from 'node:crypto' import {assertOdJsonTagFraming, getChangesetPath} from './recovery.mjs' @@ -98,7 +99,16 @@ async function records(file, errors, kind) { export async function inspectStore(options = {}) { const config = storePaths(options) const errors = [], warnings = [] - const canonicalPaths = await Promise.all([config.datafile, config.commandLog, config.commandStatus, config.integrityFile].map(async file => path.join(await fs.realpath(path.dirname(file)),path.basename(file)))) + const configuredPaths = [ + config.datafile, + config.commandLog, + config.commandStatus, + config.integrityFile + ] + const canonicalPaths = await Promise.all(configuredPaths.map(async file => { + const directory = await fs.realpath(path.dirname(file)) + return path.join(directory, path.basename(file)) + })) if (new Set(canonicalPaths).size !== canonicalPaths.length) { errors.push('Configured canonical artifact paths overlap') } @@ -219,7 +229,14 @@ export async function inspectStore(options = {}) { else if (command.status === 'accepted' || command.status === 'active') { prefixValid = false } - command.laterDatasets = commands.filter(later => later.position > command.position && later.accepted && before[later.file] != null).map(later => later.id) + const laterDatasets = from(commands) + .where({ + position: position => position > command.position, + accepted: true, + file: file => before[file] != null + }) + .select(_.id) + command.laterDatasets = [...laterDatasets] } const extension = path.extname(config.datafile), stem = path.basename(config.datafile, extension) + '.' const known = new Set(commands.map(command => command.file)) @@ -237,8 +254,12 @@ export async function inspectStore(options = {}) { if (JSON.stringify(before) !== JSON.stringify(after)) { errors.push('Store changed during inspection') } - const pending = commands.filter(c => c.status === 'accepted' || c.status === 'active') - const damaged = commands.filter(c => c.problem || (c.present && c.status !== 'done')) + const commandQuery = from(commands) + const pending = commandQuery.where({status: anyOf('accepted', 'active')}) + const damaged = commandQuery.where(anyOf( + {problem: Boolean}, + {present: Boolean, status: not('done')} + )) const ready = errors.length === 0 && pending.length === 0 && damaged.length === 0 warnings.push('History completeness and original code/hidden inputs require independent administrator evidence; current files alone cannot prove them.') return {config, files: before, fingerprint: hash(JSON.stringify(before)), errors, warnings, diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs index c3d6188..5bf49bd 100644 --- a/test/admin-recovery.mjs +++ b/test/admin-recovery.mjs @@ -26,9 +26,11 @@ test('administrator recovers trailing missing/waiting commands in log order on a const before=await fs.readFile(store.commandStatus) const plan=await planRecovery(store,{quiescent:true}) assert.equal(plan.actionable,true);assert.deepEqual(plan.rerun,['A','B']) + assert.deepEqual(structuredClone(plan), plan, 'Recovery plans remain plain cloneable data') const result=await runRecovery(store,outputs,plan) const actual=await inspectStore(result.config) assert.equal(actual.ready,true) + assert.deepEqual(structuredClone(actual.commands), actual.commands, 'Inspection does not expose JAQT proxies') assert.deepEqual(actual.data.persons.map(p=>p.name),['A','B']) assert.deepEqual(await fs.readFile(store.commandStatus),before) assert.ok((await fs.readFile(result.config.commandStatus)).subarray(0,before.length).equals(before)) @@ -250,3 +252,51 @@ test('identical historical duplicate retains first log position; conflicting dup await fs.appendFile(store.commandLog,JSONTag.stringify({id:'A',name:'addPerson',value:{name:'different'}})+'\n') assert.ok((await planRecovery(store,{quiescent:true})).blocks.some(s=>s.includes('Conflicting command ID A'))) }) + + +test('backup coverage distinguishes changed history and data from missing source data', async t => { + const {store, outputs} = await setup(t, [['A', 'accepted']]) + const plan = await planRecovery(store, {quiescent: true}) + const recovered = await runRecovery(store, outputs, plan) + const source = recovered.config + const backup = path.join(outputs, 'coverage-backup') + await backupStore(source, {to: backup, quiescent: true}) + + const changeset = path.join(path.dirname(source.datafile), 'data.A.jsontag') + const original = new Map() + for (const file of [source.datafile, source.commandLog, source.commandStatus, changeset]) { + original.set(file, await fs.readFile(file)) + } + const differentCommand = JSONTag.stringify({id: 'A', name: 'addPerson', value: {name: 'changed'}}) + const failedStatus = JSONTag.stringify({command: 'A', status: 'failed', code: 500}) + const cases = [ + {name: 'same', missing: [], sameBase: true}, + {name: 'command', file: source.commandLog, bytes: differentCommand + '\n', missing: ['A'], sameBase: true}, + {name: 'status', file: source.commandStatus, bytes: original.get(source.commandStatus) + failedStatus + '\n', missing: ['A'], sameBase: true}, + {name: 'data', file: changeset, bytes: original.get(changeset) + ' ', missing: ['A'], sameBase: true}, + {name: 'base', file: source.datafile, bytes: original.get(source.datafile) + ' ', missing: ['A'], sameBase: false}, + {name: 'missing', file: changeset, missing: [], sameBase: true} + ] + for (const scenario of cases) { + for (const [file, bytes] of original) { + await fs.writeFile(file, bytes) + } + if (scenario.file) { + if (scenario.bytes === undefined) { + await fs.unlink(scenario.file) + } + else { + await fs.writeFile(scenario.file, scenario.bytes) + } + } + const result = await restoreBackup(backup, { + to: path.join(outputs, scenario.name), + auditDir: path.join(outputs, scenario.name + '-audit'), + source, + sourceQuiescent: true + }) + assert.deepEqual(result.missingFromBackup, scenario.missing, scenario.name) + assert.equal(result.sameBase, scenario.sameBase, scenario.name) + assert.deepEqual(structuredClone(result), result, 'Restore reports remain plain cloneable data') + } +}) From dc4e0afff2b202ac85500a13abb1f603743a33ad Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:11:32 +0200 Subject: [PATCH 30/37] Record JAQT refactor verification and review status --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 9 ++ .../EVD-20260919-TTZ7C-27-results.json | 25 +++- .../evidence/EVD-20260919-TTZ7C-27-tests.txt | 113 ++++++++++++++++++ .spiral/evidence/EVD-20260919-TTZ7C-27.md | 22 ++++ .spiral/evidence/EVD-20260919-TTZ7C-27.ttl | 5 +- .spiral/project-context.md | 2 + 6 files changed, 174 insertions(+), 2 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index 951be4f..406d799 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -251,3 +251,12 @@ Final staged, range and tree provenance validation must pass before handing over for acceptance. Keep this cycle Active until the maintainer accepts it; then record acceptance in a new commit and validate the exact prospective integration against current master. Do not merge this Active branch outward. + +### Acceptance-review readability refinements + +At the maintainer's request, all blocks use explicit braces and multiline bodies. +The subsequent JAQT refactor replaces dense diagnostic/recovery/backup expressions +with named queries and comparison steps. Runtime behavior and plain-data public +reports are preserved. The full relevant suite now passes 105/105, with the latest +verification recorded in EVD-27. These refinements do not accept or integrate the +cycle and do not replace the exact earlier filesystem evidence. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json index 2ceb1bd..dff000e 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json @@ -222,5 +222,28 @@ "backup --quiescent", "restore --store --quiescent", "verify" - ] + ], + "jaqtRefactor": { + "implementation": "49fa48c123d0035df087b7a9d4c1460da92b85fa", + "passed": 105, + "failed": 0, + "skipped": 0, + "todo": 0, + "changedProductionFiles": [ + "src/server.mjs", + "src/store-inspection.mjs", + "src/admin-recovery.mjs" + ], + "plainDataCloning": "passed", + "backupCoverageVariations": [ + "same", + "command", + "status", + "data", + "base", + "missing" + ], + "changedFileLint": "passed", + "newVmRun": false + } } diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt index a81dc0e..e0622cd 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt @@ -136,3 +136,116 @@ simplystore-acceptance-examples-detail.log β„Ή skipped 0 β„Ή todo 0 β„Ή duration_ms 1023.688896 + +JAQT refactor at 49fa48c123d0035df087b7a9d4c1460da92b85fa +βœ” atomicity: command mutation followed by failure does not become visible or committed (532.107676ms) +βœ” isolation: mutating data inside a JavaScript query does not affect later queries (324.200054ms) +βœ” consistency: query during an active command sees the previous committed state (1054.010803ms) +βœ” idempotency: duplicate command ID is acknowledged but not applied twice (530.143222ms) +βœ” idempotency: retry during active command reports current status and does not enqueue twice (998.609118ms) +βœ” administrator recovers trailing missing/waiting commands in log order on a copy (241.327474ms) +βœ” later accepted dataset blocks rerun, including zero-byte no-op (16.185536ms) +βœ” later done dataset blocks rerun, including zero-byte no-op (16.552248ms) +βœ” later unsafe dataset blocks rerun, including zero-byte no-op (7.222186ms) +βœ” new later file makes an approved plan stale before executing a handler (18.110875ms) +βœ” changed code and missing explicit approval reject apply (45.191118ms) +βœ” cooperative ownership excludes competing writer and path aliases (26.89224ms) +βœ” backup and restore validate saved bytes without executing commands (266.833759ms) +βœ” read-only inspection never imports handlers and rejects conflicting done order (8.947992ms) +βœ” historical status order never makes a later log dataset eligible for bypass (6.309472ms) +βœ” interrupted recovery preserves completed attempts and blocks an unfinished candidate (177.417264ms) +βœ” retained digest mismatch stops recovery before replacing integrity or appending done (89.767431ms) +βœ” malformed tails and overlapping paths cannot be accepted as valid stores (7.603056ms) +βœ” approved recovery resumes from completed candidate prefix without repeating external effects (328.172436ms) +βœ” finishing an interrupted completion report reconstructs data without executing again (91.850579ms) +βœ” multi-directory integrity paths and file permissions survive backup and restore (91.720318ms) +βœ” backup excludes unrelated secrets while retaining declared custom artifacts (17.910951ms) +βœ” a backup remains restorable after transfer to a different directory (22.998871ms) +βœ” promotion verification rejects code changed after completed recovery (87.006294ms) +βœ” verified A prefix permits only missing B, waiting C and missing D in log order (460.71018ms) +βœ” restore reports accepted commands absent from an older backup (20.271611ms) +βœ” identical historical duplicate retains first log position; conflicting duplicate blocks (9.887978ms) +βœ” backup coverage distinguishes changed history and data from missing source data (205.716463ms) +βœ” handler lifecycle: index update failure prevents command commit (519.483947ms) +βœ” handler lifecycle: derived index file can remain after failed command (376.97463ms) +βœ” handler lifecycle: final offset write failure prevents command commit (298.168032ms) +βœ” handler lifecycle: custom finalizer rejection prevents command commit (298.790672ms) +βœ” handler lifecycle: current index update can mutate canonical state before commit (332.522035ms) +βœ” command edit indexes only its changed record using changeset UTF-8 offsets (32.151269ms) +βœ” command addition indexes new entities and preserves interior skip numbers (9.547552ms) +βœ” command offsets include mutations and new records introduced by the index hook (6.824472ms) +βœ” successive command indexes are relative to their own changeset files (7.807516ms) +βœ” no-op commands replace any stale sidecar with an empty offset index (5.904184ms) +βœ” failure to write final offsets rejects the command before success (6.037079ms) +βœ” commands await custom finalization that delegates to the default index (27.552478ms) +βœ” custom finalization can replace the default without an unconditional offset write (7.920356ms) +βœ” custom finalization rejection propagates from the command worker (4.951147ms) +βœ” default conversion offsets match every final record (85.67897ms) +βœ” conversion finalizes offsets after custom indexing changes record sizes and adds records (70.588717ms) +βœ” conversion replaces stale offsets even when the custom hook writes no index (89.021922ms) +βœ” conversion awaits custom finalization with the original receiver and final bytes (169.155138ms) +βœ” conversion propagates rejection from custom finalization (59.819893ms) +βœ” conversion initializes existing-format logs and optional base integrity without overwriting a store (166.155078ms) +βœ” runtime environment defaults to production and ignores production fault points (1.643785ms) +βœ” fault points are inert outside test environment (0.199473ms) +βœ” crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (598.803859ms) +βœ” crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (389.509455ms) +βœ” crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (390.5932ms) +βœ” crash at before-command-changeset-write preserves evidence and never automatically reruns (414.629954ms) +βœ” crash at after-command-changeset-write preserves evidence and never automatically reruns (357.3841ms) +βœ” crash at before-command-done-status preserves evidence and never automatically reruns (361.370497ms) +βœ” crash after done status but before query update recovers committed state without replay (457.528028ms) +βœ” crash ownership cannot be stolen automatically (305.466252ms) +βœ” hanging command times out unsafe and later accepted command commits (344.438148ms) +βœ” hanging load worker fails startup explicitly (200.290418ms) +βœ” integrity-enabled command writes changeset digest before done status (323.951706ms) +βœ” normal restart preserves committed state according to reconstruction oracle (430.876834ms) +βœ” accepted command changeset is not treated as committed state on startup (54.887938ms) +βœ” done command with missing changeset refuses recovery instead of silently using base data (44.489354ms) +βœ” malformed base OD-JSONTag file refuses recovery (53.9879ms) +βœ” malformed committed changeset refuses recovery (62.875247ms) +βœ” truncated committed changeset payload refuses recovery before lazy parsing can hide it (57.170068ms) +βœ” malformed uncommitted changeset is ignored during committed startup reconstruction (56.035862ms) +βœ” durable status file selects only done command changesets for startup (74.028892ms) +βœ” durable command log replays only accepted commands (11.255528ms) +βœ” active command at attempt 1 remains unchanged pending administrator assessment (4.023563ms) +βœ” active command at attempt 2 remains unchanged pending administrator assessment (5.108949ms) +βœ” malformed durable status record refuses recovery with explicit integrity error (2.544773ms) +βœ” structurally invalid durable status record refuses recovery (2.119582ms) +βœ” malformed durable command log record refuses recovery with explicit integrity error (4.181299ms) +βœ” structurally invalid durable command log record refuses recovery (2.684668ms) +βœ” OD-JSONTag framing validation catches truncated lazy records (2.12683ms) +βœ” OD-JSONTag framing validation accepts changeset skip records (0.27993ms) +βœ” integrity manifest detects same-length altered base OD-JSONTag payload (72.635582ms) +βœ” integrity manifest detects same-length altered committed changeset payload (85.968267ms) +βœ” integrity enabled requires manifest entry for base data (55.965339ms) +βœ” PL01-05 durable publication survives modeled loss of unsynced names and restart (824.722012ms) +βœ” PL10 an actual short changeset write is completed before acknowledgment (345.526846ms) +βœ” PL07-09/11/18-20 log sync failure stops mutation without false terminal success (238.53111ms) +βœ” PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (238.050442ms) +βœ” PL07-09/11/18-20 log close failure stops mutation without false terminal success (185.767947ms) +βœ” PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (287.340691ms) +βœ” PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (227.624115ms) +βœ” PL07-09/11/18-20 done sync failure stops mutation without false terminal success (249.948147ms) +βœ” PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (246.860237ms) +βœ” PL16 delayed append cannot be overtaken (321.306723ms) +βœ” PL16 delayed append cannot be overtaken by duplicate ID (267.407714ms) +βœ” PL02-04/06 missing command-log.jsontag fails open rather than recreating history (317.57945ms) +βœ” PL02-04/06 missing command-status.jsontag fails open rather than recreating history (312.701103ms) +βœ” PL02-04/06 missing data.A.jsontag fails open rather than recreating history (326.792291ms) +βœ” PL12/17 startup preserves uncertain active work without repeating external effects (287.859862ms) +βœ” directory sync failure after rename prevents done (186.775761ms) +βœ” directory close:after failure after rename prevents done (185.850535ms) +βœ” changeset open error is not masked or acknowledged (202.392105ms) +βœ” changeset write error is not masked or acknowledged (210.718358ms) +βœ” changeset close:after error is not masked or acknowledged (185.063299ms) +βœ” short command-log append preserves its entire JSONTag record (192.086459ms) +βœ” complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.346819ms) +βœ” serialized writer preserves order and stops after an uncertain operation (0.464406ms) +βœ” durable publication replaces entire contents and concurrent append retains each record once (36.437087ms) +β„Ή tests 105 +β„Ή pass 105 +β„Ή fail 0 +β„Ή skipped 0 +β„Ή todo 0 +β„Ή duration_ms 6076.58365 diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md index 8a8fc82..8b3bdee 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.md +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -141,3 +141,25 @@ literal raw spelling. All 26 trees are identical; 728 block statements were checked, with zero inline blocks. Curly/brace-style/indentation/trailing-whitespace checks pass. No runtime tests or VM experiments were repeated for this verified whitespace-only adjustment. + +## JAQT readability verification + +Verified IMP-26 at `49fa48c123d0035df087b7a9d4c1460da92b85fa` after the maintainer +approved the JAQT example and requested review of the remaining code. Changes +are limited to startup diagnostics, store inspection, administrator recovery and +backup coverage, plus a focused regression test. The review retained simple +array operations and ordered I/O loops where JAQT added no clarity. + +The complete relevant suite listed above passed **105/105**, with zero failures, +skips or TODOs. This includes the earlier recovery/order/short-write/fault cases, +plus a new backup-coverage case comparing unchanged backup state against changed +command input, status, data and base bytes, and missing source data. Structured +clone assertions confirm that inspection command lists, recovery plans and restore +reports remain ordinary data, with no exposed JAQT proxies. Startup diagnostics +were also compared against the earlier expression across 50 representative inputs +during the preceding read-only proposal; messages and ordering matched. + +ESLint passes all three changed production modules and the changed test file; +Git whitespace checks pass. No persistence protocol changed, and no new VM +power-cut run is claimed. The recorded filesystem experiment still refers to its +exact original runtime revision. Cycle acceptance remains pending. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl index 557c774..1895dca 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl @@ -8,4 +8,7 @@ project:EVD-20260919-TTZ7C-27 a sd:VerificationEvidence ; sd:status sd:Active ; sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; - sd:gitCommit "13db3bf55926cf3a3a1b2769c422881a875193c8" ] . + sd:gitCommit "13db3bf55926cf3a3a1b2769c422881a875193c8" ] ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "49fa48c123d0035df087b7a9d4c1460da92b85fa" ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 993bd29..0ec4dde 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -24,6 +24,8 @@ Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offs Active-cycle progress on 2026-09-19: `IMP-20260919-TTZ7C-26` implements the corrected `DES-20260919-TTZ7C-24/25` following explicit authorization to complete the whole cycle. Existing command/status/integrity and dataset formats remain unchanged, with no migration. Durable complete-write and file/directory barriers, ownership, log-authoritative serial acceptance/execution, command-only workers, strict startup assessment, and source-bound administrator recovery/backup/restore are implemented. `EVD-20260919-TTZ7C-27` records 101 passing regression tests plus 22 expanded administrator tests (three additional cases), direct CLI checks, and a passing four-scenario ext4/QEMU power-cut exercise with device/filesystem negative controls and latency. `DURABILITY.md` and `docs/recovery.md` define the bounded behavior and runnable procedure. Historical baseline gaps remain at `EVD-20260919-TTZ7C-23`; PL15/16 expectations were corrected by the maintainer, not silently weakened. Full rebuild remains a documented separate contract. The cycle is ready for human acceptance but not Accepted or integrated. Broader filesystem/hardware and randomized testing, undeclared custom dependencies and historical input/effect uncertainty remain explicit limits. Two unchanged legacy index `load` methods have pre-existing undefined-meta lint errors outside this write-path cycle. +Acceptance-review refinements: explicit multiline blocks and selective JAQT queries now improve the cycle code’s readability. The latest full relevant regression suite passes 105/105; EVD-27 records preserved ordering, backup coverage and plain-data report contracts. Human cycle acceptance is still pending. + ## Intake State Status: **Complete** From f14d13e2bf4b0f6ab3afeea7e9900ef60488590e Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:16:15 +0200 Subject: [PATCH 31/37] Record conditional-expression and line-width preferences --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 12 ++++++++++++ AGENTS.md | 2 ++ 2 files changed, 14 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index db3719d..bfce2fc 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -133,3 +133,15 @@ Review the remaining code for clearer query expressions and named intermediate results; use JAQT where it improves readability. Preserve behavior, ordering and public result types. This authorizes the refactor within the still-active cycle; it does not constitute cycle acceptance. + +## Conditional expressions and line width β€” 2026-09-19 + +The maintainer added these readability preferences: + +> More small changes: I prefer not to use ?: unless it is a clear win in readability. I prefer to keep lines about 80 chars wide at most, again, unless readability suffers. + +Expand dense/nested ternaries into named steps or `if`/`else`; retain short, +obvious conditional choices when they are clearer. Wrap the cycle's JavaScript +toward 80 columns, preserving exceptions for readability and literal contents. +These are review guidelines with explicit exceptions, not blanket prohibitions. +This authorizes a behavior-preserving readability revision, not cycle acceptance. diff --git a/AGENTS.md b/AGENTS.md index e5c5760..c52123e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,6 +60,8 @@ Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifa - Always use braces for control-flow bodies, including single-statement `if`, `else`, and loop bodies. Start block contents on the line after `{`, and put the closing `}` on its own line. This includes empty blocks and callbacks; `else`, `catch`, and `finally` start on the next line after `}`. +- Prefer `if`/`else` to `?:` unless a short conditional expression clearly improves readability. +- Aim for code lines no wider than about 80 characters; allow exceptions when wrapping would reduce readability. - Prefer named intermediate results over dense chains of operations. Use the existing JAQT library where it makes data selection and projection clearer; keep straightforward operations simple. - Treat `master` as the authoritative branch unless the human states otherwise. - For ordinary repository-changing Spiral cycle work, create one dedicated branch from the authoritative branch, normally `spiral/CYC-###-short-goal`. From 5d166a23b664ba9fbe34ae819feb1ed70959046c Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:22:04 +0200 Subject: [PATCH 32/37] Simplify conditionals and wrap cycle code near 80 columns --- .../implementations/IMP-20260919-TTZ7C-26.md | 18 + .../implementations/IMP-20260919-TTZ7C-26.ttl | 5 +- eslint.config.js | 30 +- scripts/convert.mjs | 60 +- scripts/power-loss-vm/guest.mjs | 166 +++- scripts/recover.mjs | 131 ++- src/admin-recovery.mjs | 737 +++++++++++---- src/command-worker-module.mjs | 65 +- src/command-worker.mjs | 10 +- src/execute-worker.mjs | 22 +- src/index.id.mjs | 20 +- src/index.offset.mjs | 57 +- src/recovery.mjs | 181 +++- src/server.mjs | 436 ++++++--- src/storage.mjs | 50 +- src/store-inspection.mjs | 277 ++++-- src/store-ownership.mjs | 81 +- src/util.mjs | 20 +- test/admin-recovery.mjs | 842 +++++++++++++----- test/after-change-handlers.mjs | 116 ++- test/conversion-offsets.mjs | 164 +++- test/durability-crash.mjs | 185 ++-- test/durability-helpers.mjs | 105 ++- test/durability-latency.mjs | 81 +- test/durability-startup.mjs | 188 ++-- test/power-loss-baseline.mjs | 430 ++++++--- test/power-loss-helpers.mjs | 152 +++- test/power-loss-preload.mjs | 178 ++-- test/storage.mjs | 100 ++- 29 files changed, 3557 insertions(+), 1350 deletions(-) diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md index c2f37db..d782905 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.md +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -83,3 +83,21 @@ code-hash expressions where JAQT would not improve the operation. Tests cover backup differences and unchanged plain-data output alongside the existing suite. This is a behavior-preserving refactor; persistence boundaries, formats, recovery eligibility and administrator authority are unchanged. + +## Conditional expressions and 80-column review + +Expanded optional configuration fields, recovery metadata loading, CLI argument +handling, index buffer conversion and nested fault/VM fixture choices into named +steps and explicit branches. Retained ten short conditional expressions where a +simple two-value choice is clearer; the reviewed files originally contained 36. +Wrapped the cycle's 26 JavaScript files toward 80 columns while preserving the +maintainer's multiline braces and each file's indentation convention. Literal +contents remain intact when splitting would reduce readability. ESLint now warns +on line width with explicit literal/URL exceptions; ternaries remain a review +judgment rather than a blanket prohibited construct. + +Formatting used pinned Prettier 3.6.2 installed only under /tmp, followed by the +existing brace rules; no package dependency or lockfile changed. The formatting +step was checked with syntax-tree comparison. Conditional rewrites additionally +receive the relevant regression and CLI verification recorded in EVD-27. Storage +barriers, canonical formats and recovery policy are unchanged. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl index 128e2d4..4c469fa 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -24,11 +24,12 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ; sd:implementationLocation "test/storage.mjs" ; sd:implementationLocation "docs/recovery.md" ; sd:implementationLocation "DURABILITY.md" ; + sd:implementationLocation "eslint.config.js" ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-25 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "529272489ca03b50271c3555987eb96dd2279ed5" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "529272489ca03b50271c3555987eb96dd2279ed5" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "f14d13e2bf4b0f6ab3afeea7e9900ef60488590e" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "f14d13e2bf4b0f6ab3afeea7e9900ef60488590e" ] ; sd:implementationChangeKind sd:BehaviorPreservingChange . diff --git a/eslint.config.js b/eslint.config.js index aaaef59..fd8fb61 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -1,8 +1,26 @@ -import js from "@eslint/js"; -import globals from "globals"; -import { defineConfig } from "eslint/config"; +import js from '@eslint/js' +import globals from 'globals' +import { defineConfig } from 'eslint/config' export default defineConfig([ - { rules: { curly: ["error", "all"], "brace-style": ["error", "stroustrup", { allowSingleLine: false }] } }, - { files: ["**/*.{js,mjs,cjs}"], plugins: { js }, extends: ["js/recommended"], languageOptions: { globals: globals.browser } }, -]); + { + rules: { + curly: ['error', 'all'], + 'brace-style': ['error', 'stroustrup', { allowSingleLine: false }], + 'max-len': ['warn', { + code: 80, + tabWidth: 4, + ignoreUrls: true, + ignoreStrings: true, + ignoreTemplateLiterals: true, + ignoreRegExpLiterals: true + }] + } + }, + { + files: ['**/*.{js,mjs,cjs}'], + plugins: { js }, + extends: ['js/recommended'], + languageOptions: { globals: globals.browser } + } +]) diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 79f160a..980c61d 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -1,4 +1,4 @@ -import {Buffer} from 'node:buffer' +import { Buffer } from 'node:buffer' import process from 'node:process' import JSONTag from '@muze-nl/jsontag' import serialize, { stringify } from '@muze-nl/od-jsontag/src/serialize.mjs' @@ -6,16 +6,21 @@ import Parser from '@muze-nl/od-jsontag' import { finalizeIndex } from '../src/index.mjs' import fs from 'node:fs' import path from 'node:path' -import {publishFile, syncFile} from '../src/storage.mjs' -import {acquireOwnership} from '../src/store-ownership.mjs' -import {appendIntegrityRecord, getDefaultIntegrityFile} from '../src/integrity.mjs' +import { publishFile, syncFile } from '../src/storage.mjs' +import { acquireOwnership } from '../src/store-ownership.mjs' +import { + appendIntegrityRecord, + getDefaultIntegrityFile +} from '../src/integrity.mjs' -const __dirname = import.meta.dirname; +const __dirname = import.meta.dirname const integrity = process.argv.includes('--integrity') const args = process.argv.slice(2).filter(value => value !== '--integrity') -if (args.length<2) { - console.log('usage: node ./convert.mjs {inputfile} {outputfile} {indexlib?} {schema?} [--integrity]') +if (args.length < 2) { + console.log( + 'usage: node ./convert.mjs {inputfile} {outputfile} {indexlib?} {schema?} [--integrity]' + ) process.exit() } @@ -23,30 +28,38 @@ if (args.length<2) { let inputFile = args[0] let outputFile = args[1] let indexFile = args[2] -if (indexFile && indexFile[0]!='/') { - indexFile = process.cwd()+'/'+indexFile +if (indexFile && indexFile[0] != '/') { + indexFile = process.cwd() + '/' + indexFile } else if (!indexFile) { - indexFile = __dirname+'/../src/index.mjs' + indexFile = __dirname + '/../src/index.mjs' } let schemaFile = args[3] -if (schemaFile && schemaFile[0]!='/') { - schemaFile = process.cwd()+'/'+schemaFile +if (schemaFile && schemaFile[0] != '/') { + schemaFile = process.cwd() + '/' + schemaFile } async function main() { const directory = path.resolve(path.dirname(outputFile)) - const logs = ['command-log.jsontag', 'command-status.jsontag'].map(name => path.join(directory, name)) + const logs = ['command-log.jsontag', 'command-status.jsontag'].map(name => + path.join(directory, name) + ) const ownership = await acquireOwnership([directory]) - if ([outputFile, ...logs, getDefaultIntegrityFile(outputFile)].some(file => fs.existsSync(file))) { + if ( + [outputFile, ...logs, getDefaultIntegrityFile(outputFile)].some(file => + fs.existsSync(file) + ) + ) { await ownership.release() - throw new Error('Conversion requires new output files; existing store is not overwritten') + throw new Error( + 'Conversion requires new output files; existing store is not overwritten' + ) } // now create indexes console.log('Using index library:', indexFile) const index = await import(indexFile).then(mod => { - return mod.default + return mod.default }) let schema = {} @@ -68,7 +81,7 @@ async function main() { // indexes need the position data which is only available after // parsing the od-jsontag data - const parser = new Parser('',false) // allow mutations + const parser = new Parser('', false) // allow mutations const odData = parser.parse(strData) let meta = { @@ -89,21 +102,26 @@ async function main() { strData = stringify(serialize(odData)) await publishFile(outputFile, strData) - // Custom indexes may change record sizes or add records after initial parsing. + // Custom indexes may change record sizes or add records after initial + // parsing. await finalizeIndex(index, strData, meta) if (integrity) { - await appendIntegrityRecord(getDefaultIntegrityFile(outputFile), outputFile, Buffer.from(strData)) + await appendIntegrityRecord( + getDefaultIntegrityFile(outputFile), + outputFile, + Buffer.from(strData) + ) } for (const file of logs) { await publishFile(file, '') } - for (const entry of fs.readdirSync(directory, {withFileTypes:true})) { + for (const entry of fs.readdirSync(directory, { withFileTypes: true })) { if (entry.isFile()) { await syncFile(path.join(directory, entry.name)) } } await ownership.release() - console.log('Converted data written to ',outputFile) + console.log('Converted data written to ', outputFile) } main() diff --git a/scripts/power-loss-vm/guest.mjs b/scripts/power-loss-vm/guest.mjs index 9dcfb4e..c4668d4 100644 --- a/scripts/power-loss-vm/guest.mjs +++ b/scripts/power-loss-vm/guest.mjs @@ -3,57 +3,145 @@ import fs from 'node:fs/promises' import JSONTag from '@muze-nl/jsontag' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import server from './src/server.mjs' -import {publishFile} from './src/storage.mjs' -import {inspectStore} from './src/store-inspection.mjs' -const mode=(await fs.readFile('/proc/cmdline','utf8')).match(/scenario=(\w+)/)?.[1] || 'done' -const config={datafile:'/store/data.jsontag',commandLog:'/store/command-log.jsontag',commandStatus:'/store/command-status.jsontag',commandsFile:'/app/commands.mjs',indexFile:'/app/index.mjs'} -console.log('ENVELOPE '+JSON.stringify({kernel:await fs.readFile('/proc/version','utf8'),mounts:await fs.readFile('/proc/mounts','utf8'),writeCache:await fs.readFile('/sys/block/vda/queue/write_cache','utf8')})) -const exists=await fs.access(config.datafile).then(()=>true,()=>false) -if(exists){ - const report=await inspectStore(config) - console.log('VERIFY '+JSON.stringify({ready:report.ready,errors:report.errors,commands:report.commands.map(c=>({id:c.id,status:c.status,present:c.present})),persons:report.data?.persons,unsyncedPresent:await fs.access('/store/unsynced').then(()=>true,()=>false)})) +import { publishFile } from './src/storage.mjs' +import { inspectStore } from './src/store-inspection.mjs' +const mode = + (await fs.readFile('/proc/cmdline', 'utf8')).match(/scenario=(\w+)/)?.[1] || + 'done' +const config = { + datafile: '/store/data.jsontag', + commandLog: '/store/command-log.jsontag', + commandStatus: '/store/command-status.jsontag', + commandsFile: '/app/commands.mjs', + indexFile: '/app/index.mjs' +} +console.log( + 'ENVELOPE ' + + JSON.stringify({ + kernel: await fs.readFile('/proc/version', 'utf8'), + mounts: await fs.readFile('/proc/mounts', 'utf8'), + writeCache: await fs.readFile( + '/sys/block/vda/queue/write_cache', + 'utf8' + ) + }) +) +const exists = await fs.access(config.datafile).then( + () => true, + () => false +) +if (exists) { + const report = await inspectStore(config) + console.log( + 'VERIFY ' + + JSON.stringify({ + ready: report.ready, + errors: report.errors, + commands: report.commands.map(c => ({ + id: c.id, + status: c.status, + present: c.present + })), + persons: report.data?.persons, + unsyncedPresent: await fs.access('/store/unsynced').then( + () => true, + () => false + ) + }) + ) process.exit(0) } -await publishFile(config.datafile,serialize(JSONTag.parse('{"persons":[]}'))) -await publishFile(config.commandLog,'');await publishFile(config.commandStatus,'') -if(mode==='unsynced'){ - await fs.writeFile('/store/unsynced','not guaranteed durable');console.log('CUT_UNSYNCED');await new Promise(()=>{ +await publishFile(config.datafile, serialize(JSONTag.parse('{"persons":[]}'))) +await publishFile(config.commandLog, '') +await publishFile(config.commandStatus, '') +if (mode === 'unsynced') { + await fs.writeFile('/store/unsynced', 'not guaranteed durable') + console.log('CUT_UNSYNCED') + await new Promise(() => { }) } -await fs.writeFile('/app/commands.mjs',`export default {addPerson(data,command){${mode==='active'?"console.log('CUT_ACTIVE');while(true){}":mode==='accepted'?"while(true){}":"data.persons.push(command.value)"}}}`) -await fs.writeFile('/app/index.mjs','export default {update(){},create(){},load(){return {}}}') -await server.run({...config,port:3000,maxWorkers:1,commandTimeout:0,wwwroot:'/app/www'}) -await new Promise(r=>setTimeout(r,100)) -const before=performance.now() -const response=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id:'A',name:'addPerson',value:{name:'A'}})}) -console.log('ACCEPTED '+JSON.stringify({http:response.status,ms:performance.now()-before})) -if(mode==='accepted'){ +let commandBody = 'data.persons.push(command.value)' +if (mode === 'active') { + commandBody = "console.log('CUT_ACTIVE');while(true){}" +} +else if (mode === 'accepted') { + commandBody = 'while(true){}' +} +const commandModule = `export default {addPerson(data,command){${commandBody}}}` +await fs.writeFile('/app/commands.mjs', commandModule) +await fs.writeFile( + '/app/index.mjs', + 'export default {update(){},create(){},load(){return {}}}' +) +await server.run({ + ...config, + port: 3000, + maxWorkers: 1, + commandTimeout: 0, + wwwroot: '/app/www' +}) +await new Promise(r => setTimeout(r, 100)) +const before = performance.now() +const response = await fetch('http://127.0.0.1:3000/command', { + method: 'POST', + headers: { 'content-type': 'application/jsontag' }, + body: JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) +}) +console.log( + 'ACCEPTED ' + + JSON.stringify({ + http: response.status, + ms: performance.now() - before + }) +) +if (mode === 'accepted') { console.log('CUT_ACCEPTED') } -if(mode==='done'){ - for(;;){ - const status=await fetch('http://127.0.0.1:3000/command/A').then(r=>r.json());if(status.status==='done'){ - break; - }await new Promise(r=>setTimeout(r,10)) +if (mode === 'done') { + for (;;) { + const status = await fetch('http://127.0.0.1:3000/command/A').then(r => + r.json() + ) + if (status.status === 'done') { + break + } + await new Promise(r => setTimeout(r, 10)) } - const samples=[{id:'A',doneMs:performance.now()-before}] - for(let i=0;i<4;i++){ - const id='L'+i,start=performance.now() - const accepted=await fetch('http://127.0.0.1:3000/command',{method:'POST',headers:{'content-type':'application/jsontag'},body:JSONTag.stringify({id,name:'addPerson',value:{name:id}})}) - if(accepted.status!==202){ + const samples = [{ id: 'A', doneMs: performance.now() - before }] + for (let i = 0; i < 4; i++) { + const id = 'L' + i, + start = performance.now() + const accepted = await fetch('http://127.0.0.1:3000/command', { + method: 'POST', + headers: { 'content-type': 'application/jsontag' }, + body: JSONTag.stringify({ + id, + name: 'addPerson', + value: { name: id } + }) + }) + if (accepted.status !== 202) { throw new Error('Acceptance failed') } - const acceptMs=performance.now()-start - for(;;){ - const status=await fetch('http://127.0.0.1:3000/command/'+id).then(r=>r.json());if(status.status==='done'){ - break; - }await new Promise(r=>setTimeout(r,10)) + const acceptMs = performance.now() - start + for (;;) { + const status = await fetch( + 'http://127.0.0.1:3000/command/' + id + ).then(r => r.json()) + if (status.status === 'done') { + break + } + await new Promise(r => setTimeout(r, 10)) } - samples.push({id,acceptMs,doneMs:performance.now()-start}) + samples.push({ id, acceptMs, doneMs: performance.now() - start }) } - console.log('CUT_DONE '+JSON.stringify({samples})) + console.log('CUT_DONE ' + JSON.stringify({ samples })) } -await new Promise(()=>{ +await new Promise(() => { }) diff --git a/scripts/recover.mjs b/scripts/recover.mjs index 205b555..26fdb12 100644 --- a/scripts/recover.mjs +++ b/scripts/recover.mjs @@ -1,64 +1,133 @@ import process from 'node:process' import fs from 'node:fs/promises' import path from 'node:path' -import {planRecovery,writePlan,applyRecovery,backupStore,restoreBackup,releaseOfflineLocks,finishRecovery,verifyCandidate} from '../src/admin-recovery.mjs' -import {inspectStore} from '../src/store-inspection.mjs' +import { + planRecovery, + writePlan, + applyRecovery, + backupStore, + restoreBackup, + releaseOfflineLocks, + finishRecovery, + verifyCandidate +} from '../src/admin-recovery.mjs' +import { inspectStore } from '../src/store-inspection.mjs' -const [action,...args] = process.argv.slice(2) +const [action, ...args] = process.argv.slice(2) const options = {} -for (let i=0;i JSON.parse(await fs.readFile(file,'utf8')) +const read = async file => JSON.parse(await fs.readFile(file, 'utf8')) const required = key => { if (typeof options[key] !== 'string') { - throw new Error(`--${key} is required`); - } return options[key] + throw new Error(`--${key} is required`) + } + return options[key] } async function main() { switch (action) { case 'inspect': { - const plan = await planRecovery(await read(required('store')), {quiescent:options.quiescent === true}) - await writePlan(required('out'),plan) + const plan = await planRecovery(await read(required('store')), { + quiescent: options.quiescent === true + }) + await writePlan(required('out'), plan) return plan } - case 'apply': - return applyRecovery(await read(required('plan')), {to:required('to'),auditDir:required('audit-dir'), - approveRerun:typeof options['approve-rerun'] === 'string' ? options['approve-rerun'].split(',').filter(Boolean) : [], - operator:required('operator'),reason:required('reason')}) + case 'apply': { + const plan = await read(required('plan')) + const to = required('to') + const auditDir = required('audit-dir') + let approveRerun = [] + if (typeof options['approve-rerun'] === 'string') { + approveRerun = options['approve-rerun'] + .split(',') + .filter(Boolean) + } + return applyRecovery(plan, { + to, + auditDir, + approveRerun, + operator: required('operator'), + reason: required('reason') + }) + } case 'finish': - return finishRecovery(required('audit-dir'),{operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped']===true}) + return finishRecovery(required('audit-dir'), { + operator: required('operator'), + reason: required('reason'), + confirmedStopped: options['confirmed-stopped'] === true + }) case 'backup': - return backupStore(await read(required('store')), {to:required('to'),quiescent:options.quiescent === true}) - case 'restore': - return restoreBackup(required('backup'), {to:required('to'),auditDir:required('audit-dir'), - source:options.store ? await read(options.store) : undefined,sourceQuiescent:options.quiescent===true}) + return backupStore(await read(required('store')), { + to: required('to'), + quiescent: options.quiescent === true + }) + case 'restore': { + const backup = required('backup') + const to = required('to') + const auditDir = required('audit-dir') + let source + if (options.store) { + source = await read(options.store) + } + return restoreBackup(backup, { + to, + auditDir, + source, + sourceQuiescent: options.quiescent === true + }) + } case 'unlock': { const store = await read(required('store')) - const release = await releaseOfflineLocks(store, {operator:required('operator'),reason:required('reason'),confirmedStopped:options['confirmed-stopped'] === true}) + const release = await releaseOfflineLocks(store, { + operator: required('operator'), + reason: required('reason'), + confirmedStopped: options['confirmed-stopped'] === true + }) const audit = path.resolve(required('out')) - // writePlan enforces new output outside every source data directory. - await writePlan(audit, {...release,config:(await inspectStore(store)).config}) + // writePlan enforces new output outside every source data + // directory. + await writePlan(audit, { + ...release, + config: (await inspectStore(store)).config + }) await release.finish() - return {audit, released:release.removed.map(item=>item.lock)} + return { audit, released: release.removed.map(item => item.lock) } } case 'verify': { const report = await read(required('report')) const current = await verifyCandidate(report) const output = path.resolve(required('out')) - await writePlan(output,{config:report.config,kind:'simplystore-activation-preview',store:{...report.config,...report.code},committed:current.committed, - instruction:'Stop the old service, preserve its configuration, and explicitly select this store. This command does not deploy or switch a service.'}) - return {verified:true,activationPreview:output} + await writePlan(output, { + config: report.config, + kind: 'simplystore-activation-preview', + store: { ...report.config, ...report.code }, + committed: current.committed, + instruction: + 'Stop the old service, preserve its configuration, and explicitly select this store. This command does not deploy or switch a service.' + }) + return { verified: true, activationPreview: output } } default: - throw new Error('Usage: recover.mjs inspect|apply|backup|restore|unlock|finish|verify --store config.json ...; see docs/recovery.md') + throw new Error( + 'Usage: recover.mjs inspect|apply|backup|restore|unlock|finish|verify --store config.json ...; see docs/recovery.md' + ) } } -main().then(result=>console.log(JSON.stringify(result,null,2))).catch(error=>{ - console.error(error.message) - process.exitCode = 1 -}) +main() + .then(result => console.log(JSON.stringify(result, null, 2))) + .catch(error => { + console.error(error.message) + process.exitCode = 1 + }) diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs index e308a99..ba77742 100644 --- a/src/admin-recovery.mjs +++ b/src/admin-recovery.mjs @@ -1,50 +1,80 @@ -import {Buffer} from 'node:buffer' +import { Buffer } from 'node:buffer' import fs from 'node:fs/promises' import path from 'node:path' -import {fileURLToPath} from 'node:url' +import { fileURLToPath } from 'node:url' import JSONTag from '@muze-nl/jsontag' -import {from, _, anyOf, not} from '@muze-nl/jaqt' +import { from, _, anyOf, not } from '@muze-nl/jaqt' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' -import {inspectStore, inventory, hash, mutableDirectories, storePaths} from './store-inspection.mjs' -import {acquireOwnership} from './store-ownership.mjs' -import {publishFile, appendRecord, durableMkdir, syncDirectory, syncFile} from './storage.mjs' -import {executeWorker} from './execute-worker.mjs' -import {nextActiveCommandStatus} from './recovery.mjs' -import {loadIntegrityManifest, verifyIntegrity} from './integrity.mjs' +import { + inspectStore, + inventory, + hash, + mutableDirectories, + storePaths +} from './store-inspection.mjs' +import { acquireOwnership } from './store-ownership.mjs' +import { + publishFile, + appendRecord, + durableMkdir, + syncDirectory, + syncFile +} from './storage.mjs' +import { executeWorker } from './execute-worker.mjs' +import { nextActiveCommandStatus } from './recovery.mjs' +import { loadIntegrityManifest, verifyIntegrity } from './integrity.mjs' -const defaultWorker = fileURLToPath(new URL('./command-worker.mjs', import.meta.url)) +const defaultWorker = fileURLToPath( + new URL('./command-worker.mjs', import.meta.url) +) const defaultIndex = fileURLToPath(new URL('./index.mjs', import.meta.url)) -const defaultCommands = fileURLToPath(new URL('./commands.mjs', import.meta.url)) +const defaultCommands = fileURLToPath( + new URL('./commands.mjs', import.meta.url) +) function deployment(options) { - return {commandsFile:path.resolve(options.commandsFile || defaultCommands), - indexFile:path.resolve(options.indexFile || defaultIndex), - commandWorker:path.resolve(options.commandWorker || defaultWorker)} + return { + commandsFile: path.resolve(options.commandsFile || defaultCommands), + indexFile: path.resolve(options.indexFile || defaultIndex), + commandWorker: path.resolve(options.commandWorker || defaultWorker) + } } async function codeHashes(code) { const entries = Object.entries(code) - const hashes = await Promise.all(entries.map(async ([key, file]) => { - const bytes = await fs.readFile(file) - return [key, hash(bytes)] - })) + const hashes = await Promise.all( + entries.map(async ([key, file]) => { + const bytes = await fs.readFile(file) + return [key, hash(bytes)] + }) + ) return Object.fromEntries(hashes) } async function canonicalNew(file) { const absolute = path.resolve(file) try { - await fs.lstat(absolute); throw new Error(`Destination already exists: ${absolute}`) + await fs.lstat(absolute) + throw new Error(`Destination already exists: ${absolute}`) } catch (error) { if (error.code !== 'ENOENT') { throw error } } - return path.join(await fs.realpath(path.dirname(absolute)), path.basename(absolute)) + return path.join( + await fs.realpath(path.dirname(absolute)), + path.basename(absolute) + ) } function outside(target, roots) { for (const root of roots) { - if (target === root || target.startsWith(root + path.sep) || root.startsWith(target + path.sep)) { - throw new Error(`Overlapping source/destination: ${target} and ${root}`) + if ( + target === root || + target.startsWith(root + path.sep) || + root.startsWith(target + path.sep) + ) { + throw new Error( + `Overlapping source/destination: ${target} and ${root}` + ) } } } @@ -53,29 +83,38 @@ async function assertFresh(plan) { if (hash(JSON.stringify(files)) !== plan.fingerprint) { throw new Error('Stale recovery plan: source inventory changed') } - if (JSON.stringify(await codeHashes(plan.code)) !== JSON.stringify(plan.codeHashes)) { + if ( + JSON.stringify(await codeHashes(plan.code)) !== + JSON.stringify(plan.codeHashes) + ) { throw new Error('Stale recovery plan: selected code changed') } } -export async function planRecovery(options, {quiescent = false} = {}) { +export async function planRecovery(options, { quiescent = false } = {}) { const report = await inspectStore(options) const code = deployment(options) const commands = from(report.commands) - const candidates = commands.where(anyOf( - {status: anyOf('accepted', 'active')}, - {status: 'done', present: false} - )) - const uncertain = commands.where(anyOf( - {condition: 'corrupt'}, - {present: Boolean, status: not('done')} - )) + const candidates = commands.where( + anyOf( + { status: anyOf('accepted', 'active') }, + { status: 'done', present: false } + ) + ) + const uncertain = commands.where( + anyOf( + { condition: 'corrupt' }, + { present: Boolean, status: not('done') } + ) + ) const blockedCandidates = candidates.where({ laterDatasets: datasets => datasets.length > 0 }) const blocks = [...report.errors] for (const command of uncertain) { - blocks.push(`${command.id}: existing uncertain dataset requires diagnosis`) + blocks.push( + `${command.id}: existing uncertain dataset requires diagnosis` + ) } for (const command of blockedCandidates) { const laterIds = command.laterDatasets.join(',') @@ -84,7 +123,7 @@ export async function planRecovery(options, {quiescent = false} = {}) { const rerun = [...candidates.select(_.id)] const summaries = commands.select(record => { - const summary = {...record} + const summary = { ...record } delete summary.command return summary }) @@ -102,26 +141,38 @@ export async function planRecovery(options, {quiescent = false} = {}) { committed: report.committed, commands: [...summaries], warnings: report.warnings, - attestation: 'Approval asserts source quiescence, complete authoritative history, complete logged command inputs, original command meaning, and assessed external effects.' + attestation: + 'Approval asserts source quiescence, complete authoritative history, complete logged command inputs, original command meaning, and assessed external effects.' } } export async function writePlan(file, plan) { const target = await canonicalNew(file) - const roots = await Promise.all(mutableDirectories(plan.config).map(dir => fs.realpath(dir))) + const roots = await Promise.all( + mutableDirectories(plan.config).map(dir => fs.realpath(dir)) + ) outside(target, roots) await publishFile(target, JSON.stringify(plan, null, 2) + '\n') } async function copyStore(config, files, target) { - const directories = [...new Set(Object.keys(files).map(file => path.dirname(file)))].sort() + const directories = [ + ...new Set(Object.keys(files).map(file => path.dirname(file))) + ].sort() let common = directories[0] - while (!directories.every(dir => dir === common || dir.startsWith(common + path.sep) || common === path.parse(common).root)) { + while ( + !directories.every( + dir => + dir === common || + dir.startsWith(common + path.sep) || + common === path.parse(common).root + ) + ) { common = path.dirname(common) } - const mapFile = file => path.join(target, path.relative(common,file)) + const mapFile = file => path.join(target, path.relative(common, file)) for (const directory of directories) { - await durableMkdir(path.join(target,path.relative(common,directory))) + await durableMkdir(path.join(target, path.relative(common, directory))) } - for (const [file,digest] of Object.entries(files)) { + for (const [file, digest] of Object.entries(files)) { if (digest === null) { continue } @@ -129,54 +180,112 @@ async function copyStore(config, files, target) { if (hash(bytes) !== digest) { throw new Error(`Source changed while copying: ${file}`) } - await publishFile(mapFile(file), bytes, {mode:(await fs.stat(file)).mode & 0o777}) + await publishFile(mapFile(file), bytes, { + mode: (await fs.stat(file)).mode & 0o777 + }) + } + const result = { + ...config, + datafile: mapFile(config.datafile), + commandLog: mapFile(config.commandLog), + commandStatus: mapFile(config.commandStatus), + integrityFile: mapFile(config.integrityFile), + requiredFiles: config.requiredFiles.map(mapFile) } - const result = {...config, datafile:mapFile(config.datafile), commandLog:mapFile(config.commandLog), - commandStatus:mapFile(config.commandStatus), integrityFile:mapFile(config.integrityFile), - requiredFiles:config.requiredFiles.map(mapFile)} if (config.schemaFile) { result.schemaFile = mapFile(config.schemaFile) } return result } async function prepareWorkspace(target, audit, roots) { - target = await canonicalNew(target); audit = await canonicalNew(audit) - outside(target, roots); outside(audit, roots); outside(target,[audit]) - await durableMkdir(target); await durableMkdir(audit) - return {target,audit} + target = await canonicalNew(target) + audit = await canonicalNew(audit) + outside(target, roots) + outside(audit, roots) + outside(target, [audit]) + await durableMkdir(target) + await durableMkdir(audit) + return { target, audit } } -export async function applyRecovery(plan, {to, auditDir, approveRerun = [], operator, reason} = {}) { +export async function applyRecovery( + plan, + { to, auditDir, approveRerun = [], operator, reason } = {} +) { if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) { - throw new Error('Plan is not actionable; inspect a quiescent source and resolve blocking evidence') + throw new Error( + 'Plan is not actionable; inspect a quiescent source and resolve blocking evidence' + ) } if (!operator || !reason) { throw new Error('Operator and assessment reason are required') } if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) { - throw new Error('Explicit approval must match the complete ordered rerun list') + throw new Error( + 'Explicit approval must match the complete ordered rerun list' + ) } const sourceOwner = await acquireOwnership(mutableDirectories(plan.config)) let candidateOwner try { await assertFresh(plan) // Do not trust editable plan fields to bypass the source predicate. - const fresh = await planRecovery({...plan.config,...plan.code}, {quiescent:true}) - if (!fresh.actionable || JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun)) { - throw new Error('Source no longer satisfies ordered recovery predicate') - } - const workspace = await prepareWorkspace(to, auditDir, sourceOwner.directories) - const {target,audit} = workspace - const rootOwner = await acquireOwnership([target],{purpose:'recovery',auditDir:audit}) - // A root lock protects partially copied candidates before layout is complete. - const original = await copyStore(plan.config, plan.files, path.join(audit,'original')) + const fresh = await planRecovery( + { ...plan.config, ...plan.code }, + { quiescent: true } + ) + if ( + !fresh.actionable || + JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun) + ) { + throw new Error( + 'Source no longer satisfies ordered recovery predicate' + ) + } + const workspace = await prepareWorkspace( + to, + auditDir, + sourceOwner.directories + ) + const { target, audit } = workspace + const rootOwner = await acquireOwnership([target], { + purpose: 'recovery', + auditDir: audit + }) + // A root lock protects partially copied candidates before layout is + // complete. + const original = await copyStore( + plan.config, + plan.files, + path.join(audit, 'original') + ) const config = await copyStore(plan.config, plan.files, target) - candidateOwner = await acquireOwnership(mutableDirectories(config).filter(dir=>dir!==target),{ancestorToken:rootOwner.token}) + candidateOwner = await acquireOwnership( + mutableDirectories(config).filter(dir => dir !== target), + { ancestorToken: rootOwner.token } + ) const planHash = hash(JSON.stringify(plan)) - await publishFile(path.join(audit,'plan.json'), JSON.stringify(plan,null,2)) - await publishFile(path.join(audit,'original.json'), JSON.stringify(original,null,2)) - const journal = path.join(audit,'attempts.jsonl') - await appendRecord(journal, JSON.stringify({event:'authorized',planHash,operator,reason,rerun:plan.rerun,config,attestation:plan.attestation})) + await publishFile( + path.join(audit, 'plan.json'), + JSON.stringify(plan, null, 2) + ) + await publishFile( + path.join(audit, 'original.json'), + JSON.stringify(original, null, 2) + ) + const journal = path.join(audit, 'attempts.jsonl') + await appendRecord( + journal, + JSON.stringify({ + event: 'authorized', + planHash, + operator, + reason, + rerun: plan.rerun, + config, + attestation: plan.attestation + }) + ) const inspection = await inspectStore(config) if (inspection.errors.length) { throw new Error(inspection.errors.join('; ')) @@ -186,86 +295,197 @@ export async function applyRecovery(plan, {to, auditDir, approveRerun = [], oper for (const bytes of inspection.buffers) { data = parser.parse(bytes) } - // Serialize committed prefix, rebuilding parser indexes without custom hooks. - let meta = {index:{id:new Map()}, data:path.dirname(config.datafile), parts:inspection.committed.length} - const buffers = [serialize(data,{meta})] + // Serialize committed prefix, rebuilding parser indexes without custom + // hooks. + let meta = { + index: { id: new Map() }, + data: path.dirname(config.datafile), + parts: inspection.committed.length + } + const buffers = [serialize(data, { meta })] if (config.schemaFile) { - meta.schema = JSONTag.parse(await fs.readFile(config.schemaFile,'utf8')) + meta.schema = JSONTag.parse( + await fs.readFile(config.schemaFile, 'utf8') + ) + } + let expectedManifest = null + if (plan.files[plan.config.integrityFile] != null) { + expectedManifest = await loadIntegrityManifest( + plan.config.integrityFile + ) } - const expectedManifest = plan.files[plan.config.integrityFile] != null ? await loadIntegrityManifest(plan.config.integrityFile) : null for (const id of plan.rerun) { await assertFresh(plan) const command = fresh.commands.find(c => c.id === id) const active = nextActiveCommandStatus(id, command.history.at(-1)) - await appendRecord(journal, JSON.stringify({event:'attempt',id,attempt:active.attempt,planHash})) + await appendRecord( + journal, + JSON.stringify({ + event: 'attempt', + id, + attempt: active.attempt, + planHash + }) + ) await appendRecord(config.commandStatus, JSONTag.stringify(active)) - const result = await executeWorker(plan.code.commandWorker, {id,command:command.line,meta,data:buffers, - commandsFile:plan.code.commandsFile,indexFile:plan.code.indexFile,datafile:config.datafile, - // Verify a retained digest before allowing replacement manifest/done records. - integrityFile:null, integrityRequired:false}, 30000) - if (!result || result.storageFailure || result.status === 'failed' || result.status === 'unsafe' || result.code >= 300) { - await appendRecord(journal, JSON.stringify({event:'stopped',id,result:result && {status:result.status,message:result.message}})) - throw new Error(`Recovery attempt ${id} uncertain or failed; inspect before approving another attempt`) + const result = await executeWorker( + plan.code.commandWorker, + { + id, + command: command.line, + meta, + data: buffers, + commandsFile: plan.code.commandsFile, + indexFile: plan.code.indexFile, + datafile: config.datafile, + // Verify a retained digest before allowing replacement + // manifest/done records. + integrityFile: null, + integrityRequired: false + }, + 30000 + ) + if ( + !result || + result.storageFailure || + result.status === 'failed' || + result.status === 'unsafe' || + result.code >= 300 + ) { + await appendRecord( + journal, + JSON.stringify({ + event: 'stopped', + id, + result: result && { + status: result.status, + message: result.message + } + }) + ) + throw new Error( + `Recovery attempt ${id} uncertain or failed; inspect before approving another attempt` + ) } for (const file of config.requiredFiles) { await syncFile(file) } if (expectedManifest && command.status === 'done') { - verifyIntegrity(expectedManifest, plan.config.integrityFile, command.file, result.data, {required:true}) + verifyIntegrity( + expectedManifest, + plan.config.integrityFile, + command.file, + result.data, + { required: true } + ) } if (config.integrity || expectedManifest) { - const {appendIntegrityRecord} = await import('./integrity.mjs') - const targetFile = path.join(path.dirname(config.datafile),path.basename(command.file)) - await appendIntegrityRecord(config.integrityFile,targetFile,result.data) + const { appendIntegrityRecord } = await import( + './integrity.mjs' + ) + const targetFile = path.join( + path.dirname(config.datafile), + path.basename(command.file) + ) + await appendIntegrityRecord( + config.integrityFile, + targetFile, + result.data + ) } - await appendRecord(config.commandStatus, JSONTag.stringify({command:id,code:200,status:'done'})) - buffers.push(result.data); Object.assign(meta,result.meta) - await appendRecord(journal, JSON.stringify({event:'done',id,attempt:active.attempt})) + await appendRecord( + config.commandStatus, + JSONTag.stringify({ command: id, code: 200, status: 'done' }) + ) + buffers.push(result.data) + Object.assign(meta, result.meta) + await appendRecord( + journal, + JSON.stringify({ event: 'done', id, attempt: active.attempt }) + ) } const complete = await inspectStore(config) if (!complete.ready) { - const problems = from(complete.commands).where({problem: Boolean}) - const diagnostic = {errors: complete.errors, commands: [...problems]} + const problems = from(complete.commands).where({ problem: Boolean }) + const diagnostic = { + errors: complete.errors, + commands: [...problems] + } const details = JSON.stringify(diagnostic) throw new Error(`Recovered candidate is not ready: ${details}`) } - const report = {kind:'simplystore-recovery-complete',planHash,config,code:plan.code,codeHashes:plan.codeHashes,files:complete.files, - fingerprint:complete.fingerprint,committed:complete.committed,operator,reason,warnings:complete.warnings} - await publishFile(path.join(audit,'complete.json'), JSON.stringify(report,null,2)) - await candidateOwner.release(); candidateOwner = null + const report = { + kind: 'simplystore-recovery-complete', + planHash, + config, + code: plan.code, + codeHashes: plan.codeHashes, + files: complete.files, + fingerprint: complete.fingerprint, + committed: complete.committed, + operator, + reason, + warnings: complete.warnings + } + await publishFile( + path.join(audit, 'complete.json'), + JSON.stringify(report, null, 2) + ) + await candidateOwner.release() + candidateOwner = null await rootOwner.release() return report } finally { - // On failure retain candidate locks and its audit. The unchanged source can be released. + // On failure retain candidate locks and its audit. The unchanged source + // can be released. await sourceOwner.release() } } -export async function backupStore(options, {to, quiescent = false} = {}) { +export async function backupStore(options, { to, quiescent = false } = {}) { if (!quiescent) { throw new Error('Backup requires a stopped/quiescent source') } - const config = storePaths(options), owner = await acquireOwnership(mutableDirectories(config)) + const config = storePaths(options), + owner = await acquireOwnership(mutableDirectories(config)) try { const report = await inspectStore(config) if (!report.ready) { throw new Error('Backup source is not a complete validated store') } - const target = await canonicalNew(to); outside(target, owner.directories) + const target = await canonicalNew(to) + outside(target, owner.directories) await durableMkdir(target) const backupOwner = await acquireOwnership([target]) - const copied = await copyStore(config, report.files, path.join(target,'store')) + const copied = await copyStore( + config, + report.files, + path.join(target, 'store') + ) const check = await inspectStore(copied) if (!check.ready) { throw new Error('Backup reconstruction failed') } - if (hash(JSON.stringify(await inventory(config))) !== report.fingerprint) { + if ( + hash(JSON.stringify(await inventory(config))) !== report.fingerprint + ) { throw new Error('Backup source changed') } - const manifest = {kind:'simplystore-backup-complete',root:target,config:copied,files:check.files,fingerprint:check.fingerprint, - committed:check.committed,sourceFingerprint:report.fingerprint,warnings:report.warnings} - await publishFile(path.join(target,'complete.json'),JSON.stringify(manifest,null,2)) + const manifest = { + kind: 'simplystore-backup-complete', + root: target, + config: copied, + files: check.files, + fingerprint: check.fingerprint, + committed: check.committed, + sourceFingerprint: report.fingerprint, + warnings: report.warnings + } + await publishFile( + path.join(target, 'complete.json'), + JSON.stringify(manifest, null, 2) + ) await backupOwner.release() return manifest } @@ -300,68 +520,129 @@ function backupCoverage(retained, restored) { } const missing = from(retained.commands) - .where({accepted: true}) + .where({ accepted: true }) .where(not(isCovered)) .select(_.id) - return {sameBase, missingFromBackup: [...missing]} + return { sameBase, missingFromBackup: [...missing] } } -export async function restoreBackup(backupDirectory, {to, auditDir, source, sourceQuiescent=false} = {}) { +export async function restoreBackup( + backupDirectory, + { to, auditDir, source, sourceQuiescent = false } = {} +) { const backup = await fs.realpath(backupDirectory) - const manifest = JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) + const manifest = JSON.parse( + await fs.readFile(path.join(backup, 'complete.json'), 'utf8') + ) if (manifest.kind !== 'simplystore-backup-complete') { throw new Error('Backup is incomplete or unrecognized') } - const oldRoot=manifest.root - if(typeof oldRoot!=='string' || !path.isAbsolute(oldRoot) || manifest.fingerprint!==hash(JSON.stringify(manifest.files))){ + const oldRoot = manifest.root + if ( + typeof oldRoot !== 'string' || + !path.isAbsolute(oldRoot) || + manifest.fingerprint !== hash(JSON.stringify(manifest.files)) + ) { throw new Error('Invalid backup inventory') } - const rebase=file=>{ - if(typeof file!=='string' || file!==path.resolve(file) || !file.startsWith(oldRoot+path.sep)){ + const rebase = file => { + if ( + typeof file !== 'string' || + file !== path.resolve(file) || + !file.startsWith(oldRoot + path.sep) + ) { throw new Error('Backup manifest points outside backup') } - return path.join(backup,path.relative(oldRoot,file)) - } - manifest.files=Object.fromEntries(Object.entries(manifest.files).map(([file,digest])=>[rebase(file),digest])) - manifest.fingerprint=hash(JSON.stringify(manifest.files)) - for(const key of ['datafile','commandLog','commandStatus','integrityFile','schemaFile']){ - if(manifest.config[key]){ - manifest.config[key]=rebase(manifest.config[key]) + return path.join(backup, path.relative(oldRoot, file)) + } + manifest.files = Object.fromEntries( + Object.entries(manifest.files).map(([file, digest]) => [ + rebase(file), + digest + ]) + ) + manifest.fingerprint = hash(JSON.stringify(manifest.files)) + for (const key of [ + 'datafile', + 'commandLog', + 'commandStatus', + 'integrityFile', + 'schemaFile' + ]) { + if (manifest.config[key]) { + manifest.config[key] = rebase(manifest.config[key]) } } - manifest.config.requiredFiles=manifest.config.requiredFiles.map(rebase) - if (mutableDirectories(manifest.config).some(dir=>dir!==backup && !dir.startsWith(backup+path.sep))) { + manifest.config.requiredFiles = manifest.config.requiredFiles.map(rebase) + if ( + mutableDirectories(manifest.config).some( + dir => dir !== backup && !dir.startsWith(backup + path.sep) + ) + ) { throw new Error('Backup configuration points outside backup') } if (source && !sourceQuiescent) { throw new Error('Source comparison requires a stopped/quiescent source') } - const owner = await acquireOwnership([...mutableDirectories(manifest.config),...(source ? mutableDirectories(storePaths(source)) : [])]) + const directories = mutableDirectories(manifest.config) + if (source) { + const sourceConfig = storePaths(source) + directories.push(...mutableDirectories(sourceConfig)) + } + const owner = await acquireOwnership(directories) try { const report = await inspectStore(manifest.config) if (!report.ready || report.fingerprint !== manifest.fingerprint) { throw new Error('Backup contents differ from completed manifest') } - const {target,audit} = await prepareWorkspace(to,auditDir,[backup,...owner.directories]) - const rootOwner = await acquireOwnership([target],{purpose:'restore',auditDir:audit}) - const config = await copyStore(manifest.config,manifest.files,target) - const candidate = await acquireOwnership(mutableDirectories(config).filter(dir=>dir!==target),{ancestorToken:rootOwner.token}) + const { target, audit } = await prepareWorkspace(to, auditDir, [ + backup, + ...owner.directories + ]) + const rootOwner = await acquireOwnership([target], { + purpose: 'restore', + auditDir: audit + }) + const config = await copyStore(manifest.config, manifest.files, target) + const candidate = await acquireOwnership( + mutableDirectories(config).filter(dir => dir !== target), + { ancestorToken: rootOwner.token } + ) const check = await inspectStore(config) if (!check.ready) { throw new Error('Restored candidate failed validation') } - let missingFromBackup = null, sameBase = null + let missingFromBackup = null, + sameBase = null if (source) { const retained = await inspectStore(source) const coverage = backupCoverage(retained, check) sameBase = coverage.sameBase missingFromBackup = coverage.missingFromBackup } - const complete = {kind:'simplystore-restore-complete',config,files:check.files,fingerprint:check.fingerprint, - committed:check.committed,missingFromBackup,sameBase,warnings:check.warnings, - coverage:source ? 'Compared with supplied retained history; completeness still requires external evidence' : 'Loss relative to current source is unknown'} - await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) - await candidate.release(); await rootOwner.release() + let coverage = 'Loss relative to current source is unknown' + if (source) { + coverage = + 'Compared with supplied retained history; ' + + 'completeness still requires external evidence' + } + const complete = { + kind: 'simplystore-restore-complete', + config, + files: check.files, + fingerprint: check.fingerprint, + committed: check.committed, + missingFromBackup, + sameBase, + warnings: check.warnings, + coverage + } + await publishFile( + path.join(audit, 'complete.json'), + JSON.stringify(complete, null, 2) + ) + await candidate.release() + await rootOwner.release() return complete } finally { @@ -369,114 +650,212 @@ export async function restoreBackup(backupDirectory, {to, auditDir, source, sour } } -// Explicit offline operation: no PID-age or liveness heuristic can authorize it. -export async function releaseOfflineLocks(options, {operator, reason, confirmedStopped = false} = {}) { +// Explicit offline operation: no PID-age or liveness heuristic can authorize +// it. +export async function releaseOfflineLocks( + options, + { operator, reason, confirmedStopped = false } = {} +) { if (!confirmedStopped || !operator || !reason) { - throw new Error('Offline release requires confirmed stopped writer, operator, and reason') + throw new Error( + 'Offline release requires confirmed stopped writer, operator, and reason' + ) } - const config = storePaths(options), report = await inspectStore(config) + const config = storePaths(options), + report = await inspectStore(config) const removed = [] - const lockDirectories = new Set(await Promise.all(mutableDirectories(config).map(dir=>fs.realpath(dir)))) + const lockDirectories = new Set( + await Promise.all( + mutableDirectories(config).map(dir => fs.realpath(dir)) + ) + ) for (const directory of [...lockDirectories]) { - for (let parent=path.dirname(directory); path.dirname(parent)!==parent; parent=path.dirname(parent)) { + for ( + let parent = path.dirname(directory); + path.dirname(parent) !== parent; + parent = path.dirname(parent) + ) { try { - await fs.access(path.join(parent,'.simplystore-lock')); lockDirectories.add(parent) + await fs.access(path.join(parent, '.simplystore-lock')) + lockDirectories.add(parent) } - catch(error) { - if(error.code!=='ENOENT'){ + catch (error) { + if (error.code !== 'ENOENT') { throw error } } } } for (const directory of [...lockDirectories].sort().reverse()) { - const lock = path.join(directory,'.simplystore-lock') + const lock = path.join(directory, '.simplystore-lock') let entries try { entries = await fs.readdir(lock) } catch (error) { if (error.code === 'ENOENT') { - continue; - } throw error + continue + } + throw error } - if (entries.some(entry => entry !== 'owner.json' && !entry.endsWith('.tmp'))) { + if ( + entries.some( + entry => entry !== 'owner.json' && !entry.endsWith('.tmp') + ) + ) { throw new Error(`Unknown lock contents: ${lock}`) } const saved = [] for (const entry of entries) { - saved.push({name:entry,bytes:(await fs.readFile(path.join(lock,entry))).toString('base64')}) + saved.push({ + name: entry, + bytes: (await fs.readFile(path.join(lock, entry))).toString( + 'base64' + ) + }) } - const ownerRecord=saved.find(item=>item.name==='owner.json') - const owner=ownerRecord ? JSON.parse(Buffer.from(ownerRecord.bytes,'base64').toString()) : null - if (owner?.purpose==='recovery' && report.ready) { + const ownerRecord = saved.find(item => item.name === 'owner.json') + let owner = null + if (ownerRecord) { + const ownerBytes = Buffer.from(ownerRecord.bytes, 'base64') + owner = JSON.parse(ownerBytes.toString()) + } + if (owner?.purpose === 'recovery' && report.ready) { let completed try { - completed=JSON.parse(await fs.readFile(path.join(owner.auditDir,'complete.json'),'utf8')) + completed = JSON.parse( + await fs.readFile( + path.join(owner.auditDir, 'complete.json'), + 'utf8' + ) + ) } catch { /* Missing report is not completion evidence. */ } - if(completed?.fingerprint!==report.fingerprint) { - throw new Error('Completed-looking recovery candidate requires finish with its retained audit before unlocking') + if (completed?.fingerprint !== report.fingerprint) { + throw new Error( + 'Completed-looking recovery candidate requires finish with its retained audit before unlocking' + ) } } - removed.push({lock,ownerFiles:saved}) + removed.push({ lock, ownerFiles: saved }) } - // Return preview only; the CLI persists its external audit before calling finish. + // Return preview only; the CLI persists its external audit before calling + // finish. const commands = report.commands.map(command => { - return {id: command.id, status: command.status, problem: command.problem} + return { + id: command.id, + status: command.status, + problem: command.problem + } }) - const inspection = {ready: report.ready, errors: report.errors, commands} - return {inspection, - operator,reason,removed,async finish() { - for (const {lock,ownerFiles} of removed) { + const inspection = { ready: report.ready, errors: report.errors, commands } + return { + inspection, + operator, + reason, + removed, + async finish() { + for (const { lock, ownerFiles } of removed) { for (const entry of ownerFiles) { - if ((await fs.readFile(path.join(lock,entry.name))).toString('base64') !== entry.bytes) { + if ( + ( + await fs.readFile(path.join(lock, entry.name)) + ).toString('base64') !== entry.bytes + ) { throw new Error('Lock changed after preview') } } for (const entry of ownerFiles) { - await fs.unlink(path.join(lock,entry.name)) + await fs.unlink(path.join(lock, entry.name)) } - await fs.rmdir(lock); await syncDirectory(path.dirname(lock)) + await fs.rmdir(lock) + await syncDirectory(path.dirname(lock)) } - }} + } + } } -export async function finishRecovery(auditDirectory, {operator,reason,confirmedStopped=false}={}) { - if(!confirmedStopped || !operator || !reason){ - throw new Error('Finishing requires confirmed stopped writer and administrator assessment') - } - const audit=await fs.realpath(auditDirectory) - const plan=JSON.parse(await fs.readFile(path.join(audit,'plan.json'),'utf8')) - const authorization=JSON.parse((await fs.readFile(path.join(audit,'attempts.jsonl'),'utf8')).split('\n')[0]) - if(authorization.planHash!==hash(JSON.stringify(plan))){ +export async function finishRecovery( + auditDirectory, + { operator, reason, confirmedStopped = false } = {} +) { + if (!confirmedStopped || !operator || !reason) { + throw new Error( + 'Finishing requires confirmed stopped writer and administrator assessment' + ) + } + const audit = await fs.realpath(auditDirectory) + const plan = JSON.parse( + await fs.readFile(path.join(audit, 'plan.json'), 'utf8') + ) + const authorization = JSON.parse( + (await fs.readFile(path.join(audit, 'attempts.jsonl'), 'utf8')).split( + '\n' + )[0] + ) + if (authorization.planHash !== hash(JSON.stringify(plan))) { throw new Error('Audit authorization does not match retained plan') } await assertFresh(plan) - const report=await inspectStore(authorization.config) - if(!report.ready || plan.rerun.some(id=>!report.committed.includes(id))){ - throw new Error('Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun') - } - const complete={kind:'simplystore-recovery-complete',planHash:authorization.planHash,config:report.config, - code:plan.code,codeHashes:plan.codeHashes,files:report.files,fingerprint:report.fingerprint,committed:report.committed,operator,reason,warnings:report.warnings} - await publishFile(path.join(audit,'complete.json'),JSON.stringify(complete,null,2)) - const release=await releaseOfflineLocks(report.config,{operator,reason,confirmedStopped}) - await publishFile(path.join(audit,'finish-unlock.json'),JSON.stringify(release)) + const report = await inspectStore(authorization.config) + if ( + !report.ready || + plan.rerun.some(id => !report.committed.includes(id)) + ) { + throw new Error( + 'Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun' + ) + } + const complete = { + kind: 'simplystore-recovery-complete', + planHash: authorization.planHash, + config: report.config, + code: plan.code, + codeHashes: plan.codeHashes, + files: report.files, + fingerprint: report.fingerprint, + committed: report.committed, + operator, + reason, + warnings: report.warnings + } + await publishFile( + path.join(audit, 'complete.json'), + JSON.stringify(complete, null, 2) + ) + const release = await releaseOfflineLocks(report.config, { + operator, + reason, + confirmedStopped + }) + await publishFile( + path.join(audit, 'finish-unlock.json'), + JSON.stringify(release) + ) await release.finish() return complete } export async function verifyCandidate(report) { - if (!['simplystore-recovery-complete','simplystore-restore-complete'].includes(report.kind)) { + if ( + ![ + 'simplystore-recovery-complete', + 'simplystore-restore-complete' + ].includes(report.kind) + ) { throw new Error('Expected completed recovery/restore report') } const current = await inspectStore(report.config) if (!current.ready || current.fingerprint !== report.fingerprint) { throw new Error('Candidate changed or is incomplete') } - if (report.code && JSON.stringify(await codeHashes(report.code)) !== JSON.stringify(report.codeHashes)) { + if ( + report.code && + JSON.stringify(await codeHashes(report.code)) !== + JSON.stringify(report.codeHashes) + ) { throw new Error('Selected code changed after recovery') } return current diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 399087d..a602493 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -1,8 +1,12 @@ import JSONTag from '@muze-nl/jsontag' -import {getIndex, resultSet, isChanged} from '@muze-nl/od-jsontag/src/symbols.mjs' +import { + getIndex, + resultSet, + isChanged +} from '@muze-nl/od-jsontag/src/symbols.mjs' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' -import {publishFile as writeFileAtomic, storageError} from './storage.mjs' +import { publishFile as writeFileAtomic, storageError } from './storage.mjs' import { faultPoint } from './faults.mjs' import { appendIntegrityRecord } from './integrity.mjs' import { finalizeIndex } from './index.mjs' @@ -14,23 +18,25 @@ let dataspace let datafile, basefile, extension, integrityFile let meta = {} let metaProxy = { - index: { - } + index: {} } const parser = new Parser() parser.immutable = false export const metaIdProxy = { - forEach: (callback) => { - parser.meta.index.id.forEach((ref,id) => { - callback({ - deref: () => { - return resultArr[ref] - } - },id) + forEach: callback => { + parser.meta.index.id.forEach((ref, id) => { + callback( + { + deref: () => { + return resultArr[ref] + } + }, + id + ) }) }, - set: (id,ref) => { + set: (id, ref) => { if (!parser.meta.index.id.has(id)) { if (ref[getIndex]) { parser.meta.index.id.set(id, ref[getIndex]) @@ -44,9 +50,9 @@ export const metaIdProxy = { resultArr[line] = ref } }, - get: (id) => { + get: id => { let index = parser.meta.index.id.get(id) - if (index || index===0) { + if (index || index === 0) { return { deref: () => { return resultArr[index] @@ -54,7 +60,7 @@ export const metaIdProxy = { } } }, - has: (id) => { + has: id => { return parser.meta.index.id.has(id) } } @@ -70,7 +76,7 @@ export async function initialize(task) { if (task.meta) { parser.meta = task.meta } - for(let jsontag of task.data) { + for (let jsontag of task.data) { dataspace = parser.parse(jsontag) } resultArr = dataspace[resultSet] @@ -82,7 +88,8 @@ export async function initialize(task) { datafile = task.datafile integrityFile = task.integrityFile extension = datafile.split('.').pop() - basefile = datafile.substring(0, datafile.length - (extension.length + 1)) //+1 for . character + // Include the dot before the extension. + basefile = datafile.substring(0, datafile.length - (extension.length + 1)) commands = await import(task.commandsFile).then(mod => { return mod.default }) @@ -107,7 +114,8 @@ export default async function runCommand(commandStr) { if (commands[task.name]) { let time = Date.now() await commands[task.name](dataspace, task, undefined, metaProxy) - //TODO: if command/task makes no changes, skip updating data.jsontag and writing it, skip response.data + // TODO: if command/task makes no changes, skip updating + // data.jsontag and writing it; skip response.data. const changes = meta.resultArray.filter(e => e[isChanged]) //FIXME: new entities should also report isChanged = true @@ -115,39 +123,46 @@ export default async function runCommand(commandStr) { changes.uuid = task.id await index.update(dataspace, meta, changes) } - const uint8sab = serialize(dataspace, {meta, changes: true}) // serialize only changes + // Serialize only changes. + const uint8sab = serialize(dataspace, { meta, changes: true }) response.data = uint8sab response.meta = { index: { id: meta.index.id } } - //TODO: write data every x commands or x minutes, in seperate thread? + // TODO: write data every x commands or x minutes, + // in a separate thread? let newfilename = basefile + '.' + task.id + '.' + extension publishing = true await faultPoint('before-command-changeset-write') await writeFileAtomic(newfilename, uint8sab) - // Final bytes include new records and mutations made by the custom index hook. + // Final bytes include new records and mutations made by the custom + // index hook. await finalizeIndex(index, uint8sab, meta, task.id) if (integrityFile) { - await appendIntegrityRecord(integrityFile, newfilename, uint8sab) + await appendIntegrityRecord( + integrityFile, + newfilename, + uint8sab + ) } await faultPoint('after-command-changeset-write') meta.parts++ response.meta.parts = meta.parts let end = Date.now() - console.log('task time',end-time) + console.log('task time', end - time) } else { console.error('Command not found', task.name) throw { code: 404, - message: "Command "+task.name+" not found" + message: 'Command ' + task.name + ' not found' } } } - catch(err) { + catch (err) { console.error('task error', err) throw publishing ? storageError(err) : err } diff --git a/src/command-worker.mjs b/src/command-worker.mjs index 6827b9e..aa98509 100644 --- a/src/command-worker.mjs +++ b/src/command-worker.mjs @@ -8,8 +8,12 @@ parentPort.on('message', async data => { result = await runCommand(data.command) } catch (error) { - result = {status: 'failed', code: 500, message: error.message, - storageFailure: Boolean(error.storageFailure)} + result = { + status: 'failed', + code: 500, + message: error.message, + storageFailure: Boolean(error.storageFailure) + } } parentPort.postMessage(result) -}) \ No newline at end of file +}) diff --git a/src/execute-worker.mjs b/src/execute-worker.mjs index 35ceaee..75d0f9d 100644 --- a/src/execute-worker.mjs +++ b/src/execute-worker.mjs @@ -1,9 +1,10 @@ -import {Worker} from 'node:worker_threads' +import { Worker } from 'node:worker_threads' export function executeWorker(filename, task, timeout = 30000) { return new Promise((resolve, reject) => { const worker = new Worker(filename) - let settled = false, timer + let settled = false, + timer const finish = async (settle, result) => { if (settled) { return @@ -14,7 +15,8 @@ export function executeWorker(filename, task, timeout = 30000) { await worker.terminate() } catch (error) { - reject(error); return + reject(error) + return } settle(result) } @@ -26,13 +28,21 @@ export function executeWorker(filename, task, timeout = 30000) { }) worker.on('exit', code => { if (!settled) { - void finish(reject, new Error(`Command worker exited without a result (${code})`)) + void finish( + reject, + new Error( + `Command worker exited without a result (${code})` + ) + ) } }) if (timeout) { timer = setTimeout(() => { - void finish(resolve, - {status: 'unsafe', code: 504, message: `command worker timed out after ${timeout}ms`}) + void finish(resolve, { + status: 'unsafe', + code: 504, + message: `command worker timed out after ${timeout}ms` + }) }, timeout) } try { diff --git a/src/index.id.mjs b/src/index.id.mjs index ada8552..6383758 100644 --- a/src/index.id.mjs +++ b/src/index.id.mjs @@ -1,12 +1,13 @@ -import {publishFileSync} from './storage.mjs' +import { publishFileSync } from './storage.mjs' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' import { getIndex } from '@muze-nl/od-jsontag/src/symbols.mjs' export default { create(data, meta) { - console.log('creating '+meta.data+'/index.id.json') - // jsontag parse automatically fills meta.index.id, so no need to create anything + console.log('creating ' + meta.data + '/index.id.json') + // jsontag parse automatically fills meta.index.id, so no need to create + // anything // just store meta.index.id in index.id.json const index = {} for (const key of meta.index.id.keys()) { @@ -19,7 +20,7 @@ export default { } index[key] = entity[getIndex] } - publishFileSync(meta.data+'/index.id.json', JSON.stringify(index)) + publishFileSync(meta.data + '/index.id.json', JSON.stringify(index)) }, update(data, meta, changes) { if (!changes.length) { @@ -32,16 +33,19 @@ export default { index[id] = entry[getIndex] } } - publishFileSync(meta.data+'/index.id.'+changes.uuid+'.json', JSON.stringify(index)) + publishFileSync( + meta.data + '/index.id.' + changes.uuid + '.json', + JSON.stringify(index) + ) }, - load(uuid=null) { + load(uuid = null) { let filename if (!uuid) { filename = 'index.id.json' } else { - filename = 'index.id.'+filename+'.json' + filename = 'index.id.' + filename + '.json' } - return JSON.parse(fs.readFileSync(meta.data+filename)) + return JSON.parse(fs.readFileSync(meta.data + filename)) } } diff --git a/src/index.offset.mjs b/src/index.offset.mjs index 3fabbff..99b6c7d 100644 --- a/src/index.offset.mjs +++ b/src/index.offset.mjs @@ -1,31 +1,51 @@ -import {publishFileSync} from './storage.mjs' +import { publishFileSync } from './storage.mjs' import fs from 'fs' import path from 'node:path' import { Buffer } from 'node:buffer' -import {publishFile as writeFileAtomic} from './storage.mjs' +import { publishFile as writeFileAtomic } from './storage.mjs' import { getIndex, position } from '@muze-nl/od-jsontag/src/symbols.mjs' import { scanOdJsonTagRecords } from './recovery.mjs' export default { async writeSerialized(buffer, meta, uuid = null) { - const filename = path.join(meta.data, uuid === null ? 'index.offset.json' : `index.offset.${uuid}.json`) - const bytes = typeof buffer === 'string' ? Buffer.from(buffer) : Buffer.from(buffer.buffer, buffer.byteOffset, buffer.byteLength) + let name = 'index.offset.json' + if (uuid !== null) { + name = `index.offset.${uuid}.json` + } + const filename = path.join(meta.data, name) + let bytes + if (typeof buffer === 'string') { + bytes = Buffer.from(buffer) + } + else { + bytes = Buffer.from( + buffer.buffer, + buffer.byteOffset, + buffer.byteLength + ) + } const index = {} - scanOdJsonTagRecords(bytes, filename, 'offset source OD-JSONTag data', (record, start, end) => { - index[record] = [start, end] - }) + scanOdJsonTagRecords( + bytes, + filename, + 'offset source OD-JSONTag data', + (record, start, end) => { + index[record] = [start, end] + } + ) await writeFileAtomic(filename, JSON.stringify(index)) }, create(data, meta) { - console.log('creating '+meta.data+'/index.offset.json') - // jsontag parse automatically fills meta.index.offset, so no need to create anything + console.log('creating ' + meta.data + '/index.offset.json') + // jsontag parse automatically fills meta.index.offset, so no need to + // create anything const index = {} const max = meta.resultArray.length - for (let i=0;i command?.status === activeCommandStatus)) { - throw new RecoveryIntegrityError('Active command requires administrator assessment; automatic rerun is disabled') + if ( + [...status.values()].some( + command => command?.status === activeCommandStatus + ) + ) { + throw new RecoveryIntegrityError( + 'Active command requires administrator assessment; automatic rerun is disabled' + ) } return status } @@ -135,27 +176,44 @@ export function nextActiveCommandStatus(commandId, currentStatus) { export function getChangesetPath(dataFile, commandId) { const extension = dataFile.split('.').pop() - const basefile = dataFile.substring(0, dataFile.length - (extension.length + 1)) + const basefile = dataFile.substring( + 0, + dataFile.length - (extension.length + 1) + ) return `${basefile}.${commandId}.${extension}` } export function assertChangesetExists(dataFile, commandId) { const changesetPath = getChangesetPath(dataFile, commandId) if (!fs.existsSync(changesetPath)) { - throw new Error(`Missing changeset for committed command ${commandId}: ${changesetPath}`) + throw new Error( + `Missing changeset for committed command ${commandId}: ${changesetPath}` + ) } return changesetPath } -export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag data') { +export function assertOdJsonTagFraming( + buffer, + file, + recordKind = 'OD-JSONTag data' +) { scanOdJsonTagRecords(buffer, file, recordKind) } -export function scanOdJsonTagRecords(buffer, file, recordKind = 'OD-JSONTag data', onRecord) { +export function scanOdJsonTagRecords( + buffer, + file, + recordKind = 'OD-JSONTag data', + onRecord +) { let offset = 0 let record = 0 while (offset < buffer.length) { - while (offset < buffer.length && (buffer[offset] === 10 || buffer[offset] === 13)) { + while ( + offset < buffer.length && + (buffer[offset] === 10 || buffer[offset] === 13) + ) { offset++ } if (offset >= buffer.length) { @@ -164,43 +222,74 @@ export function scanOdJsonTagRecords(buffer, file, recordKind = 'OD-JSONTag data if (buffer[offset] === 43) { offset++ const skipStart = offset - while (offset < buffer.length && buffer[offset] >= 48 && buffer[offset] <= 57) { + while ( + offset < buffer.length && + buffer[offset] >= 48 && + buffer[offset] <= 57 + ) { offset++ } - if (offset === skipStart || (offset < buffer.length && buffer[offset] !== 10 && buffer[offset] !== 13)) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: malformed skip record`, { - file, - recordKind - }) + if ( + offset === skipStart || + (offset < buffer.length && + buffer[offset] !== 10 && + buffer[offset] !== 13) + ) { + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: malformed skip record`, + { + file, + recordKind + } + ) } - record += Number.parseInt(buffer.subarray(skipStart, offset).toString('utf8'), 10) + record += Number.parseInt( + buffer.subarray(skipStart, offset).toString('utf8'), + 10 + ) continue } if (buffer[offset] !== 40) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: expected record length`, { - file, - recordKind - }) + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: expected record length`, + { + file, + recordKind + } + ) } offset++ const lengthStart = offset - while (offset < buffer.length && buffer[offset] >= 48 && buffer[offset] <= 57) { + while ( + offset < buffer.length && + buffer[offset] >= 48 && + buffer[offset] <= 57 + ) { offset++ } if (offset === lengthStart || buffer[offset] !== 41) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: malformed record length`, { - file, - recordKind - }) + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: malformed record length`, + { + file, + recordKind + } + ) } - const payloadLength = Number.parseInt(buffer.subarray(lengthStart, offset).toString('utf8'), 10) + const payloadLength = Number.parseInt( + buffer.subarray(lengthStart, offset).toString('utf8'), + 10 + ) offset++ const payloadEnd = offset + payloadLength if (payloadEnd > buffer.length) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: truncated record payload`, { - file, - recordKind - }) + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: truncated record payload`, + { + file, + recordKind + } + ) } onRecord?.(record++, offset, payloadEnd) offset = payloadEnd diff --git a/src/server.mjs b/src/server.mjs index c7f6f15..51d4929 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -1,20 +1,25 @@ -import {setImmediate} from 'node:timers' +import { setImmediate } from 'node:timers' import express from 'express' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' -import {from, _, anyOf, not} from '@muze-nl/jaqt' +import { from, _, anyOf, not } from '@muze-nl/jaqt' import WorkerPool from './workerPool.mjs' import { Worker } from 'worker_threads' import { fileURLToPath } from 'url' -import {appendFile} from './util.mjs' +import { appendFile } from './util.mjs' import path from 'path' import httpStatusCodes from './statusCodes.mjs' import process from 'node:process' import { nextActiveCommandStatus } from './recovery.mjs' -import {serialWriter, syncFile} from './storage.mjs' -import {inspectStore, storePaths, mutableDirectories, validCommandId} from './store-inspection.mjs' -import {acquireOwnership} from './store-ownership.mjs' -import {executeWorker} from './execute-worker.mjs' +import { serialWriter, syncFile } from './storage.mjs' +import { + inspectStore, + storePaths, + mutableDirectories, + validCommandId +} from './store-inspection.mjs' +import { acquireOwnership } from './store-ownership.mjs' +import { executeWorker } from './execute-worker.mjs' import { assertRuntimeEnvironmentConfiguration } from './runtime-environment.mjs' import { faultPoint } from './faults.mjs' import { getDefaultIntegrityFile } from './integrity.mjs' @@ -37,12 +42,19 @@ class WorkerTimeoutError extends Error { } function normalizeWorkerTimeout(name, value, defaultValue) { - if (value === 0 || value === false || value === null || value === Infinity) { + if ( + value === 0 || + value === false || + value === null || + value === Infinity + ) { return 0 } const timeout = value ?? defaultValue if (!Number.isFinite(timeout) || timeout < 0) { - throw new Error(`${name} must be a non-negative finite number, 0, false, null, or Infinity`) + throw new Error( + `${name} must be a non-negative finite number, 0, false, null, or Infinity` + ) } return timeout } @@ -52,36 +64,63 @@ async function main(options) { if (!options) { options = {} } - const port = options.port || 3000 - const datafile = options.datafile || './data.od-jsontag' - const schemaFile = options.schemaFile || null - const wwwroot = options.wwwroot || __dirname+'/www' - const maxWorkers = options.maxWorkers || 8 - const queryWorker = options.queryWorker || __dirname+'/src/query-worker.mjs' - const loadWorker = options.loadWorker || __dirname+'/src/load-worker.mjs' - const commandWorker = options.commandWorker || __dirname+'/src/command-worker.mjs' - const commandsFile = options.commandsFile || __dirname+'/src/commands.mjs' - const indexFile = options.indexFile || __dirname+'/src/index.mjs' - const commandLog = options.commandLog || './command-log.jsontag' + const port = options.port || 3000 + const datafile = options.datafile || './data.od-jsontag' + const schemaFile = options.schemaFile || null + const wwwroot = options.wwwroot || __dirname + '/www' + const maxWorkers = options.maxWorkers || 8 + const queryWorker = + options.queryWorker || __dirname + '/src/query-worker.mjs' + const loadWorker = options.loadWorker || __dirname + '/src/load-worker.mjs' + const commandWorker = + options.commandWorker || __dirname + '/src/command-worker.mjs' + const commandsFile = options.commandsFile || __dirname + '/src/commands.mjs' + const indexFile = options.indexFile || __dirname + '/src/index.mjs' + const commandLog = options.commandLog || './command-log.jsontag' const commandStatus = options.commandStatus || './command-status.jsontag' - const integrityFile = options.integrityFile || getDefaultIntegrityFile(datafile) - const integrityEnabled = Boolean(options.integrity || options.integrityFile || fs.existsSync(integrityFile)) - const access = options.access || null - const timeout = options.timeout || 1000 - const slowTimeout = options.slowTimeout || 10000 - const commandTimeout = normalizeWorkerTimeout('commandTimeout', options.commandTimeout, defaultCommandTimeout) - const loadTimeout = normalizeWorkerTimeout('loadTimeout', options.loadTimeout, defaultLoadTimeout) + const integrityFile = + options.integrityFile || getDefaultIntegrityFile(datafile) + const integrityEnabled = Boolean( + options.integrity || + options.integrityFile || + fs.existsSync(integrityFile) + ) + const activeIntegrityFile = integrityEnabled ? integrityFile : null + const access = options.access || null + const timeout = options.timeout || 1000 + const slowTimeout = options.slowTimeout || 10000 + const commandTimeout = normalizeWorkerTimeout( + 'commandTimeout', + options.commandTimeout, + defaultCommandTimeout + ) + const loadTimeout = normalizeWorkerTimeout( + 'loadTimeout', + options.loadTimeout, + defaultLoadTimeout + ) server.get('/', serveHomepage) // allow access to raw body, used to parse a query send as post body - server.use(express.raw({ - type: () => true, // parse body on all requests - limit: '50MB' - })) + server.use( + express.raw({ + type: () => true, // parse body on all requests + limit: '50MB' + }) + ) - const config = storePaths({...options, datafile, commandLog, commandStatus, integrity: integrityEnabled, - ...(integrityEnabled ? {integrityFile} : {})}) + const storeOptions = { + ...options, + datafile, + commandLog, + commandStatus, + integrity: integrityEnabled + } + if (integrityEnabled) { + storeOptions.integrityFile = integrityFile + } + const config = storePaths(storeOptions) const ownership = await acquireOwnership(mutableDirectories(config)) let inspection try { @@ -90,17 +129,17 @@ async function main(options) { const commands = from(inspection.commands) const problems = commands - .where({problem: Boolean}) + .where({ problem: Boolean }) .select(_.problem) const pending = commands - .where({status: anyOf('accepted', 'active')}) + .where({ status: anyOf('accepted', 'active') }) .select(command => { return `${command.id}: ${command.status}; administrator assessment required` }) const uncommitted = commands - .where({present: Boolean, status: not('done')}) + .where({ present: Boolean, status: not('done') }) .select(command => { return `${command.id}: uncommitted dataset` }) @@ -128,18 +167,25 @@ async function main(options) { await ownership.release() throw error } - const status = new Map(inspection.commands.map(c => [c.id, c.history.at(-1)])) + const status = new Map( + inspection.commands.map(c => [c.id, c.history.at(-1)]) + ) const commandQueue = [] const acceptSerial = serialWriter() - let storageFailed = false, closing = false + let storageFailed = false, + closing = false let runner = Promise.resolve() function failStorage(error) { if (storageFailed) { return } storageFailed = true - console.error('Storage outcome uncertain; stopping mutation. Administrator recovery required:', error) - // Retain ownership evidence; a process exit must never imply a rollback. + console.error( + 'Storage outcome uncertain; stopping mutation. Administrator recovery required:', + error + ) + // Retain ownership evidence; a process exit must never imply a + // rollback. setImmediate(() => process.exit(1)) } @@ -161,25 +207,37 @@ async function main(options) { return result } - let queryWorkerPool = new WorkerPool(maxWorkers, queryWorker, queryWorkerInitTask()) - let slowQueryWorkerPool = new WorkerPool(1, queryWorker, slowQueryWorkerInitTask()) + let queryWorkerPool = new WorkerPool( + maxWorkers, + queryWorker, + queryWorkerInitTask() + ) + let slowQueryWorkerPool = new WorkerPool( + 1, + queryWorker, + slowQueryWorkerInitTask() + ) let commandRunnerActive = false - server.get('/query/', (req,next) => handleGetQuery(req,next)) - server.get('/query/*remainder', (req,next) => handleGetQuery(req,next)) - server.get('/slowquery/', (req,next) => handleSlowGetQuery(req,next)) - server.get('/slowquery/*remainder', (req,next) => handleSlowGetQuery(req,next)) - server.post('/query/', (req,next) => handlePostQuery(req,next)) - server.post('/query/*remainder', (req,next) => handlePostQuery(req,next)) - server.post('/slowquery/', (req,next) => handleSlowPostQuery(req,next)) - server.post('/slowquery/*remainder', (req,next) => handleSlowPostQuery(req,next)) - server.post('/command', (req,next) => handlePostCommand(req,next)) - server.get('/command/:id', (req,next) => handleGetCommand(req,next)) + server.get('/query/', (req, next) => handleGetQuery(req, next)) + server.get('/query/*remainder', (req, next) => handleGetQuery(req, next)) + server.get('/slowquery/', (req, next) => handleSlowGetQuery(req, next)) + server.get('/slowquery/*remainder', (req, next) => + handleSlowGetQuery(req, next) + ) + server.post('/query/', (req, next) => handlePostQuery(req, next)) + server.post('/query/*remainder', (req, next) => handlePostQuery(req, next)) + server.post('/slowquery/', (req, next) => handleSlowPostQuery(req, next)) + server.post('/slowquery/*remainder', (req, next) => + handleSlowPostQuery(req, next) + ) + server.post('/command', (req, next) => handlePostCommand(req, next)) + server.get('/command/:id', (req, next) => handleGetCommand(req, next)) server.use(express.static(wwwroot)) const listener = server.listen(port, () => { - console.log('SimplyStore listening on port '+port) + console.log('SimplyStore listening on port ' + port) }) listener.on('error', error => { failStorage(error) @@ -195,7 +253,8 @@ async function main(options) { }) await runner - queryWorkerPool.close(); slowQueryWorkerPool.close() + queryWorkerPool.close() + slowQueryWorkerPool.close() if (!storageFailed) { await ownership.release() } @@ -211,12 +270,12 @@ async function main(options) { /* ------ */ function serveHomepage(req, res) { - res.setHeader('content-type', 'text/html'); - res.send(fs.readFileSync(wwwroot+'/home.html')) + res.setHeader('content-type', 'text/html') + res.send(fs.readFileSync(wwwroot + '/home.html')) } function loadData(commands) { - return new Promise((resolve,reject) => { + return new Promise((resolve, reject) => { let worker = new Worker(loadWorker) let settled = false let timeoutId @@ -233,7 +292,10 @@ async function main(options) { } if (loadTimeout) { timeoutId = setTimeout(() => { - const error = new WorkerTimeoutError('load worker', loadTimeout) + const error = new WorkerTimeoutError( + 'load worker', + loadTimeout + ) void finish(reject, error) }, loadTimeout) } @@ -245,11 +307,11 @@ async function main(options) { }) try { worker.postMessage({ - dataFile:datafile, + dataFile: datafile, indexFile, schemaFile, commands, - integrityFile: integrityEnabled ? integrityFile : null, + integrityFile: activeIntegrityFile, integrityRequired: integrityEnabled }) } @@ -259,56 +321,66 @@ async function main(options) { }) } - async function handleGetQuery(req, res, pool=null) { + async function handleGetQuery(req, res, pool = null) { let start = Date.now() if (!pool) { pool = queryWorkerPool } - if ( !accept(req,res, - ['application/jsontag','application/json','text/html','text/javascript','image/*'], - function(req, res, accept) { - let result = true - switch(accept) { - case 'text/html': - case 'image/*': - case 'text/javascript': - handleWebRequest(req,res,{root:wwwroot}); - result = false - break + if ( + !accept( + req, + res, + [ + 'application/jsontag', + 'application/json', + 'text/html', + 'text/javascript', + 'image/*' + ], + function (req, res, accept) { + let result = true + switch (accept) { + case 'text/html': + case 'image/*': + case 'text/javascript': + handleWebRequest(req, res, { root: wwwroot }) + result = false + break + } + return result } - return result - } - )) { + ) + ) { // done return } let path = req.params.remainder?.join('/') || '/' - console.log('query',path) + console.log('query', path) let request = { method: req.method, url: req.originalUrl, query: req.query, path: path } - if (accept(req,res,['application/jsontag'])) { + if (accept(req, res, ['application/jsontag'])) { request.jsontag = true } try { let result = await pool.run('query', request, { timeout }) sendResponse(result, res) } - catch(error) { + catch (error) { sendError(error, res) } let end = Date.now() - console.log(path, (end-start), process.memoryUsage()) + console.log(path, end - start, process.memoryUsage()) } async function handleSlowGetQuery(req, res) { return handleGetQuery(req, res, slowQueryWorkerPool) } - async function handlePostQuery(req,res,pool=null, slowTimeout=null) { + async function handlePostQuery(req, res, pool = null, slowTimeout = null) { if (!pool) { pool = queryWorkerPool } @@ -316,10 +388,15 @@ async function main(options) { slowTimeout = timeout } let start = Date.now() - if ( !accept(req,res, - ['application/jsontag','application/json']) - ) { - sendError({code:406, message:'Not Acceptable',accept:['application/json','application/jsontag']},res) + if (!accept(req, res, ['application/jsontag', 'application/json'])) { + sendError( + { + code: 406, + message: 'Not Acceptable', + accept: ['application/json', 'application/jsontag'] + }, + res + ) return } let path = req.params.remainder?.join('/') || '/' @@ -330,18 +407,18 @@ async function main(options) { path: path, body: req.body.toString() } - if (accept(req,res,['application/jsontag'])) { + if (accept(req, res, ['application/jsontag'])) { request.jsontag = true } try { - let result = await pool.run('query', request, {slowTimeout}) + let result = await pool.run('query', request, { slowTimeout }) sendResponse(result, res) } - catch(error) { + catch (error) { sendError(error, res) } let end = Date.now() - console.log(path, (end-start), process.memoryUsage()) + console.log(path, end - start, process.memoryUsage()) // queryWorkerPool.memoryUsage() } @@ -351,7 +428,13 @@ async function main(options) { async function handlePostCommand(req, res) { if (storageFailed || closing) { - return sendResponse({code:503, body: JSON.stringify({message:'Store is unavailable'})}, res) + return sendResponse( + { + code: 503, + body: JSON.stringify({ message: 'Store is unavailable' }) + }, + res + ) } try { await acceptSerial(async () => { @@ -359,7 +442,15 @@ async function main(options) { throw new Error('Store is unavailable') } if (closing) { - return sendResponse({code:503,body:JSON.stringify({message:'Store is closing'})},res) + return sendResponse( + { + code: 503, + body: JSON.stringify({ + message: 'Store is closing' + }) + }, + res + ) } const accepted = await checkCommand(req, res) if (!accepted) { @@ -368,14 +459,28 @@ async function main(options) { if (storageFailed) { throw new Error('Store failed during acceptance') } - commandQueue.push({id: accepted.id, command: accepted.line}) - sendResponse({code:202, body:JSON.stringify(status.get(accepted.id))}, res) + commandQueue.push({ id: accepted.id, command: accepted.line }) + sendResponse( + { + code: 202, + body: JSON.stringify(status.get(accepted.id)) + }, + res + ) }) void drainCommandQueue() } catch (error) { if (!res.headersSent) { - sendResponse({code:500, body:JSON.stringify({message:'Storage outcome uncertain'})}, res) + sendResponse( + { + code: 500, + body: JSON.stringify({ + message: 'Storage outcome uncertain' + }) + }, + res + ) } failStorage(error) } @@ -384,17 +489,27 @@ async function main(options) { function handleGetCommand(req, res) { if (status.has(req.params.id)) { let result = status.get(req.params.id) - sendResponse({ - jsontag: false, - body: JSON.stringify(result) - },res) + sendResponse( + { + jsontag: false, + body: JSON.stringify(result) + }, + res + ) } else { - sendResponse({ - code: 404, - jsontag: false, - body: JSON.stringify({code: 404, message: "Command not found", details: req.params.id}) - }, res) + sendResponse( + { + code: 404, + jsontag: false, + body: JSON.stringify({ + code: 404, + message: 'Command not found', + details: req.params.id + }) + }, + res + ) } } @@ -408,27 +523,61 @@ async function main(options) { while (commandQueue.length && !storageFailed) { const command = commandQueue.shift() console.log('starting command', command.id) - const active = nextActiveCommandStatus(command.id, status.get(command.id)) + const active = nextActiveCommandStatus( + command.id, + status.get(command.id) + ) await appendFile(commandStatus, JSONTag.stringify(active)) if (storageFailed) { throw new Error('Store failed before execution') } status.set(command.id, active) - await faultPoint('after-active-status-before-command-worker') - const result = await executeWorker(commandWorker, {...command, - meta, data: jsontagBuffers, commandsFile, indexFile, datafile, - integrityFile: integrityEnabled ? integrityFile : null, - integrityRequired: integrityEnabled}, commandTimeout) + await faultPoint( + 'after-active-status-before-command-worker' + ) + const result = await executeWorker( + commandWorker, + { + ...command, + meta, + data: jsontagBuffers, + commandsFile, + indexFile, + datafile, + integrityFile: activeIntegrityFile, + integrityRequired: integrityEnabled + }, + commandTimeout + ) if (storageFailed) { throw new Error('Store failed during execution') } if (result?.storageFailure) { - throw new Error(result.message || 'Worker persistence failure') + throw new Error( + result.message || 'Worker persistence failure' + ) } - if (!result || result.status === 'failed' || result.status === 'unsafe' || result.code >= 300) { - const terminal = {command: command.id, status: result?.status === 'unsafe' ? 'unsafe' : 'failed', - code: result?.code || 500, message: result?.message || 'Command failed', attempt: active.attempt} - await appendFile(commandStatus, JSONTag.stringify(terminal)) + if ( + !result || + result.status === 'failed' || + result.status === 'unsafe' || + result.code >= 300 + ) { + let terminalStatus = 'failed' + if (result?.status === 'unsafe') { + terminalStatus = 'unsafe' + } + const terminal = { + command: command.id, + status: terminalStatus, + code: result?.code || 500, + message: result?.message || 'Command failed', + attempt: active.attempt + } + await appendFile( + commandStatus, + JSONTag.stringify(terminal) + ) status.set(command.id, terminal) continue } @@ -436,15 +585,25 @@ async function main(options) { await syncFile(file) } await faultPoint('before-command-done-status') - const done = {command: command.id, code:200, status:'done'} + const done = { + command: command.id, + code: 200, + status: 'done' + } await appendFile(commandStatus, JSONTag.stringify(done)) - await faultPoint('after-command-done-status-before-query-update') + await faultPoint( + 'after-command-done-status-before-query-update' + ) status.set(command.id, done) if (result.data) { jsontagBuffers.push(result.data) Object.assign(meta, result.meta) - const task = {name:'update', req:{body:result.data, meta}} - queryWorkerPool.update(task); slowQueryWorkerPool.update(task) + const task = { + name: 'update', + req: { body: result.data, meta } + } + queryWorkerPool.update(task) + slowQueryWorkerPool.update(task) } } } @@ -470,37 +629,40 @@ async function main(options) { status: 'accepted' } } - catch(err) { + catch (err) { error = { code: 400, - message: "Bad request", + message: 'Bad request', details: err.message } - console.error('Error: ',err) - sendResponse({code: 400, body: JSON.stringify(error)}, res) + console.error('Error: ', err) + sendResponse({ code: 400, body: JSON.stringify(error) }, res) return false } if (!command || !validCommandId(command.id)) { error = { code: 422, - message: "Command has no id", + message: 'Command has no id', details: command } - sendResponse({code: 422, body: JSON.stringify(error)}, res) + sendResponse({ code: 422, body: JSON.stringify(error) }, res) return false } else if (status.has(command.id)) { - const currentStatus = Object.assign({command: command.id}, status.get(command.id)) - sendResponse({body: JSON.stringify(currentStatus)}, res) + const currentStatus = Object.assign( + { command: command.id }, + status.get(command.id) + ) + sendResponse({ body: JSON.stringify(currentStatus) }, res) return false } else if (!command.name) { error = { code: 422, - message: "Command has no name", + message: 'Command has no name', details: command } - sendResponse({code:422, body: JSON.stringify(error)}, res) + sendResponse({ code: 422, body: JSON.stringify(error) }, res) return false } const line = JSONTag.stringify(command) @@ -509,7 +671,7 @@ async function main(options) { await appendFile(commandStatus, JSONTag.stringify(commandOK)) status.set(command.id, commandOK) await faultPoint('after-command-accepted-status-before-response') - return {id:command.id, line} + return { id: command.id, line } } } @@ -518,12 +680,12 @@ function sendResponse(response, res) { res.status(response.code) } if (response.jsontag) { - res.setHeader('content-type','application/jsontag') + res.setHeader('content-type', 'application/jsontag') } else { - res.setHeader('content-type','application/json') + res.setHeader('content-type', 'application/json') } - res.send(response.body)+"\n" + res.send(response.body) + '\n' } function sendError(error, res) { @@ -534,7 +696,7 @@ function sendError(error, res) { else { res.status(500) } - res.setHeader('content-type','application/json') + res.setHeader('content-type', 'application/json') res.send(JSON.stringify(error)) } @@ -545,7 +707,7 @@ function accept(req, res, mimetypes, handler) { let accept = req.accepts(mimetypes) if (!accept) { res.status(406) - res.send("

406 Unacceptable

\n") + res.send('

406 Unacceptable

\n') return false } if (typeof handler === 'function') { @@ -554,20 +716,20 @@ function accept(req, res, mimetypes, handler) { return true } -function handleWebRequest(req,res,options) { - let path = req.path; +function handleWebRequest(req, res, options) { + let path = req.path path = path.replace(/[^a-z0-9_.\-/]*/gi, '') // whitelist acceptable file paths path = path.replace(/\.+/g, '.') // blacklist '..' if (!path) { path = '/' } - if (path.substring(path.length-1)==='/') { + if (path.substring(path.length - 1) === '/') { path += 'index.html' } const fileOptions = { root: options.root } - if (fs.existsSync(fileOptions.root+path)) { + if (fs.existsSync(fileOptions.root + path)) { res.sendFile(path, fileOptions) } else { diff --git a/src/storage.mjs b/src/storage.mjs index 1aaed60..7a81070 100644 --- a/src/storage.mjs +++ b/src/storage.mjs @@ -1,8 +1,8 @@ import fs from 'node:fs' import fsp from 'node:fs/promises' import path from 'node:path' -import {randomUUID} from 'node:crypto' -import {Buffer} from 'node:buffer' +import { randomUUID } from 'node:crypto' +import { Buffer } from 'node:buffer' export function storageError(error) { if (!(error instanceof Error)) { @@ -52,8 +52,17 @@ export async function writeAll(handle, data) { const bytes = Buffer.isBuffer(data) ? data : Buffer.from(data) let offset = 0 while (offset < bytes.length) { - const {bytesWritten} = await handle.write(bytes, offset, bytes.length - offset, null) - if (!Number.isInteger(bytesWritten) || bytesWritten <= 0 || bytesWritten > bytes.length - offset) { + const { bytesWritten } = await handle.write( + bytes, + offset, + bytes.length - offset, + null + ) + if ( + !Number.isInteger(bytesWritten) || + bytesWritten <= 0 || + bytesWritten > bytes.length - offset + ) { throw storageError(new Error('Invalid or zero-progress write')) } offset += bytesWritten @@ -97,7 +106,7 @@ export async function durableMkdir(directory) { await syncDirectory(path.dirname(directory)) } -export async function publishFile(file, data, {mode} = {}) { +export async function publishFile(file, data, { mode } = {}) { const temporary = `${file}.${randomUUID()}.tmp` let renamed = false try { @@ -113,13 +122,18 @@ export async function publishFile(file, data, {mode} = {}) { throw error } } - await withHandle(temporary, 'wx', async handle => { - if (mode !== undefined) { - await handle.chmod(mode) - } - await writeAll(handle, data) - await handle.sync() - }, mode) + await withHandle( + temporary, + 'wx', + async handle => { + if (mode !== undefined) { + await handle.chmod(mode) + } + await writeAll(handle, data) + await handle.sync() + }, + mode + ) await fsp.rename(temporary, file) renamed = true await syncDirectory(path.dirname(file)) @@ -152,7 +166,9 @@ export function appendRecord(file, record) { // Existing synchronous index hooks keep their contract while gaining barriers. export function publishFileSync(file, data) { const temporary = `${file}.${randomUUID()}.tmp` - let fd, failure, renamed = false + let fd, + failure, + renamed = false try { const bytes = Buffer.from(data) let mode @@ -170,12 +186,16 @@ export function publishFileSync(file, data) { } fd = fs.openSync(temporary, 'wx', mode) if (mode !== undefined) { - fs.fchmodSync(fd,mode) + fs.fchmodSync(fd, mode) } let offset = 0 while (offset < bytes.length) { const count = fs.writeSync(fd, bytes, offset, bytes.length - offset) - if (!Number.isInteger(count) || count <= 0 || count > bytes.length - offset) { + if ( + !Number.isInteger(count) || + count <= 0 || + count > bytes.length - offset + ) { throw new Error('Invalid or zero-progress write') } offset += count diff --git a/src/store-inspection.mjs b/src/store-inspection.mjs index 092a27a..5168bd8 100644 --- a/src/store-inspection.mjs +++ b/src/store-inspection.mjs @@ -1,42 +1,100 @@ import fs from 'node:fs/promises' import path from 'node:path' import JSONTag from '@muze-nl/jsontag' -import {from, _, anyOf, not} from '@muze-nl/jaqt' +import { from, _, anyOf, not } from '@muze-nl/jaqt' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' -import {createHash} from 'node:crypto' -import {assertOdJsonTagFraming, getChangesetPath} from './recovery.mjs' -import {getDefaultIntegrityFile, loadIntegrityManifest, verifyIntegrity} from './integrity.mjs' +import { createHash } from 'node:crypto' +import { assertOdJsonTagFraming, getChangesetPath } from './recovery.mjs' +import { + getDefaultIntegrityFile, + loadIntegrityManifest, + verifyIntegrity +} from './integrity.mjs' export const hash = bytes => createHash('sha256').update(bytes).digest('hex') -export const validCommandId = id => typeof id === 'string' && id.length > 0 && !/[/\\\0]/.test(id) +export const validCommandId = id => + typeof id === 'string' && id.length > 0 && !/[/\\\0]/.test(id) export function storePaths(options = {}) { const datafile = path.resolve(options.datafile || './data.od-jsontag') - return {datafile, commandLog: path.resolve(options.commandLog || './command-log.jsontag'), - commandStatus: path.resolve(options.commandStatus || './command-status.jsontag'), - integrityFile: path.resolve(options.integrityFile || getDefaultIntegrityFile(datafile)), - integrity: options.integrity === undefined ? Boolean(options.integrityFile) : Boolean(options.integrity), - requiredFiles: (options.requiredFiles || []).map(file => path.resolve(file)), - ...(options.schemaFile ? {schemaFile: path.resolve(options.schemaFile)} : {})} + let integrity + if (options.integrity === undefined) { + integrity = Boolean(options.integrityFile) + } + else { + integrity = Boolean(options.integrity) + } + const config = { + datafile, + commandLog: path.resolve(options.commandLog || './command-log.jsontag'), + commandStatus: path.resolve( + options.commandStatus || './command-status.jsontag' + ), + integrityFile: path.resolve( + options.integrityFile || getDefaultIntegrityFile(datafile) + ), + integrity, + requiredFiles: (options.requiredFiles || []).map(file => + path.resolve(file) + ) + } + if (options.schemaFile) { + config.schemaFile = path.resolve(options.schemaFile) + } + return config } export function mutableDirectories(config) { - return [...new Set([config.datafile, config.commandLog, config.commandStatus, - config.integrityFile, ...config.requiredFiles].map(file => path.dirname(file)))].sort() + return [ + ...new Set( + [ + config.datafile, + config.commandLog, + config.commandStatus, + config.integrityFile, + ...config.requiredFiles + ].map(file => path.dirname(file)) + ) + ].sort() } export async function inventory(config) { const files = {} - const expected = new Set([config.datafile,config.commandLog,config.commandStatus,config.integrityFile,...config.requiredFiles,...(config.schemaFile?[config.schemaFile]:[])]) - const extension = path.extname(config.datafile), stem = path.basename(config.datafile,extension)+'.' + const artifacts = [ + config.datafile, + config.commandLog, + config.commandStatus, + config.integrityFile, + ...config.requiredFiles + ] + if (config.schemaFile) { + artifacts.push(config.schemaFile) + } + const expected = new Set(artifacts) + const extension = path.extname(config.datafile), + stem = path.basename(config.datafile, extension) + '.' for (const directory of mutableDirectories(config)) { - const entries = await fs.readdir(directory, {withFileTypes: true}) - for (const entry of entries.sort((a,b) => a.name.localeCompare(b.name))) { + const entries = await fs.readdir(directory, { withFileTypes: true }) + for (const entry of entries.sort((a, b) => + a.name.localeCompare(b.name) + )) { if (entry.name === '.simplystore-lock') { continue } const file = path.join(directory, entry.name) - const dataArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith(stem) && (entry.name.endsWith(extension) || entry.name.endsWith('.tmp')) - const indexArtifact = directory===path.dirname(config.datafile) && entry.name.startsWith('index.') - const temporary = [...expected].some(name=>file.startsWith(name+'.') && file.endsWith('.tmp')) - if (!expected.has(file) && !dataArtifact && !indexArtifact && !temporary) { + const dataArtifact = + directory === path.dirname(config.datafile) && + entry.name.startsWith(stem) && + (entry.name.endsWith(extension) || entry.name.endsWith('.tmp')) + const indexArtifact = + directory === path.dirname(config.datafile) && + entry.name.startsWith('index.') + const temporary = [...expected].some( + name => file.startsWith(name + '.') && file.endsWith('.tmp') + ) + if ( + !expected.has(file) && + !dataArtifact && + !indexArtifact && + !temporary + ) { continue } if (entry.isSymbolicLink()) { @@ -47,9 +105,15 @@ export async function inventory(config) { } } } - for (const file of [config.datafile, config.commandLog, config.commandStatus, - ...(config.integrity ? [config.integrityFile] : []), ...config.requiredFiles, - ...(config.schemaFile ? [config.schemaFile] : [])]) { + const required = [config.datafile, config.commandLog, config.commandStatus] + if (config.integrity) { + required.push(config.integrityFile) + } + required.push(...config.requiredFiles) + if (config.schemaFile) { + required.push(config.schemaFile) + } + for (const file of required) { if (!(file in files)) { try { files[file] = hash(await fs.readFile(file)) @@ -62,7 +126,9 @@ export async function inventory(config) { } } } - return Object.fromEntries(Object.entries(files).sort(([a],[b]) => a.localeCompare(b))) + return Object.fromEntries( + Object.entries(files).sort(([a], [b]) => a.localeCompare(b)) + ) } async function records(file, errors, kind) { @@ -71,7 +137,8 @@ async function records(file, errors, kind) { text = await fs.readFile(file, 'utf8') } catch (error) { - errors.push(`${kind}: ${error.message}`); return [] + errors.push(`${kind}: ${error.message}`) + return [] } if (text && !text.endsWith('\n')) { errors.push(`${kind}: incomplete final record in ${file}`) @@ -86,7 +153,7 @@ async function records(file, errors, kind) { if (!value || typeof value !== 'object' || Array.isArray(value)) { throw new Error('record is not an object') } - result.push({value, line, lineNumber: index + 1}) + result.push({ value, line, lineNumber: index + 1 }) } catch (error) { errors.push(`${kind} ${file}:${index + 1}: ${error.message}`) @@ -95,50 +162,80 @@ async function records(file, errors, kind) { return result } -// Reads canonical files only; deliberately never imports user handlers/index modules. +// Reads canonical files only; deliberately never imports user handlers/index +// modules. export async function inspectStore(options = {}) { const config = storePaths(options) - const errors = [], warnings = [] + const errors = [], + warnings = [] const configuredPaths = [ config.datafile, config.commandLog, config.commandStatus, config.integrityFile ] - const canonicalPaths = await Promise.all(configuredPaths.map(async file => { - const directory = await fs.realpath(path.dirname(file)) - return path.join(directory, path.basename(file)) - })) + const canonicalPaths = await Promise.all( + configuredPaths.map(async file => { + const directory = await fs.realpath(path.dirname(file)) + return path.join(directory, path.basename(file)) + }) + ) if (new Set(canonicalPaths).size !== canonicalPaths.length) { errors.push('Configured canonical artifact paths overlap') } const before = await inventory(config) const log = await records(config.commandLog, errors, 'command log') - const statuses = await records(config.commandStatus, errors, 'command status') - const commands = [], byId = new Map() + const statuses = await records( + config.commandStatus, + errors, + 'command status' + ) + const commands = [], + byId = new Map() for (const record of log) { - const {id, name} = record.value + const { id, name } = record.value if (!validCommandId(id) || typeof name !== 'string' || !name) { - errors.push(`Invalid command at log line ${record.lineNumber}`); continue + errors.push(`Invalid command at log line ${record.lineNumber}`) + continue } if (byId.has(id)) { - if (JSONTag.stringify(byId.get(id).command) !== JSONTag.stringify(record.value)) { + if ( + JSONTag.stringify(byId.get(id).command) !== + JSONTag.stringify(record.value) + ) { errors.push(`Conflicting command ID ${id}`) } continue } - const command = {id, position: commands.length, command: record.value, line: record.line, - history: [], accepted: false, status: null, file: getChangesetPath(config.datafile, id)} - byId.set(id, command); commands.push(command) + const command = { + id, + position: commands.length, + command: record.value, + line: record.line, + history: [], + accepted: false, + status: null, + file: getChangesetPath(config.datafile, id) + } + byId.set(id, command) + commands.push(command) } const doneOrder = new Map() - for (const {value, lineNumber} of statuses) { + for (const { value, lineNumber } of statuses) { const command = byId.get(value.command) if (!command) { - errors.push(`Status line ${lineNumber} has no logged command: ${value.command}`); continue + errors.push( + `Status line ${lineNumber} has no logged command: ${value.command}` + ) + continue } - if (!['accepted', 'active', 'done', 'failed', 'unsafe'].includes(value.status)) { - errors.push(`Unknown status for ${command.id}: ${value.status}`); continue + if ( + !['accepted', 'active', 'done', 'failed', 'unsafe'].includes( + value.status + ) + ) { + errors.push(`Unknown status for ${command.id}: ${value.status}`) + continue } if (value.status === 'accepted') { command.accepted = true @@ -147,9 +244,11 @@ export async function inspectStore(options = {}) { errors.push(`No acceptance evidence for ${command.id}`) } if (value.status === 'done') { - doneOrder.delete(command.id); doneOrder.set(command.id, command.position) + doneOrder.delete(command.id) + doneOrder.set(command.id, command.position) } - command.history.push(value); command.status = value.status + command.history.push(value) + command.status = value.status } let lastDonePosition = -1 for (const [id, position] of doneOrder) { @@ -157,17 +256,22 @@ export async function inspectStore(options = {}) { continue } if (position < lastDonePosition) { - errors.push(`Committed execution differs from command-log order at ${id}`) + errors.push( + `Committed execution differs from command-log order at ${id}` + ) } lastDonePosition = position } let manifest try { - // Preserve optional existence-based compatibility; required integrity is explicit. + // Preserve optional existence-based compatibility; required integrity + // is explicit. const enabled = config.integrity || before[config.integrityFile] != null if (enabled) { if (before[config.integrityFile] == null) { - throw new Error(`Missing integrity manifest ${config.integrityFile}`) + throw new Error( + `Missing integrity manifest ${config.integrityFile}` + ) } await records(config.integrityFile, errors, 'integrity manifest') manifest = await loadIntegrityManifest(config.integrityFile) @@ -177,13 +281,17 @@ export async function inspectStore(options = {}) { errors.push(error.message) } const parser = new Parser() - const buffers = [], committed = [] - let data, prefixValid = errors.length === 0 + const buffers = [], + committed = [] + let data, + prefixValid = errors.length === 0 async function validateData(file) { const bytes = await fs.readFile(file) assertOdJsonTagFraming(bytes, file) if (manifest) { - verifyIntegrity(manifest, config.integrityFile, file, bytes, {required: true}) + verifyIntegrity(manifest, config.integrityFile, file, bytes, { + required: true + }) } return bytes } @@ -192,10 +300,12 @@ export async function inspectStore(options = {}) { if (!bytes.length) { throw new Error('Empty base dataset') } - data = parser.parse(bytes); buffers.push(bytes) + data = parser.parse(bytes) + buffers.push(bytes) } catch (error) { - errors.push(error.message); prefixValid = false + errors.push(error.message) + prefixValid = false } for (const command of commands) { command.present = before[command.file] != null @@ -208,7 +318,8 @@ export async function inspectStore(options = {}) { await validateData(command.file) } catch (error) { - command.condition = 'corrupt'; command.problem = error.message + command.condition = 'corrupt' + command.problem = error.message } } if (command.status === 'done') { @@ -219,14 +330,20 @@ export async function inspectStore(options = {}) { else if (prefixValid) { try { const bytes = await validateData(command.file) - data = parser.parse(bytes); buffers.push(bytes); committed.push(command.id) + data = parser.parse(bytes) + buffers.push(bytes) + committed.push(command.id) } catch (error) { - command.problem = error.message; prefixValid = false + command.problem = error.message + prefixValid = false } } } - else if (command.status === 'accepted' || command.status === 'active') { + else if ( + command.status === 'accepted' || + command.status === 'active' + ) { prefixValid = false } const laterDatasets = from(commands) @@ -238,10 +355,18 @@ export async function inspectStore(options = {}) { .select(_.id) command.laterDatasets = [...laterDatasets] } - const extension = path.extname(config.datafile), stem = path.basename(config.datafile, extension) + '.' + const extension = path.extname(config.datafile), + stem = path.basename(config.datafile, extension) + '.' const known = new Set(commands.map(command => command.file)) for (const file of Object.keys(before)) { - if (path.dirname(file) === path.dirname(config.datafile) && path.basename(file).startsWith(stem) && file.endsWith(extension) && file !== config.datafile && file !== config.integrityFile && !known.has(file)) { + if ( + path.dirname(file) === path.dirname(config.datafile) && + path.basename(file).startsWith(stem) && + file.endsWith(extension) && + file !== config.datafile && + file !== config.integrityFile && + !known.has(file) + ) { errors.push(`Unexplained changeset: ${file}`) } } @@ -255,13 +380,25 @@ export async function inspectStore(options = {}) { errors.push('Store changed during inspection') } const commandQuery = from(commands) - const pending = commandQuery.where({status: anyOf('accepted', 'active')}) - const damaged = commandQuery.where(anyOf( - {problem: Boolean}, - {present: Boolean, status: not('done')} - )) - const ready = errors.length === 0 && pending.length === 0 && damaged.length === 0 - warnings.push('History completeness and original code/hidden inputs require independent administrator evidence; current files alone cannot prove them.') - return {config, files: before, fingerprint: hash(JSON.stringify(before)), errors, warnings, - commands, committed, ready, data, buffers} + const pending = commandQuery.where({ status: anyOf('accepted', 'active') }) + const damaged = commandQuery.where( + anyOf({ problem: Boolean }, { present: Boolean, status: not('done') }) + ) + const ready = + errors.length === 0 && pending.length === 0 && damaged.length === 0 + warnings.push( + 'History completeness and original code/hidden inputs require independent administrator evidence; current files alone cannot prove them.' + ) + return { + config, + files: before, + fingerprint: hash(JSON.stringify(before)), + errors, + warnings, + commands, + committed, + ready, + data, + buffers + } } diff --git a/src/store-ownership.mjs b/src/store-ownership.mjs index 07ebf1e..0ee6c5e 100644 --- a/src/store-ownership.mjs +++ b/src/store-ownership.mjs @@ -2,26 +2,42 @@ import fs from 'node:fs/promises' import path from 'node:path' import os from 'node:os' import process from 'node:process' -import {randomUUID} from 'node:crypto' -import {publishFile, syncDirectory, syncAncestors} from './storage.mjs' +import { randomUUID } from 'node:crypto' +import { publishFile, syncDirectory, syncAncestors } from './storage.mjs' -export async function acquireOwnership(directories, {ancestorToken, purpose, auditDir} = {}) { - const canonical = [...new Set(await Promise.all(directories.map(dir => fs.realpath(dir))))].sort() +export async function acquireOwnership( + directories, + { ancestorToken, purpose, auditDir } = {} +) { + const canonical = [ + ...new Set(await Promise.all(directories.map(dir => fs.realpath(dir)))) + ].sort() for (const directory of canonical) { - for (let parent = path.dirname(directory); ; parent = path.dirname(parent)) { + for ( + let parent = path.dirname(directory); + ; + parent = path.dirname(parent) + ) { if (!canonical.includes(parent)) { const lock = path.join(parent, '.simplystore-lock') try { await fs.access(lock) let owner try { - owner = JSON.parse(await fs.readFile(path.join(lock,'owner.json'),'utf8')) + owner = JSON.parse( + await fs.readFile( + path.join(lock, 'owner.json'), + 'utf8' + ) + ) } catch { /* An incomplete lock remains exclusive. */ } if (!ancestorToken || owner?.token !== ancestorToken) { - throw new Error(`Store is locked by unfinished parent operation: ${lock}`) + throw new Error( + `Store is locked by unfinished parent operation: ${lock}` + ) } } catch (error) { @@ -35,7 +51,8 @@ export async function acquireOwnership(directories, {ancestorToken, purpose, aud } } } - const token = randomUUID(), held = [] + const token = randomUUID(), + held = [] try { for (const directory of canonical) { const lock = path.join(directory, '.simplystore-lock') @@ -44,32 +61,52 @@ export async function acquireOwnership(directories, {ancestorToken, purpose, aud } catch (error) { if (error.code === 'EEXIST') { - throw new Error(`Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.`) + throw new Error( + `Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.` + ) } throw error } held.push(lock) - await publishFile(path.join(lock, 'owner.json'), JSON.stringify({token, purpose, auditDir, pid: process.pid, host: os.hostname(), started: new Date().toISOString()})) + await publishFile( + path.join(lock, 'owner.json'), + JSON.stringify({ + token, + purpose, + auditDir, + pid: process.pid, + host: os.hostname(), + started: new Date().toISOString() + }) + ) await syncAncestors(directory) } } catch (error) { - // No store writes have begun, so only our own acquired locks can be released. + // No store writes have begun, so only our own acquired locks can be + // released. for (const lock of held.reverse()) { - await fs.rm(lock, {recursive: true}); await syncDirectory(path.dirname(lock)) + await fs.rm(lock, { recursive: true }) + await syncDirectory(path.dirname(lock)) } throw error } - return {token, directories: canonical, async release() { - for (const lock of [...held].reverse()) { - const owner = JSON.parse(await fs.readFile(path.join(lock, 'owner.json'), 'utf8')) - if (owner.token !== token) { - throw new Error(`Ownership changed: ${lock}`) + return { + token, + directories: canonical, + async release() { + for (const lock of [...held].reverse()) { + const owner = JSON.parse( + await fs.readFile(path.join(lock, 'owner.json'), 'utf8') + ) + if (owner.token !== token) { + throw new Error(`Ownership changed: ${lock}`) + } + await fs.unlink(path.join(lock, 'owner.json')) + await fs.rmdir(lock) + await syncDirectory(path.dirname(lock)) } - await fs.unlink(path.join(lock, 'owner.json')) - await fs.rmdir(lock) - await syncDirectory(path.dirname(lock)) + held.length = 0 } - held.length = 0 - }} + } } diff --git a/src/util.mjs b/src/util.mjs index aa3bcea..1f3965d 100644 --- a/src/util.mjs +++ b/src/util.mjs @@ -1,5 +1,5 @@ import JSONTag from '@muze-nl/jsontag' -import {appendRecord} from './storage.mjs' +import { appendRecord } from './storage.mjs' export function deepFreeze(obj) { Object.freeze(obj) @@ -16,14 +16,16 @@ export function isString(s) { } export function joinArgs(args) { - return args = args.map(arg => { - if (isString(arg)) { - return arg - } - else { - return JSONTag.stringify(arg) - } - }).join(' ') + return (args = args + .map(arg => { + if (isString(arg)) { + return arg + } + else { + return JSONTag.stringify(arg) + } + }) + .join(' ')) } /** diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs index 5bf49bd..fa9ec53 100644 --- a/test/admin-recovery.mjs +++ b/test/admin-recovery.mjs @@ -1,281 +1,681 @@ -import {Buffer} from 'node:buffer' +import { Buffer } from 'node:buffer' import test from 'node:test' import assert from 'node:assert/strict' import fs from 'node:fs/promises' import path from 'node:path' import os from 'node:os' import JSONTag from '@muze-nl/jsontag' -import {makeServerFixture} from './durability-helpers.mjs' -import {planRecovery,applyRecovery,backupStore,restoreBackup,verifyCandidate} from '../src/admin-recovery.mjs' -import {inspectStore} from '../src/store-inspection.mjs' -import {acquireOwnership} from '../src/store-ownership.mjs' +import { makeServerFixture } from './durability-helpers.mjs' +import { + planRecovery, + applyRecovery, + backupStore, + restoreBackup, + verifyCandidate +} from '../src/admin-recovery.mjs' +import { inspectStore } from '../src/store-inspection.mjs' +import { acquireOwnership } from '../src/store-ownership.mjs' async function setup(t, entries) { - const store=await makeServerFixture(t) - const outputs=await fs.mkdtemp(path.join(os.tmpdir(),'simplystore-admin-output-')) - t.after(()=>fs.rm(outputs,{recursive:true,force:true})) - await fs.writeFile(store.commandLog,entries.map(([id])=>JSONTag.stringify({id,name:'addPerson',value:{name:id}})).join('\n')+'\n') - await fs.writeFile(store.commandStatus,entries.flatMap(([id,status])=>[{command:id,status:'accepted',code:202},...status==='accepted'?[]:[{command:id,status,code:200}]]).map(s=>JSONTag.stringify(s)).join('\n')+'\n') - return {store,outputs} + const store = await makeServerFixture(t) + const outputs = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-admin-output-') + ) + t.after(() => fs.rm(outputs, { recursive: true, force: true })) + await fs.writeFile( + store.commandLog, + entries + .map(([id]) => + JSONTag.stringify({ + id, + name: 'addPerson', + value: { name: id } + }) + ) + .join('\n') + '\n' + ) + const statuses = [] + for (const [id, status] of entries) { + statuses.push({ command: id, status: 'accepted', code: 202 }) + if (status !== 'accepted') { + statuses.push({ command: id, status, code: 200 }) + } + } + const statusLines = statuses.map(status => JSONTag.stringify(status)) + await fs.writeFile(store.commandStatus, statusLines.join('\n') + '\n') + return { store, outputs } } -async function runRecovery(store,outputs,plan) { - return applyRecovery(plan,{to:path.join(outputs,'recovered'),auditDir:path.join(outputs,'audit'),approveRerun:plan.rerun,operator:'test admin',reason:'Verified complete log and command inputs; side effects assessed'}) +async function runRecovery(store, outputs, plan) { + return applyRecovery(plan, { + to: path.join(outputs, 'recovered'), + auditDir: path.join(outputs, 'audit'), + approveRerun: plan.rerun, + operator: 'test admin', + reason: 'Verified complete log and command inputs; side effects assessed' + }) } -test('administrator recovers trailing missing/waiting commands in log order on a copy',async t=>{ - const {store,outputs}=await setup(t,[['A','done'],['B','accepted']]) - const before=await fs.readFile(store.commandStatus) - const plan=await planRecovery(store,{quiescent:true}) - assert.equal(plan.actionable,true);assert.deepEqual(plan.rerun,['A','B']) - assert.deepEqual(structuredClone(plan), plan, 'Recovery plans remain plain cloneable data') - const result=await runRecovery(store,outputs,plan) - const actual=await inspectStore(result.config) - assert.equal(actual.ready,true) - assert.deepEqual(structuredClone(actual.commands), actual.commands, 'Inspection does not expose JAQT proxies') - assert.deepEqual(actual.data.persons.map(p=>p.name),['A','B']) - assert.deepEqual(await fs.readFile(store.commandStatus),before) - assert.ok((await fs.readFile(result.config.commandStatus)).subarray(0,before.length).equals(before)) - assert.deepEqual((await fs.readFile(result.config.commandLog)).toString(),(await fs.readFile(store.commandLog)).toString()) +test('administrator recovers trailing missing/waiting commands in log order on a copy', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'done'], + ['B', 'accepted'] + ]) + const before = await fs.readFile(store.commandStatus) + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(plan.rerun, ['A', 'B']) + assert.deepEqual( + structuredClone(plan), + plan, + 'Recovery plans remain plain cloneable data' + ) + const result = await runRecovery(store, outputs, plan) + const actual = await inspectStore(result.config) + assert.equal(actual.ready, true) + assert.deepEqual( + structuredClone(actual.commands), + actual.commands, + 'Inspection does not expose JAQT proxies' + ) + assert.deepEqual( + actual.data.persons.map(p => p.name), + ['A', 'B'] + ) + assert.deepEqual(await fs.readFile(store.commandStatus), before) + assert.ok( + (await fs.readFile(result.config.commandStatus)) + .subarray(0, before.length) + .equals(before) + ) + assert.deepEqual( + (await fs.readFile(result.config.commandLog)).toString(), + (await fs.readFile(store.commandLog)).toString() + ) }) -for (const state of ['accepted','done','unsafe']) { - test(`later ${state} dataset blocks rerun, including zero-byte no-op`,async t=>{ - const {store}=await setup(t,[['A','done'],['B',state]]) - await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') - const plan=await planRecovery(store,{quiescent:true}) - assert.equal(plan.actionable,false) - assert.ok(plan.blocks.some(s=>s.includes('later accepted datasets'))) +for (const state of ['accepted', 'done', 'unsafe']) { + test(`later ${state} dataset blocks rerun, including zero-byte no-op`, async t => { + const { store } = await setup(t, [ + ['A', 'done'], + ['B', state] + ]) + await fs.writeFile(path.join(store.dir, 'data.B.jsontag'), '') + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, false) + assert.ok(plan.blocks.some(s => s.includes('later accepted datasets'))) }) } -test('new later file makes an approved plan stale before executing a handler',async t=>{ - const {store,outputs}=await setup(t,[['A','done'],['B','accepted']]) - const plan=await planRecovery(store,{quiescent:true}) - await fs.writeFile(path.join(store.dir,'data.B.jsontag'),'') - await assert.rejects(runRecovery(store,outputs,plan),/Stale recovery plan/) - await assert.rejects(fs.access(path.join(outputs,'recovered')),/ENOENT/) +test('new later file makes an approved plan stale before executing a handler', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'done'], + ['B', 'accepted'] + ]) + const plan = await planRecovery(store, { quiescent: true }) + await fs.writeFile(path.join(store.dir, 'data.B.jsontag'), '') + await assert.rejects( + runRecovery(store, outputs, plan), + /Stale recovery plan/ + ) + await assert.rejects(fs.access(path.join(outputs, 'recovered')), /ENOENT/) }) -test('changed code and missing explicit approval reject apply',async t=>{ - const {store,outputs}=await setup(t,[['A','accepted']]) - const plan=await planRecovery(store,{quiescent:true}) - await assert.rejects(applyRecovery(plan,{to:path.join(outputs,'x'),auditDir:path.join(outputs,'y'),operator:'admin',reason:'test'}),/approval/) - await fs.appendFile(store.commandsFile,'\n// changed\n') - await assert.rejects(runRecovery(store,outputs,plan),/Stale recovery plan/) +test('changed code and missing explicit approval reject apply', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const plan = await planRecovery(store, { quiescent: true }) + await assert.rejects( + applyRecovery(plan, { + to: path.join(outputs, 'x'), + auditDir: path.join(outputs, 'y'), + operator: 'admin', + reason: 'test' + }), + /approval/ + ) + await fs.appendFile(store.commandsFile, '\n// changed\n') + await assert.rejects( + runRecovery(store, outputs, plan), + /Stale recovery plan/ + ) }) -test('cooperative ownership excludes competing writer and path aliases',async t=>{ - const {store,outputs}=await setup(t,[]) - const owner=await acquireOwnership([store.dir]) +test('cooperative ownership excludes competing writer and path aliases', async t => { + const { store, outputs } = await setup(t, []) + const owner = await acquireOwnership([store.dir]) try { - await fs.symlink(store.dir,path.join(outputs,'alias')) - await assert.rejects(acquireOwnership([path.join(outputs,'alias')]),/locked/) + await fs.symlink(store.dir, path.join(outputs, 'alias')) + await assert.rejects( + acquireOwnership([path.join(outputs, 'alias')]), + /locked/ + ) } finally { await owner.release() } }) -test('backup and restore validate saved bytes without executing commands',async t=>{ - const {store,outputs}=await setup(t,[['A','done']]) - const result=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) - const backup=path.join(outputs,'backup') - await backupStore(result.config,{to:backup,quiescent:true}) - const restored=await restoreBackup(backup,{to:path.join(outputs,'restored'),auditDir:path.join(outputs,'restore-audit'),source:store,sourceQuiescent:true}) - assert.deepEqual((await inspectStore(restored.config)).data.persons.map(p=>p.name),['A']) - assert.deepEqual(restored.missingFromBackup,[]) - const saved=JSON.parse(await fs.readFile(path.join(backup,'complete.json'),'utf8')) - await fs.appendFile(saved.config.commandLog,' ') - await assert.rejects(restoreBackup(backup,{to:path.join(outputs,'bad'),auditDir:path.join(outputs,'bad-audit')}),/Backup contents/) +test('backup and restore validate saved bytes without executing commands', async t => { + const { store, outputs } = await setup(t, [['A', 'done']]) + const result = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + const backup = path.join(outputs, 'backup') + await backupStore(result.config, { to: backup, quiescent: true }) + const restored = await restoreBackup(backup, { + to: path.join(outputs, 'restored'), + auditDir: path.join(outputs, 'restore-audit'), + source: store, + sourceQuiescent: true + }) + assert.deepEqual( + (await inspectStore(restored.config)).data.persons.map(p => p.name), + ['A'] + ) + assert.deepEqual(restored.missingFromBackup, []) + const saved = JSON.parse( + await fs.readFile(path.join(backup, 'complete.json'), 'utf8') + ) + await fs.appendFile(saved.config.commandLog, ' ') + await assert.rejects( + restoreBackup(backup, { + to: path.join(outputs, 'bad'), + auditDir: path.join(outputs, 'bad-audit') + }), + /Backup contents/ + ) }) -test('read-only inspection never imports handlers and rejects conflicting done order',async t=>{ - const {store}=await setup(t,[['A','accepted'],['B','accepted']]) - await fs.writeFile(store.commandsFile,"throw new Error('must not be imported')") - const before=await fs.readFile(store.commandStatus) - const plan=await planRecovery(store,{quiescent:true}) - assert.equal(plan.actionable,true);assert.deepEqual(await fs.readFile(store.commandStatus),before) - await fs.appendFile(store.commandStatus,[{command:'B',status:'done'},{command:'A',status:'done'}].map(JSON.stringify).join('\n')+'\n') - assert.ok((await planRecovery(store,{quiescent:true})).blocks.some(s=>s.includes('differs from command-log order'))) +test('read-only inspection never imports handlers and rejects conflicting done order', async t => { + const { store } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + await fs.writeFile( + store.commandsFile, + "throw new Error('must not be imported')" + ) + const before = await fs.readFile(store.commandStatus) + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(await fs.readFile(store.commandStatus), before) + await fs.appendFile( + store.commandStatus, + [ + { command: 'B', status: 'done' }, + { command: 'A', status: 'done' } + ] + .map(JSON.stringify) + .join('\n') + '\n' + ) + assert.ok( + (await planRecovery(store, { quiescent: true })).blocks.some(s => + s.includes('differs from command-log order') + ) + ) }) -test('historical status order never makes a later log dataset eligible for bypass',async t=>{ - const {store}=await setup(t,[['A','accepted'],['B','accepted'],['C','accepted']]) - await fs.writeFile(store.commandStatus,[{command:'C',status:'accepted'},{command:'B',status:'accepted'},{command:'A',status:'accepted'}].map(JSON.stringify).join('\n')+'\n') - await fs.writeFile(path.join(store.dir,'data.C.jsontag'),'') - const plan=await planRecovery(store,{quiescent:true}) - assert.ok(plan.blocks.some(s=>s.includes('B: later accepted datasets exist: C'))) +test('historical status order never makes a later log dataset eligible for bypass', async t => { + const { store } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'], + ['C', 'accepted'] + ]) + await fs.writeFile( + store.commandStatus, + [ + { command: 'C', status: 'accepted' }, + { command: 'B', status: 'accepted' }, + { command: 'A', status: 'accepted' } + ] + .map(JSON.stringify) + .join('\n') + '\n' + ) + await fs.writeFile(path.join(store.dir, 'data.C.jsontag'), '') + const plan = await planRecovery(store, { quiescent: true }) + assert.ok( + plan.blocks.some(s => s.includes('B: later accepted datasets exist: C')) + ) }) -test('interrupted recovery preserves completed attempts and blocks an unfinished candidate',async t=>{ - const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted']]) - const witness=path.join(outputs,'effects') - await fs.writeFile(store.commandsFile,`import fs from 'node:fs';export default {addPerson(data,c){fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n');if(c.id==='B')throw new Error('external effect already happened');data.persons.push(c.value)}}`) - const plan=await planRecovery(store,{quiescent:true}) - await assert.rejects(runRecovery(store,outputs,plan),/uncertain or failed/) - const authorization=JSON.parse((await fs.readFile(path.join(outputs,'audit','attempts.jsonl'),'utf8')).split('\n')[0]) - const partial=await inspectStore(authorization.config) - assert.equal(partial.ready,false);assert.deepEqual(partial.committed,['A']) - assert.equal(partial.commands.find(c=>c.id==='B').status,'active') - assert.equal(await fs.readFile(witness,'utf8'),'A\nB\n') - await assert.rejects(acquireOwnership([path.dirname(authorization.config.datafile)]),/locked/) - await assert.rejects(fs.access(path.join(outputs,'audit','complete.json')),/ENOENT/) - const preview=await planRecovery({...authorization.config,commandsFile:store.commandsFile,indexFile:store.indexFile},{quiescent:true}) - assert.deepEqual(preview.rerun,['B'],'completed A is reconstructed, never selected for re-execution') +test('interrupted recovery preserves completed attempts and blocks an unfinished candidate', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + const witness = path.join(outputs, 'effects') + await fs.writeFile( + store.commandsFile, + `import fs from 'node:fs';export default {addPerson(data,c){fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n');if(c.id==='B')throw new Error('external effect already happened');data.persons.push(c.value)}}` + ) + const plan = await planRecovery(store, { quiescent: true }) + await assert.rejects( + runRecovery(store, outputs, plan), + /uncertain or failed/ + ) + const authorization = JSON.parse( + ( + await fs.readFile( + path.join(outputs, 'audit', 'attempts.jsonl'), + 'utf8' + ) + ).split('\n')[0] + ) + const partial = await inspectStore(authorization.config) + assert.equal(partial.ready, false) + assert.deepEqual(partial.committed, ['A']) + assert.equal(partial.commands.find(c => c.id === 'B').status, 'active') + assert.equal(await fs.readFile(witness, 'utf8'), 'A\nB\n') + await assert.rejects( + acquireOwnership([path.dirname(authorization.config.datafile)]), + /locked/ + ) + await assert.rejects( + fs.access(path.join(outputs, 'audit', 'complete.json')), + /ENOENT/ + ) + const preview = await planRecovery( + { + ...authorization.config, + commandsFile: store.commandsFile, + indexFile: store.indexFile + }, + { quiescent: true } + ) + assert.deepEqual( + preview.rerun, + ['B'], + 'completed A is reconstructed, never selected for re-execution' + ) }) -test('retained digest mismatch stops recovery before replacing integrity or appending done',async t=>{ - const {store,outputs}=await setup(t,[['A','done']]) - const {appendIntegrityRecord}=await import('../src/integrity.mjs') - const integrity=path.join(store.dir,'data.integrity.jsontag') - await appendIntegrityRecord(integrity,store.datafile,await fs.readFile(store.datafile)) - await appendIntegrityRecord(integrity,path.join(store.dir,'data.A.jsontag'),Buffer.from('different historical bytes')) - store.integrityFile=integrity - const plan=await planRecovery(store,{quiescent:true}) - await assert.rejects(runRecovery(store,outputs,plan),/Integrity mismatch/) - const authorization=JSON.parse((await fs.readFile(path.join(outputs,'audit','attempts.jsonl'),'utf8')).split('\n')[0]) - assert.deepEqual(await fs.readFile(authorization.config.integrityFile),await fs.readFile(integrity)) - const statuses=(await fs.readFile(authorization.config.commandStatus,'utf8')).trim().split('\n').map(JSONTag.parse) - assert.equal(statuses.at(-1).status,'active') +test('retained digest mismatch stops recovery before replacing integrity or appending done', async t => { + const { store, outputs } = await setup(t, [['A', 'done']]) + const { appendIntegrityRecord } = await import('../src/integrity.mjs') + const integrity = path.join(store.dir, 'data.integrity.jsontag') + await appendIntegrityRecord( + integrity, + store.datafile, + await fs.readFile(store.datafile) + ) + await appendIntegrityRecord( + integrity, + path.join(store.dir, 'data.A.jsontag'), + Buffer.from('different historical bytes') + ) + store.integrityFile = integrity + const plan = await planRecovery(store, { quiescent: true }) + await assert.rejects( + runRecovery(store, outputs, plan), + /Integrity mismatch/ + ) + const authorization = JSON.parse( + ( + await fs.readFile( + path.join(outputs, 'audit', 'attempts.jsonl'), + 'utf8' + ) + ).split('\n')[0] + ) + assert.deepEqual( + await fs.readFile(authorization.config.integrityFile), + await fs.readFile(integrity) + ) + const statuses = ( + await fs.readFile(authorization.config.commandStatus, 'utf8') + ) + .trim() + .split('\n') + .map(JSONTag.parse) + assert.equal(statuses.at(-1).status, 'active') }) -test('malformed tails and overlapping paths cannot be accepted as valid stores',async t=>{ - const {store}=await setup(t,[['A','accepted']]) - await fs.appendFile(store.commandLog,'{"id":"unfinished"') - assert.equal((await planRecovery(store,{quiescent:true})).actionable,false) - assert.ok((await inspectStore({...store,commandStatus:store.commandLog})).errors.some(e=>e.includes('paths overlap'))) +test('malformed tails and overlapping paths cannot be accepted as valid stores', async t => { + const { store } = await setup(t, [['A', 'accepted']]) + await fs.appendFile(store.commandLog, '{"id":"unfinished"') + assert.equal( + (await planRecovery(store, { quiescent: true })).actionable, + false + ) + assert.ok( + ( + await inspectStore({ ...store, commandStatus: store.commandLog }) + ).errors.some(e => e.includes('paths overlap')) + ) }) -test('approved recovery resumes from completed candidate prefix without repeating external effects',async t=>{ - const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted']]) - const witness=path.join(outputs,'idempotent-effects') - await fs.writeFile(store.commandsFile,`import fs from 'node:fs';export default {addPerson(data,c){ +test('approved recovery resumes from completed candidate prefix without repeating external effects', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + const witness = path.join(outputs, 'idempotent-effects') + await fs.writeFile( + store.commandsFile, + `import fs from 'node:fs';export default {addPerson(data,c){ const previous=fs.existsSync(${JSON.stringify(witness)})?fs.readFileSync(${JSON.stringify(witness)},'utf8'):''; if(!previous.split('\\n').includes(c.id)){ fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n'); if(c.id==='B')throw new Error('connection lost after effect'); } data.persons.push(c.value) - }}`) - await assert.rejects(runRecovery(store,outputs,await planRecovery(store,{quiescent:true})),/uncertain or failed/) - const authorization=JSON.parse((await fs.readFile(path.join(outputs,'audit','attempts.jsonl'),'utf8')).split('\n')[0]) - const partial={...authorization.config,commandsFile:store.commandsFile,indexFile:store.indexFile} - const plan=await planRecovery(partial,{quiescent:true}) - assert.deepEqual(plan.rerun,['B']) - const {releaseOfflineLocks}=await import('../src/admin-recovery.mjs') - const release=await releaseOfflineLocks(partial,{operator:'test admin',reason:'Inspected retained recovery audit; idempotent B can retry',confirmedStopped:true}) - await fs.writeFile(path.join(outputs,'unlock-evidence.json'),JSON.stringify(release)) + }}` + ) + await assert.rejects( + runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ), + /uncertain or failed/ + ) + const authorization = JSON.parse( + ( + await fs.readFile( + path.join(outputs, 'audit', 'attempts.jsonl'), + 'utf8' + ) + ).split('\n')[0] + ) + const partial = { + ...authorization.config, + commandsFile: store.commandsFile, + indexFile: store.indexFile + } + const plan = await planRecovery(partial, { quiescent: true }) + assert.deepEqual(plan.rerun, ['B']) + const { releaseOfflineLocks } = await import('../src/admin-recovery.mjs') + const release = await releaseOfflineLocks(partial, { + operator: 'test admin', + reason: 'Inspected retained recovery audit; idempotent B can retry', + confirmedStopped: true + }) + await fs.writeFile( + path.join(outputs, 'unlock-evidence.json'), + JSON.stringify(release) + ) await release.finish() - const retry=path.join(outputs,'retry');await fs.mkdir(retry) - const result=await runRecovery(partial,retry,plan) - assert.deepEqual((await inspectStore(result.config)).data.persons.map(p=>p.name),['A','B']) - assert.equal(await fs.readFile(witness,'utf8'),'A\nB\n') + const retry = path.join(outputs, 'retry') + await fs.mkdir(retry) + const result = await runRecovery(partial, retry, plan) + assert.deepEqual( + (await inspectStore(result.config)).data.persons.map(p => p.name), + ['A', 'B'] + ) + assert.equal(await fs.readFile(witness, 'utf8'), 'A\nB\n') }) -test('finishing an interrupted completion report reconstructs data without executing again',async t=>{ - const {store,outputs}=await setup(t,[['A','accepted']]) - const result=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) - const audit=path.join(outputs,'audit') - // Fixture models the boundary after done, before completion report/lock release. - await fs.unlink(path.join(audit,'complete.json')) - await acquireOwnership([path.dirname(result.config.datafile)],{purpose:'recovery',auditDir:audit}) - const {finishRecovery,releaseOfflineLocks}=await import('../src/admin-recovery.mjs') - await assert.rejects(releaseOfflineLocks(result.config,{operator:'admin',reason:'test',confirmedStopped:true}),/requires finish/) - const before=await fs.readFile(result.config.commandStatus) - const finished=await finishRecovery(audit,{operator:'admin',reason:'All authorized commands already committed',confirmedStopped:true}) - assert.deepEqual(await fs.readFile(result.config.commandStatus),before) - assert.deepEqual(finished.committed,['A']) - const owner=await acquireOwnership([path.dirname(result.config.datafile)]);await owner.release() +test('finishing an interrupted completion report reconstructs data without executing again', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const result = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + const audit = path.join(outputs, 'audit') + // Fixture models the boundary after done, before completion report/lock + // release. + await fs.unlink(path.join(audit, 'complete.json')) + await acquireOwnership([path.dirname(result.config.datafile)], { + purpose: 'recovery', + auditDir: audit + }) + const { finishRecovery, releaseOfflineLocks } = await import( + '../src/admin-recovery.mjs' + ) + await assert.rejects( + releaseOfflineLocks(result.config, { + operator: 'admin', + reason: 'test', + confirmedStopped: true + }), + /requires finish/ + ) + const before = await fs.readFile(result.config.commandStatus) + const finished = await finishRecovery(audit, { + operator: 'admin', + reason: 'All authorized commands already committed', + confirmedStopped: true + }) + assert.deepEqual(await fs.readFile(result.config.commandStatus), before) + assert.deepEqual(finished.committed, ['A']) + const owner = await acquireOwnership([path.dirname(result.config.datafile)]) + await owner.release() }) -test('multi-directory integrity paths and file permissions survive backup and restore',async t=>{ - const {store,outputs}=await setup(t,[]) - const logdir=path.join(store.dir,'logs');await fs.mkdir(logdir) - store.commandLog=path.join(logdir,'commands.jsontag');store.commandStatus=path.join(logdir,'status.jsontag') - await fs.writeFile(store.commandLog,'');await fs.writeFile(store.commandStatus,'') - store.integrityFile=path.join(logdir,'integrity.jsontag') - const {appendIntegrityRecord}=await import('../src/integrity.mjs') - await appendIntegrityRecord(store.integrityFile,store.datafile,await fs.readFile(store.datafile)) - await fs.chmod(store.datafile,0o600) - const backup=path.join(outputs,'backup');await backupStore(store,{to:backup,quiescent:true}) - const restored=await restoreBackup(backup,{to:path.join(outputs,'restored'),auditDir:path.join(outputs,'audit')}) - assert.equal((await inspectStore(restored.config)).ready,true) - assert.equal((await fs.stat(restored.config.datafile)).mode&0o777,0o600) - assert.deepEqual(await fs.readFile(restored.config.integrityFile),await fs.readFile(store.integrityFile)) +test('multi-directory integrity paths and file permissions survive backup and restore', async t => { + const { store, outputs } = await setup(t, []) + const logdir = path.join(store.dir, 'logs') + await fs.mkdir(logdir) + store.commandLog = path.join(logdir, 'commands.jsontag') + store.commandStatus = path.join(logdir, 'status.jsontag') + await fs.writeFile(store.commandLog, '') + await fs.writeFile(store.commandStatus, '') + store.integrityFile = path.join(logdir, 'integrity.jsontag') + const { appendIntegrityRecord } = await import('../src/integrity.mjs') + await appendIntegrityRecord( + store.integrityFile, + store.datafile, + await fs.readFile(store.datafile) + ) + await fs.chmod(store.datafile, 0o600) + const backup = path.join(outputs, 'backup') + await backupStore(store, { to: backup, quiescent: true }) + const restored = await restoreBackup(backup, { + to: path.join(outputs, 'restored'), + auditDir: path.join(outputs, 'audit') + }) + assert.equal((await inspectStore(restored.config)).ready, true) + assert.equal((await fs.stat(restored.config.datafile)).mode & 0o777, 0o600) + assert.deepEqual( + await fs.readFile(restored.config.integrityFile), + await fs.readFile(store.integrityFile) + ) }) -test('backup excludes unrelated secrets while retaining declared custom artifacts',async t=>{ - const {store,outputs}=await setup(t,[]) - await fs.writeFile(path.join(store.dir,'.env'),'private application secret') - const required=path.join(store.dir,'derived-custom.txt');await fs.writeFile(required,'required output') - const report=await backupStore({...store,requiredFiles:[required]},{to:path.join(outputs,'backup'),quiescent:true}) - assert.equal(Object.keys(report.files).some(file=>path.basename(file)==='.env'),false) - assert.equal(await fs.readFile(report.config.requiredFiles[0],'utf8'),'required output') +test('backup excludes unrelated secrets while retaining declared custom artifacts', async t => { + const { store, outputs } = await setup(t, []) + await fs.writeFile( + path.join(store.dir, '.env'), + 'private application secret' + ) + const required = path.join(store.dir, 'derived-custom.txt') + await fs.writeFile(required, 'required output') + const report = await backupStore( + { ...store, requiredFiles: [required] }, + { to: path.join(outputs, 'backup'), quiescent: true } + ) + assert.equal( + Object.keys(report.files).some(file => path.basename(file) === '.env'), + false + ) + assert.equal( + await fs.readFile(report.config.requiredFiles[0], 'utf8'), + 'required output' + ) }) -test('a backup remains restorable after transfer to a different directory',async t=>{ - const {store,outputs}=await setup(t,[]) - const old=path.join(outputs,'old-backup');await backupStore(store,{to:old,quiescent:true}) - const moved=path.join(outputs,'transferred-backup');await fs.rename(old,moved) - const result=await restoreBackup(moved,{to:path.join(outputs,'restored'),auditDir:path.join(outputs,'restore-audit')}) - assert.equal((await inspectStore(result.config)).ready,true) +test('a backup remains restorable after transfer to a different directory', async t => { + const { store, outputs } = await setup(t, []) + const old = path.join(outputs, 'old-backup') + await backupStore(store, { to: old, quiescent: true }) + const moved = path.join(outputs, 'transferred-backup') + await fs.rename(old, moved) + const result = await restoreBackup(moved, { + to: path.join(outputs, 'restored'), + auditDir: path.join(outputs, 'restore-audit') + }) + assert.equal((await inspectStore(result.config)).ready, true) }) -test('promotion verification rejects code changed after completed recovery',async t=>{ - const {store,outputs}=await setup(t,[['A','accepted']]) - const result=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) - assert.equal((await verifyCandidate(result)).ready,true) - await fs.appendFile(store.commandsFile,'\n// altered after recovery\n') - await assert.rejects(verifyCandidate(result),/Selected code changed/) +test('promotion verification rejects code changed after completed recovery', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const result = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + assert.equal((await verifyCandidate(result)).ready, true) + await fs.appendFile(store.commandsFile, '\n// altered after recovery\n') + await assert.rejects(verifyCandidate(result), /Selected code changed/) }) -test('verified A prefix permits only missing B, waiting C and missing D in log order',async t=>{ - const {store,outputs}=await setup(t,[['A','accepted'],['B','accepted'],['C','accepted'],['D','accepted']]) - const complete=await runRecovery(store,outputs,await planRecovery(store,{quiescent:true})) - const source={...complete.config,commandsFile:store.commandsFile,indexFile:store.indexFile} - for(const id of ['B','C','D']){ - await fs.unlink(path.join(path.dirname(source.datafile),`data.${id}.jsontag`)) +test('verified A prefix permits only missing B, waiting C and missing D in log order', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'], + ['C', 'accepted'], + ['D', 'accepted'] + ]) + const complete = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + const source = { + ...complete.config, + commandsFile: store.commandsFile, + indexFile: store.indexFile } - await fs.appendFile(source.commandStatus,JSONTag.stringify({command:'C',status:'accepted',code:202})+'\n') - const prefix=await fs.readFile(path.join(path.dirname(source.datafile),'data.A.jsontag')) - const plan=await planRecovery(source,{quiescent:true}) - assert.equal(plan.actionable,true);assert.deepEqual(plan.committed,['A']);assert.deepEqual(plan.rerun,['B','C','D']) - const next=path.join(outputs,'next');await fs.mkdir(next) - const result=await runRecovery(source,next,plan) - assert.deepEqual((await inspectStore(result.config)).data.persons.map(p=>p.name),['A','B','C','D']) - assert.deepEqual(await fs.readFile(path.join(path.dirname(result.config.datafile),'data.A.jsontag')),prefix) + for (const id of ['B', 'C', 'D']) { + await fs.unlink( + path.join(path.dirname(source.datafile), `data.${id}.jsontag`) + ) + } + await fs.appendFile( + source.commandStatus, + JSONTag.stringify({ command: 'C', status: 'accepted', code: 202 }) + + '\n' + ) + const prefix = await fs.readFile( + path.join(path.dirname(source.datafile), 'data.A.jsontag') + ) + const plan = await planRecovery(source, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(plan.committed, ['A']) + assert.deepEqual(plan.rerun, ['B', 'C', 'D']) + const next = path.join(outputs, 'next') + await fs.mkdir(next) + const result = await runRecovery(source, next, plan) + assert.deepEqual( + (await inspectStore(result.config)).data.persons.map(p => p.name), + ['A', 'B', 'C', 'D'] + ) + assert.deepEqual( + await fs.readFile( + path.join(path.dirname(result.config.datafile), 'data.A.jsontag') + ), + prefix + ) }) -test('restore reports accepted commands absent from an older backup',async t=>{ - const {store,outputs}=await setup(t,[]) - const backup=path.join(outputs,'old-backup');await backupStore(store,{to:backup,quiescent:true}) - await fs.appendFile(store.commandLog,JSONTag.stringify({id:'later',name:'addPerson',value:{name:'later'}})+'\n') - await fs.appendFile(store.commandStatus,JSONTag.stringify({command:'later',status:'accepted',code:202})+'\n') - const result=await restoreBackup(backup,{to:path.join(outputs,'rollback'),auditDir:path.join(outputs,'audit'),source:store,sourceQuiescent:true}) - assert.equal(result.sameBase,true);assert.deepEqual(result.missingFromBackup,['later']) - assert.deepEqual(result.committed,[]) +test('restore reports accepted commands absent from an older backup', async t => { + const { store, outputs } = await setup(t, []) + const backup = path.join(outputs, 'old-backup') + await backupStore(store, { to: backup, quiescent: true }) + await fs.appendFile( + store.commandLog, + JSONTag.stringify({ + id: 'later', + name: 'addPerson', + value: { name: 'later' } + }) + '\n' + ) + await fs.appendFile( + store.commandStatus, + JSONTag.stringify({ command: 'later', status: 'accepted', code: 202 }) + + '\n' + ) + const result = await restoreBackup(backup, { + to: path.join(outputs, 'rollback'), + auditDir: path.join(outputs, 'audit'), + source: store, + sourceQuiescent: true + }) + assert.equal(result.sameBase, true) + assert.deepEqual(result.missingFromBackup, ['later']) + assert.deepEqual(result.committed, []) }) -test('identical historical duplicate retains first log position; conflicting duplicate blocks',async t=>{ - const {store}=await setup(t,[['A','accepted'],['B','accepted']]) - await fs.appendFile(store.commandLog,JSONTag.stringify({id:'A',name:'addPerson',value:{name:'A'}})+'\n') - const plan=await planRecovery(store,{quiescent:true}) - assert.equal(plan.actionable,true);assert.deepEqual(plan.rerun,['A','B']) - await fs.appendFile(store.commandLog,JSONTag.stringify({id:'A',name:'addPerson',value:{name:'different'}})+'\n') - assert.ok((await planRecovery(store,{quiescent:true})).blocks.some(s=>s.includes('Conflicting command ID A'))) +test('identical historical duplicate retains first log position; conflicting duplicate blocks', async t => { + const { store } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + await fs.appendFile( + store.commandLog, + JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) + '\n' + ) + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(plan.rerun, ['A', 'B']) + await fs.appendFile( + store.commandLog, + JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'different' } + }) + '\n' + ) + assert.ok( + (await planRecovery(store, { quiescent: true })).blocks.some(s => + s.includes('Conflicting command ID A') + ) + ) }) - test('backup coverage distinguishes changed history and data from missing source data', async t => { - const {store, outputs} = await setup(t, [['A', 'accepted']]) - const plan = await planRecovery(store, {quiescent: true}) + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const plan = await planRecovery(store, { quiescent: true }) const recovered = await runRecovery(store, outputs, plan) const source = recovered.config const backup = path.join(outputs, 'coverage-backup') - await backupStore(source, {to: backup, quiescent: true}) + await backupStore(source, { to: backup, quiescent: true }) const changeset = path.join(path.dirname(source.datafile), 'data.A.jsontag') const original = new Map() - for (const file of [source.datafile, source.commandLog, source.commandStatus, changeset]) { + for (const file of [ + source.datafile, + source.commandLog, + source.commandStatus, + changeset + ]) { original.set(file, await fs.readFile(file)) } - const differentCommand = JSONTag.stringify({id: 'A', name: 'addPerson', value: {name: 'changed'}}) - const failedStatus = JSONTag.stringify({command: 'A', status: 'failed', code: 500}) + const differentCommand = JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'changed' } + }) + const failedStatus = JSONTag.stringify({ + command: 'A', + status: 'failed', + code: 500 + }) const cases = [ - {name: 'same', missing: [], sameBase: true}, - {name: 'command', file: source.commandLog, bytes: differentCommand + '\n', missing: ['A'], sameBase: true}, - {name: 'status', file: source.commandStatus, bytes: original.get(source.commandStatus) + failedStatus + '\n', missing: ['A'], sameBase: true}, - {name: 'data', file: changeset, bytes: original.get(changeset) + ' ', missing: ['A'], sameBase: true}, - {name: 'base', file: source.datafile, bytes: original.get(source.datafile) + ' ', missing: ['A'], sameBase: false}, - {name: 'missing', file: changeset, missing: [], sameBase: true} + { name: 'same', missing: [], sameBase: true }, + { + name: 'command', + file: source.commandLog, + bytes: differentCommand + '\n', + missing: ['A'], + sameBase: true + }, + { + name: 'status', + file: source.commandStatus, + bytes: original.get(source.commandStatus) + failedStatus + '\n', + missing: ['A'], + sameBase: true + }, + { + name: 'data', + file: changeset, + bytes: original.get(changeset) + ' ', + missing: ['A'], + sameBase: true + }, + { + name: 'base', + file: source.datafile, + bytes: original.get(source.datafile) + ' ', + missing: ['A'], + sameBase: false + }, + { name: 'missing', file: changeset, missing: [], sameBase: true } ] for (const scenario of cases) { for (const [file, bytes] of original) { @@ -295,8 +695,16 @@ test('backup coverage distinguishes changed history and data from missing source source, sourceQuiescent: true }) - assert.deepEqual(result.missingFromBackup, scenario.missing, scenario.name) + assert.deepEqual( + result.missingFromBackup, + scenario.missing, + scenario.name + ) assert.equal(result.sameBase, scenario.sameBase, scenario.name) - assert.deepEqual(structuredClone(result), result, 'Restore reports remain plain cloneable data') + assert.deepEqual( + structuredClone(result), + result, + 'Restore reports remain plain cloneable data' + ) } }) diff --git a/test/after-change-handlers.mjs b/test/after-change-handlers.mjs index f2dc5c0..089cdac 100644 --- a/test/after-change-handlers.mjs +++ b/test/after-change-handlers.mjs @@ -1,4 +1,4 @@ -import {waitForExit} from './durability-helpers.mjs' +import { waitForExit } from './durability-helpers.mjs' import assert from 'node:assert/strict' import fs from 'node:fs/promises' import path from 'node:path' @@ -24,7 +24,9 @@ async function waitForCommandStatus(port, commandId, expectedStatus) { } await new Promise(resolve => setTimeout(resolve, 25)) } - assert.fail(`Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}`) + assert.fail( + `Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}` + ) } async function postAndWait(port, command, expectedStatus = 'done') { @@ -58,20 +60,29 @@ test('handler lifecycle: index update failure prevents command commit', async t ` }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { - id: 'index-fails', - name: 'addPerson', - value: {name: 'Should Not Commit'} - }, 'failed') + const status = await postAndWait( + port, + { + id: 'index-fails', + name: 'addPerson', + value: { name: 'Should Not Commit' } + }, + 'failed' + ) assert.equal(status.status, 'failed') assert.match(status.message, /index update failed/) assert.deepEqual(await queryNames(port), ['Initial']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) - await assert.rejects(fs.access(path.join(fixture.dir, 'data.index-fails.jsontag')), /ENOENT/) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) + await assert.rejects( + fs.access(path.join(fixture.dir, 'data.index-fails.jsontag')), + /ENOENT/ + ) }) test('handler lifecycle: derived index file can remain after failed command', async t => { @@ -91,21 +102,39 @@ export default { ` }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { - id: 'index-writes-then-fails', - name: 'addPerson', - value: {name: 'Should Not Commit'} - }, 'failed') + const status = await postAndWait( + port, + { + id: 'index-writes-then-fails', + name: 'addPerson', + value: { name: 'Should Not Commit' } + }, + 'failed' + ) assert.equal(status.status, 'failed') assert.match(status.message, /derived index failed after write/) assert.deepEqual(await queryNames(port), ['Initial']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) - await assert.doesNotReject(fs.access(path.join(fixture.dir, 'derived-before-failure.index-writes-then-fails.txt'))) - await assert.rejects(fs.access(path.join(fixture.dir, 'data.index-writes-then-fails.jsontag')), /ENOENT/) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) + await assert.doesNotReject( + fs.access( + path.join( + fixture.dir, + 'derived-before-failure.index-writes-then-fails.txt' + ) + ) + ) + await assert.rejects( + fs.access( + path.join(fixture.dir, 'data.index-writes-then-fails.jsontag') + ), + /ENOENT/ + ) }) test('handler lifecycle: final offset write failure prevents command commit', async t => { @@ -116,40 +145,47 @@ test('handler lifecycle: final offset write failure prevents command commit', as const commandId = 'offset-write-fails' await fs.mkdir(path.join(fixture.dir, `index.offset.${commandId}.json`)) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) const response = await postCommand(port, { id: commandId, name: 'addPerson', - value: {name: 'Should Not Commit'} + value: { name: 'Should Not Commit' } }) - assert.equal(response.status,202) - assert.equal((await waitForExit(server.child)).code,1) - assert.match(server.getOutput(),/Storage outcome uncertain/) + assert.equal(response.status, 202) + assert.equal((await waitForExit(server.child)).code, 1) + assert.match(server.getOutput(), /Storage outcome uncertain/) assert.match(server.getOutput(), /EISDIR|EEXIST|ENOTEMPTY/) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) }) test('handler lifecycle: custom finalizer rejection prevents command commit', async t => { const fixture = await makeServerFixture(t, { initialData: '{"persons":[{"name":"Initial"}]}', commandsSource: baseCommands, - indexSource: 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}' + indexSource: + 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}' }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) const response = await postCommand(port, { - id: 'custom-finalizer-fails', name: 'addPerson', value: {name: 'Should Not Commit'} + id: 'custom-finalizer-fails', + name: 'addPerson', + value: { name: 'Should Not Commit' } }) - assert.equal(response.status,202) - assert.equal((await waitForExit(server.child)).code,1) - assert.match(server.getOutput(),/Storage outcome uncertain/) + assert.equal(response.status, 202) + assert.equal((await waitForExit(server.child)).code, 1) + assert.match(server.getOutput(), /Storage outcome uncertain/) assert.match(server.getOutput(), /custom finalization failed/) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) }) test('handler lifecycle: current index update can mutate canonical state before commit', async t => { @@ -166,15 +202,23 @@ test('handler lifecycle: current index update can mutate canonical state before ` }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) await postAndWait(port, { id: 'index-mutates', name: 'addPerson', - value: {name: 'Command Mutation'} + value: { name: 'Command Mutation' } }) - assert.deepEqual(await queryNames(port), ['Initial', 'Command Mutation', 'Indexer Mutation']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial', 'Command Mutation', 'Indexer Mutation']) + assert.deepEqual(await queryNames(port), [ + 'Initial', + 'Command Mutation', + 'Indexer Mutation' + ]) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial', + 'Command Mutation', + 'Indexer Mutation' + ]) }) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs index 2621429..8878641 100644 --- a/test/conversion-offsets.mjs +++ b/test/conversion-offsets.mjs @@ -10,53 +10,79 @@ import { promisify } from 'node:util' import Parser from '@muze-nl/od-jsontag' const run = promisify(execFile) -const converter = fileURLToPath(new URL('../scripts/convert.mjs', import.meta.url)) +const converter = fileURLToPath( + new URL('../scripts/convert.mjs', import.meta.url) +) const baseIndex = new URL('../src/index.mjs', import.meta.url).href async function fixture(t) { const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-convert-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) + t.after(() => fs.rm(dir, { recursive: true, force: true })) const input = path.join(dir, 'input.jsontag') const output = path.join(dir, 'data.jsontag') - await fs.writeFile(input, '{"items":[{"name":"first"},{"name":"a long original name"}]}') - return {dir, input, output} + await fs.writeFile( + input, + '{"items":[{"name":"first"},{"name":"a long original name"}]}' + ) + return { dir, input, output } } -async function convert({input, output}, indexFile) { - await run(process.execPath, [converter, input, output, ...indexFile ? [indexFile] : []], {timeout: 10000}) +async function convert({ input, output }, indexFile) { + const args = [converter, input, output] + if (indexFile) { + args.push(indexFile) + } + await run(process.execPath, args, { timeout: 10000 }) } -async function verifyOffsets({dir, output}) { +async function verifyOffsets({ dir, output }) { const bytes = await fs.readFile(output) - const offsets = JSON.parse(await fs.readFile(path.join(dir, 'index.offset.json'), 'utf8')) + const offsets = JSON.parse( + await fs.readFile(path.join(dir, 'index.offset.json'), 'utf8') + ) const payloads = [] let cursor = 0 - // Derive byte ranges directly from disk framing, independently of Parser.meta. + // Derive byte ranges directly from disk framing, independently of + // Parser.meta. while (cursor < bytes.length) { - const match = /^\((\d+)\)/.exec(bytes.subarray(cursor, cursor + 32).toString('ascii')) + const match = /^\((\d+)\)/.exec( + bytes.subarray(cursor, cursor + 32).toString('ascii') + ) assert.ok(match, `record ${payloads.length} has a length prefix`) const start = cursor + match[0].length const end = start + Number(match[1]) assert.ok(end <= bytes.length, 'record ends within the file') - assert.deepEqual(offsets[payloads.length], [start, end], `record ${payloads.length} byte range`) - payloads.push(bytes.subarray(...offsets[payloads.length]).toString('utf8')) + assert.deepEqual( + offsets[payloads.length], + [start, end], + `record ${payloads.length} byte range` + ) + payloads.push( + bytes.subarray(...offsets[payloads.length]).toString('utf8') + ) cursor = end if (cursor < bytes.length) { assert.equal(bytes[cursor], 10, 'records separated by a newline') cursor++ } } - assert.equal(Object.keys(offsets).length, payloads.length, 'no missing or extra offsets') - return {payloads, data: new Parser('').parse(bytes.toString('utf8'))} + assert.equal( + Object.keys(offsets).length, + payloads.length, + 'no missing or extra offsets' + ) + return { payloads, data: new Parser('').parse(bytes.toString('utf8')) } } test('default conversion offsets match every final record', async t => { const files = await fixture(t) await convert(files) - const {payloads, data} = await verifyOffsets(files) + const { payloads, data } = await verifyOffsets(files) assert.equal(payloads.length, 3) assert.equal(data.items[1].name, 'a long original name') - const ids = JSON.parse(await fs.readFile(path.join(files.dir, 'index.id.json'), 'utf8')) + const ids = JSON.parse( + await fs.readFile(path.join(files.dir, 'index.id.json'), 'utf8') + ) assert.match(payloads[ids['/first']], /id="\/first"/) assert.match(payloads[ids['/second']], /id="\/second"/) }) @@ -64,7 +90,9 @@ test('default conversion offsets match every final record', async t => { test('conversion finalizes offsets after custom indexing changes record sizes and adds records', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} + await fs.writeFile( + indexFile, + `import index from ${JSON.stringify(baseIndex)} export default { create(data, meta) { index.create(data, meta) @@ -74,13 +102,20 @@ export default { data.items.push({name: 'new derived record'}) } } -`) +` + ) await convert(files, indexFile) - const {payloads, data} = await verifyOffsets(files) + const { payloads, data } = await verifyOffsets(files) assert.equal(payloads.length, 4) - assert.equal(data.items[0].name, 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a much longer derived name') + assert.equal( + data.items[0].name, + 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a much longer derived name' + ) assert.equal(data.items[0].derived[0].name, 'x') - assert.equal(Object.getOwnPropertyDescriptor(data.items[0], 'derived').enumerable, false) + assert.equal( + Object.getOwnPropertyDescriptor(data.items[0], 'derived').enumerable, + false + ) assert.equal(data.items[2].name, 'new derived record') assert.match(payloads[3], /new derived record/) }) @@ -88,10 +123,16 @@ export default { test('conversion replaces stale offsets even when the custom hook writes no index', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, 'export default Object.freeze({ create(data) { data.items[0].name = "changed" } })\n') - await fs.writeFile(path.join(files.dir, 'index.offset.json'), '{"0":[0,1],"99":[2,3]}') + await fs.writeFile( + indexFile, + 'export default Object.freeze({ create(data) { data.items[0].name = "changed" } })\n' + ) + await fs.writeFile( + path.join(files.dir, 'index.offset.json'), + '{"0":[0,1],"99":[2,3]}' + ) await convert(files, indexFile) - const {payloads, data} = await verifyOffsets(files) + const { payloads, data } = await verifyOffsets(files) assert.equal(payloads.length, 3) assert.equal(data.items[0].name, 'changed') }) @@ -99,7 +140,9 @@ test('conversion replaces stale offsets even when the custom hook writes no inde test('conversion awaits custom finalization with the original receiver and final bytes', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} + await fs.writeFile( + indexFile, + `import index from ${JSON.stringify(baseIndex)} import fs from 'node:fs/promises' import {setTimeout} from 'node:timers/promises' export default { @@ -112,29 +155,68 @@ export default { await fs.writeFile(meta.data + '/finalized.json', JSON.stringify({marker: this.marker, uuid})) } } -`) +` + ) await convert(files, indexFile) await verifyOffsets(files) - assert.deepEqual(JSON.parse(await fs.readFile(path.join(files.dir, 'finalized.json'), 'utf8')), {marker: 'custom receiver', uuid: null}) + assert.deepEqual( + JSON.parse( + await fs.readFile(path.join(files.dir, 'finalized.json'), 'utf8') + ), + { marker: 'custom receiver', uuid: null } + ) }) test('conversion propagates rejection from custom finalization', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, 'export default {create() {}, async finalize() { throw new Error("custom finalization failed") }}\n') - await assert.rejects(convert(files, indexFile), /custom finalization failed/) - await assert.rejects(fs.access(path.join(files.dir, 'index.offset.json')), /ENOENT/) + await fs.writeFile( + indexFile, + 'export default {create() {}, async finalize() { throw new Error("custom finalization failed") }}\n' + ) + await assert.rejects( + convert(files, indexFile), + /custom finalization failed/ + ) + await assert.rejects( + fs.access(path.join(files.dir, 'index.offset.json')), + /ENOENT/ + ) }) -test('conversion initializes existing-format logs and optional base integrity without overwriting a store',async t=>{ - const files=await fixture(t) - await run(process.execPath,[converter,files.input,files.output,'--integrity'],{timeout:10000}) - const {loadIntegrityManifest,verifyIntegrity}=await import('../src/integrity.mjs') - const integrity=path.join(files.dir,'data.integrity.jsontag') - assert.equal(verifyIntegrity(await loadIntegrityManifest(integrity),integrity,files.output,await fs.readFile(files.output),{required:true}),true) - assert.equal(await fs.readFile(path.join(files.dir,'command-log.jsontag'),'utf8'),'') - assert.equal(await fs.readFile(path.join(files.dir,'command-status.jsontag'),'utf8'),'') - const before=await fs.readFile(files.output) - await assert.rejects(convert(files),/existing store is not overwritten/) - assert.deepEqual(await fs.readFile(files.output),before) +test('conversion initializes existing-format logs and optional base integrity without overwriting a store', async t => { + const files = await fixture(t) + await run( + process.execPath, + [converter, files.input, files.output, '--integrity'], + { timeout: 10000 } + ) + const { loadIntegrityManifest, verifyIntegrity } = await import( + '../src/integrity.mjs' + ) + const integrity = path.join(files.dir, 'data.integrity.jsontag') + assert.equal( + verifyIntegrity( + await loadIntegrityManifest(integrity), + integrity, + files.output, + await fs.readFile(files.output), + { required: true } + ), + true + ) + assert.equal( + await fs.readFile(path.join(files.dir, 'command-log.jsontag'), 'utf8'), + '' + ) + assert.equal( + await fs.readFile( + path.join(files.dir, 'command-status.jsontag'), + 'utf8' + ), + '' + ) + const before = await fs.readFile(files.output) + await assert.rejects(convert(files), /existing store is not overwritten/) + assert.deepEqual(await fs.readFile(files.output), before) }) diff --git a/test/durability-crash.mjs b/test/durability-crash.mjs index 598f10a..ef1152c 100644 --- a/test/durability-crash.mjs +++ b/test/durability-crash.mjs @@ -3,9 +3,16 @@ import fs from 'node:fs/promises' import path from 'node:path' import test from 'node:test' import { faultPoint } from '../src/faults.mjs' -import { appendIntegrityRecord, getDefaultIntegrityFile, loadIntegrityManifest } from '../src/integrity.mjs' +import { + appendIntegrityRecord, + getDefaultIntegrityFile, + loadIntegrityManifest +} from '../src/integrity.mjs' import { getChangesetPath } from '../src/recovery.mjs' -import { assertRuntimeEnvironmentConfiguration, getRuntimeEnvironment } from '../src/runtime-environment.mjs' +import { + assertRuntimeEnvironmentConfiguration, + getRuntimeEnvironment +} from '../src/runtime-environment.mjs' import { unlockStoppedFixture, getCommandStatus, @@ -28,7 +35,10 @@ async function postCommandExpectingCrash(port, command) { assert.equal(response.status, 202) } catch (error) { - assert.match(error.message, /fetch failed|terminated|socket|other side closed|aborted/i) + assert.match( + error.message, + /fetch failed|terminated|socket|other side closed|aborted/i + ) } } @@ -40,8 +50,14 @@ async function postCommandStatus(port, command, expectedHttpStatus = 200) { async function assertServerStateMatchesOracle(port, fixture, expectedNames) { const persons = await queryPersons(port) - assert.deepEqual(persons.map(person => person.name), expectedNames) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), expectedNames) + assert.deepEqual( + persons.map(person => person.name), + expectedNames + ) + assert.deepEqual( + await reconstructCommittedPersonNames(fixture), + expectedNames + ) } async function waitForCommandStatus(port, commandId, expectedStatus) { @@ -54,34 +70,47 @@ async function waitForCommandStatus(port, commandId, expectedStatus) { } await new Promise(resolve => setTimeout(resolve, 25)) } - assert.fail(`Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}`) + assert.fail( + `Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}` + ) } test('runtime environment defaults to production and ignores production fault points', () => { assert.equal(getRuntimeEnvironment({}), 'production') assert.doesNotThrow(() => assertRuntimeEnvironmentConfiguration({})) - assert.doesNotThrow(() => assertRuntimeEnvironmentConfiguration({ - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - })) - assert.doesNotThrow(() => assertRuntimeEnvironmentConfiguration({ - SIMPLYSTORE_ENV: 'test', - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - })) + assert.doesNotThrow(() => + assertRuntimeEnvironmentConfiguration({ + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }) + ) + assert.doesNotThrow(() => + assertRuntimeEnvironmentConfiguration({ + SIMPLYSTORE_ENV: 'test', + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }) + ) assert.throws( - () => assertRuntimeEnvironmentConfiguration({SIMPLYSTORE_ENV: 'prod'}), + () => + assertRuntimeEnvironmentConfiguration({ SIMPLYSTORE_ENV: 'prod' }), /Invalid SIMPLYSTORE_ENV/ ) }) test('fault points are inert outside test environment', async () => { - assert.equal(await faultPoint('before-command-done-status', { - SIMPLYSTORE_ENV: 'production', - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - }), false) - assert.equal(await faultPoint('before-command-done-status', { - SIMPLYSTORE_ENV: 'development', - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - }), false) + assert.equal( + await faultPoint('before-command-done-status', { + SIMPLYSTORE_ENV: 'production', + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }), + false + ) + assert.equal( + await faultPoint('before-command-done-status', { + SIMPLYSTORE_ENV: 'development', + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }), + false + ) }) for (const faultPointName of [ @@ -93,19 +122,27 @@ for (const faultPointName of [ 'before-command-done-status' ]) { test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { - const fixture=await makeServerFixture(t) - const port=await getOpenPort() - const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:faultPointName}) - await waitForServer(first.child,first.getOutput,port) - await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) - assert.equal((await waitForExit(first.child)).signal,'SIGKILL') - const before=await fs.readFile(fixture.commandStatus) - await unlockStoppedFixture(t,fixture) - const second=startServer(t,fixture,{port}) - assert.equal((await waitForExit(second.child)).code,1) - assert.match(second.getOutput(),/Administrative recovery required/) - assert.deepEqual(await fs.readFile(fixture.commandStatus),before) - assert.equal((await readCommandLogRecords(fixture))[0].id,'A') + const fixture = await makeServerFixture(t) + const port = await getOpenPort() + const first = startServer(t, fixture, { + port, + runtimeEnvironment: 'test', + faultPoint: faultPointName + }) + await waitForServer(first.child, first.getOutput, port) + await postCommandExpectingCrash(port, { + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) + assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') + const before = await fs.readFile(fixture.commandStatus) + await unlockStoppedFixture(t, fixture) + const second = startServer(t, fixture, { port }) + assert.equal((await waitForExit(second.child)).code, 1) + assert.match(second.getOutput(), /Administrative recovery required/) + assert.deepEqual(await fs.readFile(fixture.commandStatus), before) + assert.equal((await readCommandLogRecords(fixture))[0].id, 'A') }) } @@ -115,7 +152,7 @@ test('crash after done status but before query update recovers committed state w const command = { id: 'done-before-query-update', name: 'addPerson', - value: {name: 'Committed'} + value: { name: 'Committed' } } const first = startServer(t, fixture, { @@ -127,36 +164,55 @@ test('crash after done status but before query update recovers committed state w await postCommandExpectingCrash(port, command) assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - await unlockStoppedFixture(t,fixture) + await unlockStoppedFixture(t, fixture) const statusAfterCrash = await readCommandStatusRecords(fixture) assert.equal(statusAfterCrash.at(-1).status, 'done') - const second = startServer(t, fixture, {port}) + const second = startServer(t, fixture, { port }) await waitForServer(second.child, second.getOutput, port) const retryStatus = await postCommandStatus(port, { id: command.id, name: 'addPerson', - value: {name: 'Duplicate'} + value: { name: 'Duplicate' } }) assert.equal(retryStatus.command, command.id) assert.equal(retryStatus.status, 'done') await assertServerStateMatchesOracle(port, fixture, ['Committed']) - assert.equal((await readCommandStatusRecords(fixture)).filter(record => record.status === 'active').length, 1) + assert.equal( + (await readCommandStatusRecords(fixture)).filter( + record => record.status === 'active' + ).length, + 1 + ) assert.equal((await getCommandStatus(port, command.id)).status, 'done') }) -test('crash ownership cannot be stolen automatically',async t=>{ - const fixture=await makeServerFixture(t),port=await getOpenPort() - const first=startServer(t,fixture,{port,runtimeEnvironment:'test',faultPoint:'before-command-done-status'}) - await waitForServer(first.child,first.getOutput,port) - await postCommandExpectingCrash(port,{id:'A',name:'addPerson',value:{name:'A'}}) +test('crash ownership cannot be stolen automatically', async t => { + const fixture = await makeServerFixture(t), + port = await getOpenPort() + const first = startServer(t, fixture, { + port, + runtimeEnvironment: 'test', + faultPoint: 'before-command-done-status' + }) + await waitForServer(first.child, first.getOutput, port) + await postCommandExpectingCrash(port, { + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) await waitForExit(first.child) - const second=startServer(t,fixture,{port}) - assert.equal((await waitForExit(second.child)).code,1) - assert.match(second.getOutput(),/Store is locked/) - assert.equal((await readCommandStatusRecords(fixture)).filter(s=>s.status==='active').length,1) + const second = startServer(t, fixture, { port }) + assert.equal((await waitForExit(second.child)).code, 1) + assert.match(second.getOutput(), /Store is locked/) + assert.equal( + (await readCommandStatusRecords(fixture)).filter( + s => s.status === 'active' + ).length, + 1 + ) }) test('hanging command times out unsafe and later accepted command commits', async t => { @@ -187,11 +243,15 @@ test('hanging command times out unsafe and later accepted command commits', asyn const queuedResponse = await postCommand(port, { id: 'after-timeout', name: 'addPerson', - value: {name: 'After Timeout'} + value: { name: 'After Timeout' } }) assert.equal(queuedResponse.status, 202) - const timeoutStatus = await waitForCommandStatus(port, 'timeout-command', 'unsafe') + const timeoutStatus = await waitForCommandStatus( + port, + 'timeout-command', + 'unsafe' + ) assert.equal(timeoutStatus.code, 504) assert.equal(timeoutStatus.attempt, 1) assert.match(timeoutStatus.message, /command worker timed out after 100ms/) @@ -201,7 +261,7 @@ test('hanging command times out unsafe and later accepted command commits', asyn const retryStatus = await postCommandStatus(port, { id: 'timeout-command', name: 'addPerson', - value: {name: 'Retry'} + value: { name: 'Retry' } }) assert.equal(retryStatus.command, 'timeout-command') assert.equal(retryStatus.status, 'unsafe') @@ -221,12 +281,15 @@ test('hanging command times out unsafe and later accepted command commits', asyn test('hanging load worker fails startup explicitly', async t => { const fixture = await makeServerFixture(t) const loadWorker = path.join(fixture.dir, 'hang-load-worker.mjs') - await fs.writeFile(loadWorker, `import { parentPort } from 'node:worker_threads' + await fs.writeFile( + loadWorker, + `import { parentPort } from 'node:worker_threads' parentPort.on('message', () => { while (true) {} }) -`) +` + ) const port = await getOpenPort() const running = startServer(t, fixture, { port, @@ -244,7 +307,11 @@ test('integrity-enabled command writes changeset digest before done status', asy const fixture = await makeServerFixture(t) const port = await getOpenPort() const integrityFile = getDefaultIntegrityFile(fixture.datafile) - await appendIntegrityRecord(integrityFile, fixture.datafile, await fs.readFile(fixture.datafile)) + await appendIntegrityRecord( + integrityFile, + fixture.datafile, + await fs.readFile(fixture.datafile) + ) const running = startServer(t, fixture, { port, @@ -256,7 +323,7 @@ test('integrity-enabled command writes changeset digest before done status', asy const response = await postCommand(port, { id: commandId, name: 'addPerson', - value: {name: 'Ada'} + value: { name: 'Ada' } }) assert.equal(response.status, 202) await waitForCommandStatus(port, commandId, 'done') @@ -282,13 +349,13 @@ test('integrity-enabled command writes changeset digest before done status', asy test('normal restart preserves committed state according to reconstruction oracle', async t => { const fixture = await makeServerFixture(t) const port = await getOpenPort() - const first = startServer(t, fixture, {port}) + const first = startServer(t, fixture, { port }) await waitForServer(first.child, first.getOutput, port) const response = await postCommand(port, { id: 'normal-command', name: 'addPerson', - value: {name: 'Normal'} + value: { name: 'Normal' } }) assert.equal(response.status, 202) await waitForCommandStatus(port, 'normal-command', 'done') @@ -298,7 +365,7 @@ test('normal restart preserves committed state according to reconstruction oracl await assertServerStateMatchesOracle(port, fixture, ['Normal']) await stopServer(first.child) - const second = startServer(t, fixture, {port}) + const second = startServer(t, fixture, { port }) await waitForServer(second.child, second.getOutput, port) await assertServerStateMatchesOracle(port, fixture, ['Normal']) diff --git a/test/durability-helpers.mjs b/test/durability-helpers.mjs index be32b36..5957609 100644 --- a/test/durability-helpers.mjs +++ b/test/durability-helpers.mjs @@ -10,7 +10,9 @@ import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import { getChangesetPath } from '../src/recovery.mjs' -export const rootDir = path.dirname(path.dirname(fileURLToPath(import.meta.url))) +export const rootDir = path.dirname( + path.dirname(fileURLToPath(import.meta.url)) +) const serverModule = path.join(rootDir, 'src/server.mjs') export async function getOpenPort() { @@ -26,7 +28,7 @@ export async function getOpenPort() { export async function makeServerFixture(t, options = {}) { const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-crash-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) + t.after(() => fs.rm(dir, { recursive: true, force: true })) const datafile = path.join(dir, 'data.jsontag') const commandsFile = path.join(dir, 'commands.mjs') @@ -37,21 +39,43 @@ export async function makeServerFixture(t, options = {}) { await fs.writeFile(commandLog, '') await fs.writeFile(commandStatus, '') - await fs.writeFile(datafile, serialize(JSONTag.parse(options.initialData || '{"persons":[]}'))) - await fs.writeFile(commandsFile, options.commandsSource || `export default { + await fs.writeFile( + datafile, + serialize(JSONTag.parse(options.initialData || '{"persons":[]}')) + ) + await fs.writeFile( + commandsFile, + options.commandsSource || + `export default { addPerson: (dataspace, command) => { dataspace.persons.push(command.value) } } -`) - await fs.writeFile(indexFile, options.indexSource || 'export default { create() {}, update() {}, load() { return {} } }\n') - await fs.writeFile(runner, `import SimplyStore from ${JSON.stringify(serverModule)} +` + ) + await fs.writeFile( + indexFile, + options.indexSource || + 'export default { create() {}, update() {}, load() { return {} } }\n' + ) + await fs.writeFile( + runner, + `import SimplyStore from ${JSON.stringify(serverModule)} const options = JSON.parse(process.env.SIMPLYSTORE_TEST_OPTIONS) SimplyStore.run(options) -`) +` + ) - return {dir, datafile, commandsFile, indexFile, commandLog, commandStatus, runner} + return { + dir, + datafile, + commandsFile, + indexFile, + commandLog, + commandStatus, + runner + } } export function startServer(t, fixture, options = {}) { @@ -91,7 +115,7 @@ export function startServer(t, fixture, options = {}) { t.after(() => stopServer(child)) - return {child, getOutput: () => output} + return { child, getOutput: () => output } } export async function stopServer(child) { @@ -107,12 +131,20 @@ export async function waitForServer(child, getOutput, port) { const timeout = setTimeout(() => { finished = true child.off('exit', onExit) - reject(new Error(`Server did not start on port ${port}:\n${getOutput()}`)) + reject( + new Error( + `Server did not start on port ${port}:\n${getOutput()}` + ) + ) }, 5000) const onExit = (code, signal) => { clearTimeout(timeout) finished = true - reject(new Error(`Server exited before startup (${code || signal}):\n${getOutput()}`)) + reject( + new Error( + `Server exited before startup (${code || signal}):\n${getOutput()}` + ) + ) } const checkReady = () => { if (finished) { @@ -135,7 +167,7 @@ export async function waitForServer(child, getOutput, port) { export async function waitForExit(child, timeoutMs = 5000) { return new Promise((resolve, reject) => { if (child.exitCode !== null || child.signalCode) { - resolve({code: child.exitCode, signal: child.signalCode}) + resolve({ code: child.exitCode, signal: child.signalCode }) return } const timeout = setTimeout(() => { @@ -144,7 +176,7 @@ export async function waitForExit(child, timeoutMs = 5000) { }, timeoutMs) const onExit = (code, signal) => { clearTimeout(timeout) - resolve({code, signal}) + resolve({ code, signal }) } child.once('exit', onExit) }) @@ -172,20 +204,27 @@ export async function queryPersons(port) { signal: AbortSignal.timeout(5000) }) if (!response.ok) { - throw new Error(`Query failed with ${response.status}: ${await response.text()}`) + throw new Error( + `Query failed with ${response.status}: ${await response.text()}` + ) } return JSONTag.parse(await response.text()) } export async function getCommandStatus(port, commandId) { - const response = await fetch(`http://127.0.0.1:${port}/command/${commandId}`, { - headers: { - accept: 'application/json' - }, - signal: AbortSignal.timeout(5000) - }) + const response = await fetch( + `http://127.0.0.1:${port}/command/${commandId}`, + { + headers: { + accept: 'application/json' + }, + signal: AbortSignal.timeout(5000) + } + ) if (!response.ok) { - throw new Error(`Command status failed with ${response.status}: ${await response.text()}`) + throw new Error( + `Command status failed with ${response.status}: ${await response.text()}` + ) } return response.json() } @@ -230,7 +269,9 @@ export async function reconstructCommittedDataset(fixture) { if (applied.has(command.id) || finalStatus.get(command.id) !== 'done') { continue } - data = parser.parse(await fs.readFile(getChangesetPath(fixture.datafile, command.id))) + data = parser.parse( + await fs.readFile(getChangesetPath(fixture.datafile, command.id)) + ) applied.add(command.id) } return data @@ -243,11 +284,17 @@ export async function reconstructCommittedPersonNames(fixture) { // Explicit administrator action in disposable fixtures whose writer has exited. export async function unlockStoppedFixture(t, fixture) { - const {releaseOfflineLocks} = await import('../src/admin-recovery.mjs') - const {publishFile} = await import('../src/storage.mjs') - const audit = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-unlock-audit-')) - t.after(() => fs.rm(audit, {recursive:true,force:true})) - const release = await releaseOfflineLocks(fixture, {operator:'test administrator',reason:'Fixture writer has exited; inspect before startup',confirmedStopped:true}) - await publishFile(path.join(audit,'release.json'),JSON.stringify(release)) + const { releaseOfflineLocks } = await import('../src/admin-recovery.mjs') + const { publishFile } = await import('../src/storage.mjs') + const audit = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-unlock-audit-') + ) + t.after(() => fs.rm(audit, { recursive: true, force: true })) + const release = await releaseOfflineLocks(fixture, { + operator: 'test administrator', + reason: 'Fixture writer has exited; inspect before startup', + confirmedStopped: true + }) + await publishFile(path.join(audit, 'release.json'), JSON.stringify(release)) await release.finish() } diff --git a/test/durability-latency.mjs b/test/durability-latency.mjs index 2284432..c5d4926 100644 --- a/test/durability-latency.mjs +++ b/test/durability-latency.mjs @@ -1,28 +1,69 @@ import process from 'node:process' import test from 'node:test' import assert from 'node:assert/strict' -import {performance} from 'node:perf_hooks' -import {execFileSync} from 'node:child_process' -import {makeServerFixture,getOpenPort,startServer,waitForServer,postCommand,getCommandStatus} from './durability-helpers.mjs' +import { performance } from 'node:perf_hooks' +import { execFileSync } from 'node:child_process' +import { + makeServerFixture, + getOpenPort, + startServer, + waitForServer, + postCommand, + getCommandStatus +} from './durability-helpers.mjs' -test('measure local startup, acceptance, and completion latency',async t=>{ - const store=await makeServerFixture(t),port=await getOpenPort() - const boot=performance.now(),server=startServer(t,store,{port}) - await waitForServer(server.child,server.getOutput,port) - const startupMs=performance.now()-boot,accepted=[],completed=[] - for(let i=0;i<30;i++){ - const start=performance.now(),id=`latency-${i}` - assert.equal((await postCommand(port,{id,name:'addPerson',value:{name:id}})).status,202) - accepted.push(performance.now()-start) - for(;;){ - const status=await getCommandStatus(port,id);if(status.status==='done'){ - break; - }assert.equal(['accepted','active'].includes(status.status),true);await new Promise(r=>setTimeout(r,2)) +test('measure local startup, acceptance, and completion latency', async t => { + const store = await makeServerFixture(t), + port = await getOpenPort() + const boot = performance.now(), + server = startServer(t, store, { port }) + await waitForServer(server.child, server.getOutput, port) + const startupMs = performance.now() - boot, + accepted = [], + completed = [] + for (let i = 0; i < 30; i++) { + const start = performance.now(), + id = `latency-${i}` + assert.equal( + ( + await postCommand(port, { + id, + name: 'addPerson', + value: { name: id } + }) + ).status, + 202 + ) + accepted.push(performance.now() - start) + for (;;) { + const status = await getCommandStatus(port, id) + if (status.status === 'done') { + break + } + assert.equal(['accepted', 'active'].includes(status.status), true) + await new Promise(r => setTimeout(r, 2)) } - completed.push(performance.now()-start) + completed.push(performance.now() - start) } - const stats=values=>{ - const v=[...values].sort((a,b)=>a-b);return {p50Ms:v[Math.floor(v.length*.5)],p95Ms:v[Math.floor(v.length*.95)],maxMs:v.at(-1)} + const stats = values => { + const v = [...values].sort((a, b) => a - b) + return { + p50Ms: v[Math.floor(v.length * 0.5)], + p95Ms: v[Math.floor(v.length * 0.95)], + maxMs: v.at(-1) + } } - t.diagnostic(JSON.stringify({samples:30,startupMs,acceptance:stats(accepted),completion:stats(completed),filesystem:execFileSync('stat',['-f','-c','%T',store.dir],{encoding:'utf8'}).trim(),node:process.version,notes:'Single localhost client; completion includes a fresh worker and polling; no production performance claim'})) + t.diagnostic( + JSON.stringify({ + samples: 30, + startupMs, + acceptance: stats(accepted), + completion: stats(completed), + filesystem: execFileSync('stat', ['-f', '-c', '%T', store.dir], { + encoding: 'utf8' + }).trim(), + node: process.version, + notes: 'Single localhost client; completion includes a fresh worker and polling; no production performance claim' + }) + ) }) diff --git a/test/durability-startup.mjs b/test/durability-startup.mjs index 89e9055..b5ad9cc 100644 --- a/test/durability-startup.mjs +++ b/test/durability-startup.mjs @@ -9,7 +9,10 @@ import { fileURLToPath } from 'node:url' import JSONTag from '@muze-nl/jsontag' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' -import { appendIntegrityRecord, getDefaultIntegrityFile } from '../src/integrity.mjs' +import { + appendIntegrityRecord, + getDefaultIntegrityFile +} from '../src/integrity.mjs' import { getCommittedCommandIds, getChangesetPath, @@ -17,7 +20,7 @@ import { loadCommandLog, loadCommandStatus, recoverActiveCommands, - RecoveryIntegrityError, + RecoveryIntegrityError } from '../src/recovery.mjs' const rootDir = path.dirname(path.dirname(fileURLToPath(import.meta.url))) @@ -29,21 +32,29 @@ function parseOd(buffer) { } async function makeFixture(t) { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-durability-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) + const dir = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-durability-') + ) + t.after(() => fs.rm(dir, { recursive: true, force: true })) const dataFile = path.join(dir, 'data.jsontag') const indexFile = path.join(dir, 'index.mjs') const base = serialize(JSONTag.parse('{"persons":[]}')) await fs.writeFile(dataFile, base) - await fs.writeFile(indexFile, 'export default { create() {}, update() {}, load() { return {} } }\n') + await fs.writeFile( + indexFile, + 'export default { create() {}, update() {}, load() { return {} } }\n' + ) - return {dir, dataFile, indexFile} + return { dir, dataFile, indexFile } } async function writeJsonTagLines(file, records) { - await fs.writeFile(file, records.map(record => JSONTag.stringify(record)).join('\n') + '\n') + await fs.writeFile( + file, + records.map(record => JSONTag.stringify(record)).join('\n') + '\n' + ) } async function writeChangeset(dataFile, commandId, change) { @@ -54,14 +65,26 @@ async function writeChangeset(dataFile, commandId, change) { change(data) const extension = dataFile.split('.').pop() - const basefile = dataFile.substring(0, dataFile.length - (extension.length + 1)) - await fs.writeFile(`${basefile}.${commandId}.${extension}`, serialize(data, { - meta: parser.meta, - changes: true - })) + const basefile = dataFile.substring( + 0, + dataFile.length - (extension.length + 1) + ) + await fs.writeFile( + `${basefile}.${commandId}.${extension}`, + serialize(data, { + meta: parser.meta, + changes: true + }) + ) } -function loadDataset({dataFile, indexFile, commands, integrityFile = null, integrityRequired = false}) { +function loadDataset({ + dataFile, + indexFile, + commands, + integrityFile = null, + integrityRequired = false +}) { return new Promise((resolve, reject) => { const worker = new Worker(loadWorker) worker.on('message', result => { @@ -90,11 +113,14 @@ async function writeIntegrityEntry(integrityFile, file) { test('accepted command changeset is not treated as committed state on startup', async t => { const fixture = await makeFixture(t) await writeChangeset(fixture.dataFile, 'accepted-command', data => { - data.persons.push({name: 'Ada'}) + data.persons.push({ name: 'Ada' }) }) const status = new Map([ - ['accepted-command', {command: 'accepted-command', code: 202, status: 'accepted'}] + [ + 'accepted-command', + { command: 'accepted-command', code: 202, status: 'accepted' } + ] ]) const result = await loadDataset({ ...fixture, @@ -102,13 +128,24 @@ test('accepted command changeset is not treated as committed state on startup', }) const data = parseOd(result.data) - assert.equal(data.persons.length, 0, 'accepted-but-not-done changesets must not become visible after restart') + assert.equal( + data.persons.length, + 0, + 'accepted-but-not-done changesets must not become visible after restart' + ) }) test('done command with missing changeset refuses recovery instead of silently using base data', async t => { const fixture = await makeFixture(t) const status = new Map([ - ['done-command-with-missing-changeset', {command: 'done-command-with-missing-changeset', code: 200, status: 'done'}] + [ + 'done-command-with-missing-changeset', + { + command: 'done-command-with-missing-changeset', + code: 200, + status: 'done' + } + ] ]) await assert.rejects( @@ -137,9 +174,15 @@ test('malformed base OD-JSONTag file refuses recovery', async t => { test('malformed committed changeset refuses recovery', async t => { const fixture = await makeFixture(t) const status = new Map([ - ['malformed-changeset', {command: 'malformed-changeset', code: 200, status: 'done'}] + [ + 'malformed-changeset', + { command: 'malformed-changeset', code: 200, status: 'done' } + ] ]) - await fs.writeFile(getChangesetPath(fixture.dataFile, 'malformed-changeset'), '{"persons":[{"name":"Bad"}]}') + await fs.writeFile( + getChangesetPath(fixture.dataFile, 'malformed-changeset'), + '{"persons":[{"name":"Bad"}]}' + ) await assert.rejects( loadDataset({ @@ -154,14 +197,17 @@ test('truncated committed changeset payload refuses recovery before lazy parsing const fixture = await makeFixture(t) const commandId = 'truncated-changeset' const status = new Map([ - [commandId, {command: commandId, code: 200, status: 'done'}] + [commandId, { command: commandId, code: 200, status: 'done' }] ]) await writeChangeset(fixture.dataFile, commandId, data => { - data.persons.push({name: 'Truncated'}) + data.persons.push({ name: 'Truncated' }) }) const changesetPath = getChangesetPath(fixture.dataFile, commandId) const changeset = await fs.readFile(changesetPath) - await fs.writeFile(changesetPath, changeset.subarray(0, changeset.length - 3)) + await fs.writeFile( + changesetPath, + changeset.subarray(0, changeset.length - 3) + ) await assert.rejects( loadDataset({ @@ -174,9 +220,15 @@ test('truncated committed changeset payload refuses recovery before lazy parsing test('malformed uncommitted changeset is ignored during committed startup reconstruction', async t => { const fixture = await makeFixture(t) - await fs.writeFile(getChangesetPath(fixture.dataFile, 'accepted-command'), '{"persons":[{"name":"Ignored"}]}') + await fs.writeFile( + getChangesetPath(fixture.dataFile, 'accepted-command'), + '{"persons":[{"name":"Ignored"}]}' + ) const status = new Map([ - ['accepted-command', {command: 'accepted-command', code: 202, status: 'accepted'}] + [ + 'accepted-command', + { command: 'accepted-command', code: 202, status: 'accepted' } + ] ]) const result = await loadDataset({ @@ -193,18 +245,18 @@ test('durable status file selects only done command changesets for startup', asy const statusFile = path.join(fixture.dir, 'command-status.jsontag') await writeChangeset(fixture.dataFile, 'accepted-command', data => { - data.persons.push({name: 'Accepted'}) + data.persons.push({ name: 'Accepted' }) }) await writeChangeset(fixture.dataFile, 'done-command', data => { - data.persons.push({name: 'Done'}) + data.persons.push({ name: 'Done' }) }) await writeJsonTagLines(statusFile, [ - {command: 'accepted-command', code: 202, status: 'accepted'}, - {command: 'done-command', code: 202, status: 'accepted'}, - {command: 'done-command', code: 200, status: 'done'}, - {command: 'failed-command', code: 202, status: 'accepted'}, - {command: 'failed-command', code: 500, status: 'failed'} + { command: 'accepted-command', code: 202, status: 'accepted' }, + { command: 'done-command', code: 202, status: 'accepted' }, + { command: 'done-command', code: 200, status: 'done' }, + { command: 'failed-command', code: 202, status: 'accepted' }, + { command: 'failed-command', code: 500, status: 'failed' } ]) const status = loadCommandStatus(statusFile) @@ -214,7 +266,10 @@ test('durable status file selects only done command changesets for startup', asy }) const data = parseOd(result.data) - assert.deepEqual(data.persons.map(person => person.name), ['Done']) + assert.deepEqual( + data.persons.map(person => person.name), + ['Done'] + ) }) test('durable command log replays only accepted commands', async t => { @@ -223,18 +278,22 @@ test('durable command log replays only accepted commands', async t => { const commandLog = path.join(fixture.dir, 'command-log.jsontag') await writeJsonTagLines(statusFile, [ - {command: 'accepted-command', code: 202, status: 'accepted'}, - {command: 'done-command', code: 200, status: 'done'}, - {command: 'failed-command', code: 500, status: 'failed'} + { command: 'accepted-command', code: 202, status: 'accepted' }, + { command: 'done-command', code: 200, status: 'done' }, + { command: 'failed-command', code: 500, status: 'failed' } ]) await writeJsonTagLines(commandLog, [ - {id: 'accepted-command', name: 'addPerson', value: {name: 'Accepted'}}, - {id: 'done-command', name: 'addPerson', value: {name: 'Done'}}, - {id: 'failed-command', name: 'addPerson', value: {name: 'Failed'}} + { + id: 'accepted-command', + name: 'addPerson', + value: { name: 'Accepted' } + }, + { id: 'done-command', name: 'addPerson', value: { name: 'Done' } }, + { id: 'failed-command', name: 'addPerson', value: { name: 'Failed' } } ]) const taskDefaults = { - meta: {source: 'test-meta'}, + meta: { source: 'test-meta' }, data: ['test-data'], commandsFile: '/commands.mjs', indexFile: '/index.mjs', @@ -248,7 +307,11 @@ test('durable command log replays only accepted commands', async t => { assert.deepEqual(commands[0], { ...taskDefaults, id: 'accepted-command', - command: JSONTag.stringify({id: 'accepted-command', name: 'addPerson', value: {name: 'Accepted'}}), + command: JSONTag.stringify({ + id: 'accepted-command', + name: 'addPerson', + value: { name: 'Accepted' } + }) }) assert.equal(JSONTag.parse(commands[0].command).name, 'addPerson') }) @@ -257,12 +320,17 @@ for (const attempt of [1, 2]) { test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { const fixture = await makeFixture(t) const file = path.join(fixture.dir, 'status.jsontag') - await writeJsonTagLines(file, [{command:'A',status:'active',attempt}]) + await writeJsonTagLines(file, [ + { command: 'A', status: 'active', attempt } + ]) const before = await fs.readFile(file) const status = loadCommandStatus(file) - await assert.rejects(recoverActiveCommands(status,file,{maxCrashAttempts:2}), /administrator assessment/) - assert.equal(status.get('A').status,'active') - assert.deepEqual(await fs.readFile(file),before) + await assert.rejects( + recoverActiveCommands(status, file, { maxCrashAttempts: 2 }), + /administrator assessment/ + ) + assert.equal(status.get('A').status, 'active') + assert.deepEqual(await fs.readFile(file), before) }) } @@ -289,9 +357,7 @@ test('malformed durable status record refuses recovery with explicit integrity e test('structurally invalid durable status record refuses recovery', async t => { const fixture = await makeFixture(t) const statusFile = path.join(fixture.dir, 'command-status.jsontag') - await writeJsonTagLines(statusFile, [ - {code: 202, status: 'accepted'} - ]) + await writeJsonTagLines(statusFile, [{ code: 202, status: 'accepted' }]) let error try { @@ -334,7 +400,7 @@ test('structurally invalid durable command log record refuses recovery', async t const commandLog = path.join(fixture.dir, 'command-log.jsontag') const status = new Map() await writeJsonTagLines(commandLog, [ - {name: 'addPerson', value: {name: 'No id'}} + { name: 'addPerson', value: { name: 'No id' } } ]) let error @@ -353,11 +419,18 @@ test('structurally invalid durable command log record refuses recovery', async t }) test('OD-JSONTag framing validation catches truncated lazy records', () => { - const buffer = Buffer.from('(16){"persons":[~1]}\n(14){"name":"Ada"', 'utf8') + const buffer = Buffer.from( + '(16){"persons":[~1]}\n(14){"name":"Ada"', + 'utf8' + ) let error try { - assertOdJsonTagFraming(buffer, 'data.truncated.jsontag', 'changeset OD-JSONTag data') + assertOdJsonTagFraming( + buffer, + 'data.truncated.jsontag', + 'changeset OD-JSONTag data' + ) } catch (caught) { error = caught @@ -370,10 +443,17 @@ test('OD-JSONTag framing validation catches truncated lazy records', () => { }) test('OD-JSONTag framing validation accepts changeset skip records', () => { - const buffer = Buffer.from('(18){"persons":[~1-2]}\n+1\n(15){"name":"Once"}', 'utf8') + const buffer = Buffer.from( + '(18){"persons":[~1-2]}\n+1\n(15){"name":"Once"}', + 'utf8' + ) assert.doesNotThrow(() => { - assertOdJsonTagFraming(buffer, 'data.patch.jsontag', 'changeset OD-JSONTag data') + assertOdJsonTagFraming( + buffer, + 'data.patch.jsontag', + 'changeset OD-JSONTag data' + ) }) }) @@ -400,12 +480,12 @@ test('integrity manifest detects same-length altered committed changeset payload const fixture = await makeFixture(t) const commandId = 'tampered-changeset' const status = new Map([ - [commandId, {command: commandId, code: 200, status: 'done'}] + [commandId, { command: commandId, code: 200, status: 'done' }] ]) const integrityFile = getDefaultIntegrityFile(fixture.dataFile) await writeIntegrityEntry(integrityFile, fixture.dataFile) await writeChangeset(fixture.dataFile, commandId, data => { - data.persons.push({name: 'Ada'}) + data.persons.push({ name: 'Ada' }) }) const changesetPath = getChangesetPath(fixture.dataFile, commandId) await writeIntegrityEntry(integrityFile, changesetPath) diff --git a/test/power-loss-baseline.mjs b/test/power-loss-baseline.mjs index 6a88db0..e88b00d 100644 --- a/test/power-loss-baseline.mjs +++ b/test/power-loss-baseline.mjs @@ -1,169 +1,339 @@ -import {Buffer} from 'node:buffer' +import { Buffer } from 'node:buffer' // The original baseline is retained at EVD-20260919-TTZ7C-23's Git version. -// These assertions implement the corrected command-only / authoritative-log contracts. +// These assertions implement the corrected command-only / authoritative-log +// contracts. import test from 'node:test' import assert from 'node:assert/strict' import fs from 'node:fs/promises' import path from 'node:path' import JSONTag from '@muze-nl/jsontag' -import {fixture,launch,submit,completion,eventually,rows,crash,copyImage,loseUnpublishedName} from './power-loss-helpers.mjs' -import {getOpenPort,waitForServer,waitForExit,queryPersons,readCommandLogRecords,readCommandStatusRecords,unlockStoppedFixture} from './durability-helpers.mjs' -import {inspectStore} from '../src/store-inspection.mjs' +import { + fixture, + launch, + submit, + completion, + eventually, + rows, + crash, + copyImage, + loseUnpublishedName +} from './power-loss-helpers.mjs' +import { + getOpenPort, + waitForServer, + waitForExit, + queryPersons, + readCommandLogRecords, + readCommandStatusRecords, + unlockStoppedFixture +} from './durability-helpers.mjs' +import { inspectStore } from '../src/store-inspection.mjs' -async function ready(t,store,options={}) { - const port=await getOpenPort(), server=await launch(t,store,port,options) - await waitForServer(server.child,server.getOutput,port) - return {...server,port} +async function ready(t, store, options = {}) { + const port = await getOpenPort(), + server = await launch(t, store, port, options) + await waitForServer(server.child, server.getOutput, port) + return { ...server, port } } -const command=id=>({id,name:'addPerson',value:{name:id}}) -async function commit(store,server,id='A') { - assert.equal((await submit(store,server.port,command(id))).http,202) - assert.equal((await completion(store,server.port,id)).status,'done') +const command = id => ({ id, name: 'addPerson', value: { name: id } }) +async function commit(store, server, id = 'A') { + assert.equal((await submit(store, server.port, command(id))).http, 202) + assert.equal((await completion(store, server.port, id)).status, 'done') } -test('PL01-05 durable publication survives modeled loss of unsynced names and restart',async t=>{ - const store=await fixture(t),server=await ready(t,store) - await commit(store,server) - const trace=await rows(store.trace) - const changeset=path.join(store.dir,'data.A.jsontag') - const rename=trace.findIndex(e=>e.op==='rename'&&e.file===changeset) - assert.ok(rename>0) - assert.ok(trace.slice(0,rename).some(e=>e.op==='sync'&&e.file===trace[rename].from)) - const dirSync=trace.findIndex((e,i)=>i>rename&&e.op==='sync'&&e.file===store.dir) - const doneSync=trace.findIndex((e,i)=>i>dirSync&&e.op==='datasync'&&e.file===store.commandStatus&&Buffer.from(e.bytes,'base64').toString().includes('"status":"done"')) - assert.ok(dirSync>rename&&doneSync>dirSync) +test('PL01-05 durable publication survives modeled loss of unsynced names and restart', async t => { + const store = await fixture(t), + server = await ready(t, store) + await commit(store, server) + const trace = await rows(store.trace) + const changeset = path.join(store.dir, 'data.A.jsontag') + const rename = trace.findIndex( + e => e.op === 'rename' && e.file === changeset + ) + assert.ok(rename > 0) + assert.ok( + trace + .slice(0, rename) + .some(e => e.op === 'sync' && e.file === trace[rename].from) + ) + const dirSync = trace.findIndex( + (e, i) => i > rename && e.op === 'sync' && e.file === store.dir + ) + const doneSync = trace.findIndex( + (e, i) => + i > dirSync && + e.op === 'datasync' && + e.file === store.commandStatus && + Buffer.from(e.bytes, 'base64') + .toString() + .includes('"status":"done"') + ) + assert.ok(dirSync > rename && doneSync > dirSync) await crash(server.child) - const image=await copyImage(t,store) - assert.equal(await loseUnpublishedName(store,image,changeset),false) - await unlockStoppedFixture(t,image) - const restarted=await ready(t,image) - assert.deepEqual((await queryPersons(restarted.port)).map(p=>p.name),['A']) + const image = await copyImage(t, store) + assert.equal(await loseUnpublishedName(store, image, changeset), false) + await unlockStoppedFixture(t, image) + const restarted = await ready(t, image) + assert.deepEqual( + (await queryPersons(restarted.port)).map(p => p.name), + ['A'] + ) }) -test('PL10 an actual short changeset write is completed before acknowledgment',async t=>{ - const store=await fixture(t),server=await ready(t,store,{fault:{op:'write',file:'data.A.jsontag.',action:'short-write'}}) - await commit(store,server) - const trace=await rows(store.trace) - assert.ok(trace.some(e=>e.op==='injected'&&e.action==='short-write')) - assert.ok(trace.filter(e=>e.op==='write'&&path.basename(e.file).startsWith('data.A.jsontag.')).length>=2) - assert.deepEqual((await inspectStore(store)).data.persons.map(p=>p.name),['A']) +test('PL10 an actual short changeset write is completed before acknowledgment', async t => { + const store = await fixture(t), + server = await ready(t, store, { + fault: { + op: 'write', + file: 'data.A.jsontag.', + action: 'short-write' + } + }) + await commit(store, server) + const trace = await rows(store.trace) + assert.ok( + trace.some(e => e.op === 'injected' && e.action === 'short-write') + ) + assert.ok( + trace.filter( + e => + e.op === 'write' && + path.basename(e.file).startsWith('data.A.jsontag.') + ).length >= 2 + ) + assert.deepEqual( + (await inspectStore(store)).data.persons.map(p => p.name), + ['A'] + ) }) -for (const [name,op,file,contains] of [ - ['log sync','datasync','command-log.jsontag','"id":"A"'], - ['acceptance sync','datasync','command-status.jsontag','"status":"accepted"'], - ['log close','close:after','command-log.jsontag','"id":"A"'], - ['changeset sync','sync','data.A.jsontag.'], - ['changeset rename','rename','data.A.jsontag'], - ['done sync','datasync','command-status.jsontag','"status":"done"'] +for (const [name, op, file, contains] of [ + ['log sync', 'datasync', 'command-log.jsontag', '"id":"A"'], + [ + 'acceptance sync', + 'datasync', + 'command-status.jsontag', + '"status":"accepted"' + ], + ['log close', 'close:after', 'command-log.jsontag', '"id":"A"'], + ['changeset sync', 'sync', 'data.A.jsontag.'], + ['changeset rename', 'rename', 'data.A.jsontag'], + ['done sync', 'datasync', 'command-status.jsontag', '"status":"done"'] ]) { - test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`,async t=>{ - const store=await fixture(t),server=await ready(t,store,{fault:{op,file,contains,action:'error'}}) + test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`, async t => { + const store = await fixture(t), + server = await ready(t, store, { + fault: { op, file, contains, action: 'error' } + }) try { - await submit(store,server.port,command('A')) + await submit(store, server.port, command('A')) } - catch(error){ - assert.match(error.message,/fetch failed|terminated|socket/) + catch (error) { + assert.match(error.message, /fetch failed|terminated|socket/) } - assert.equal((await waitForExit(server.child)).code,1) - assert.match(server.getOutput(),/Storage outcome uncertain/) - assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) - const status=await readCommandStatusRecords(store) - // Failed sync can leave a complete but uncertain done record; never append failed after it. - const done=status.findIndex(s=>s.status==='done') - if(done>=0){ - assert.equal(status.slice(done+1).some(s=>s.status==='failed'),false) + assert.equal((await waitForExit(server.child)).code, 1) + assert.match(server.getOutput(), /Storage outcome uncertain/) + assert.ok((await rows(store.trace)).some(e => e.op === 'injected')) + const status = await readCommandStatusRecords(store) + // Failed sync can leave a complete but uncertain done record; never + // append failed after it. + const done = status.findIndex(s => s.status === 'done') + if (done >= 0) { + assert.equal( + status.slice(done + 1).some(s => s.status === 'failed'), + false + ) } else { - assert.equal(status.some(s=>s.status==='done'),false) + assert.equal( + status.some(s => s.status === 'done'), + false + ) } }) } -test('PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip',async t=>{ - const store=await fixture(t,{commandsSource:`export default {addPerson(data,command,request,meta){ +test('PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip', async t => { + const store = await fixture(t, { + commandsSource: `export default {addPerson(data,command,request,meta){ if(request!==undefined)throw new Error('unlogged input leaked') if(!meta.index.id)throw new Error('index unavailable') data.persons.push(command.value) - }}`}) - const worker=path.join(store.dir,'custom-worker.mjs') - await fs.writeFile(worker,`import {parentPort} from 'node:worker_threads' -import runCommand,{initialize} from ${JSON.stringify(new URL('../src/command-worker-module.mjs',import.meta.url).href)} -parentPort.on('message',async task=>{if('request' in task)throw new Error('request leaked to worker');await initialize(task);parentPort.postMessage(await runCommand(task.command,{query:{name:'wrong'}}))})`) - const server=await ready(t,store,{server:{commandWorker:worker}}) - const input={id:'A',name:'addPerson',value:{name:'cafΓ© ζΌ’ε­— πŸ˜€',date:new Date('2026-09-19T00:00:00Z')}} - assert.equal((await submit(store,server.port,input,'?name=wrong')).http,202) - assert.equal((await completion(store,server.port,'A')).status,'done') - assert.equal(JSONTag.stringify((await readCommandLogRecords(store))[0]),JSONTag.stringify(input)) - assert.equal((await queryPersons(server.port))[0].name,input.value.name) + }}` + }) + const worker = path.join(store.dir, 'custom-worker.mjs') + await fs.writeFile( + worker, + `import {parentPort} from 'node:worker_threads' +import runCommand,{initialize} from ${JSON.stringify(new URL('../src/command-worker-module.mjs', import.meta.url).href)} +parentPort.on('message',async task=>{if('request' in task)throw new Error('request leaked to worker');await initialize(task);parentPort.postMessage(await runCommand(task.command,{query:{name:'wrong'}}))})` + ) + const server = await ready(t, store, { server: { commandWorker: worker } }) + const input = { + id: 'A', + name: 'addPerson', + value: { name: 'cafΓ© ζΌ’ε­— πŸ˜€', date: new Date('2026-09-19T00:00:00Z') } + } + assert.equal( + (await submit(store, server.port, input, '?name=wrong')).http, + 202 + ) + assert.equal((await completion(store, server.port, 'A')).status, 'done') + assert.equal( + JSONTag.stringify((await readCommandLogRecords(store))[0]), + JSONTag.stringify(input) + ) + assert.equal((await queryPersons(server.port))[0].name, input.value.name) }) -for(const duplicate of [false,true]){ - test(`PL16 delayed append cannot be overtaken${duplicate?' by duplicate ID':''}`,async t=>{ - const store=await fixture(t),server=await ready(t,store,{fault:{op:'datasync:after',file:'command-log.jsontag',contains:'"id":"A"',action:'pause'}}) - const a=submit(store,server.port,command('A')) - await eventually(async()=> (await rows(store.trace)).some(e=>e.op==='injected'),'A append gate') - const b=submit(store,server.port,command(duplicate?'A':'B')) - await new Promise(resolve=>setTimeout(resolve,30)) - assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),['A']) - await fs.writeFile(store.release,'') - assert.equal((await a).http,202);assert.ok([200,202].includes((await b).http)) - await completion(store,server.port,duplicate?'A':'B') - const expected=duplicate?['A']:['A','B'] - assert.deepEqual((await readCommandLogRecords(store)).map(c=>c.id),expected) - assert.deepEqual((await readCommandStatusRecords(store)).filter(s=>s.status==='active').map(s=>s.command),expected) - assert.deepEqual((await queryPersons(server.port)).map(p=>p.name),expected) +for (const duplicate of [false, true]) { + test(`PL16 delayed append cannot be overtaken${duplicate ? ' by duplicate ID' : ''}`, async t => { + const store = await fixture(t), + server = await ready(t, store, { + fault: { + op: 'datasync:after', + file: 'command-log.jsontag', + contains: '"id":"A"', + action: 'pause' + } + }) + const a = submit(store, server.port, command('A')) + await eventually( + async () => + (await rows(store.trace)).some(e => e.op === 'injected'), + 'A append gate' + ) + const b = submit(store, server.port, command(duplicate ? 'A' : 'B')) + await new Promise(resolve => setTimeout(resolve, 30)) + assert.deepEqual( + (await readCommandLogRecords(store)).map(c => c.id), + ['A'] + ) + await fs.writeFile(store.release, '') + assert.equal((await a).http, 202) + assert.ok([200, 202].includes((await b).http)) + await completion(store, server.port, duplicate ? 'A' : 'B') + const expected = duplicate ? ['A'] : ['A', 'B'] + assert.deepEqual( + (await readCommandLogRecords(store)).map(c => c.id), + expected + ) + assert.deepEqual( + (await readCommandStatusRecords(store)) + .filter(s => s.status === 'active') + .map(s => s.command), + expected + ) + assert.deepEqual( + (await queryPersons(server.port)).map(p => p.name), + expected + ) }) } -for(const artifact of ['command-log.jsontag','command-status.jsontag','data.A.jsontag']){ - test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`,async t=>{ - const store=await fixture(t),server=await ready(t,store);await commit(store,server) - await crash(server.child);await unlockStoppedFixture(t,store) - await fs.unlink(path.join(store.dir,artifact)) - const before=await fs.readdir(store.dir) - const restarted=await launch(t,store,await getOpenPort()) - assert.equal((await waitForExit(restarted.child)).code,1) - assert.match(restarted.getOutput(),/Administrative recovery required/) - assert.deepEqual((await fs.readdir(store.dir)).sort(),before.sort()) +for (const artifact of [ + 'command-log.jsontag', + 'command-status.jsontag', + 'data.A.jsontag' +]) { + test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`, async t => { + const store = await fixture(t), + server = await ready(t, store) + await commit(store, server) + await crash(server.child) + await unlockStoppedFixture(t, store) + await fs.unlink(path.join(store.dir, artifact)) + const before = await fs.readdir(store.dir) + const restarted = await launch(t, store, await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code, 1) + assert.match(restarted.getOutput(), /Administrative recovery required/) + assert.deepEqual((await fs.readdir(store.dir)).sort(), before.sort()) }) } -test('PL12/17 startup preserves uncertain active work without repeating external effects',async t=>{ - const store=await fixture(t,{commandsSource:`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify('/tmp/placeholder-effect')},'') ;data.persons.push(command.value)}}`}) - const witness=path.join(store.audit,'effects') - await fs.writeFile(store.commandsFile,`import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify(witness)},command.id+'\\n');data.persons.push(command.value)}}`) - const server=await ready(t,store,{fault:{op:'write',file:'command-status.jsontag',contains:'"status":"done"',action:'error'}}) - await submit(store,server.port,command('A'));await waitForExit(server.child).catch(error=>{ - throw new Error(server.getOutput(),{cause:error}) +test('PL12/17 startup preserves uncertain active work without repeating external effects', async t => { + const store = await fixture(t, { + commandsSource: `import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify('/tmp/placeholder-effect')},'') ;data.persons.push(command.value)}}` + }) + const witness = path.join(store.audit, 'effects') + await fs.writeFile( + store.commandsFile, + `import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify(witness)},command.id+'\\n');data.persons.push(command.value)}}` + ) + const server = await ready(t, store, { + fault: { + op: 'write', + file: 'command-status.jsontag', + contains: '"status":"done"', + action: 'error' + } }) - const before=await fs.readFile(store.commandStatus) - await unlockStoppedFixture(t,store) - const restarted=await launch(t,store,await getOpenPort()) - assert.equal((await waitForExit(restarted.child)).code,1) - assert.deepEqual(await fs.readFile(store.commandStatus),before) - assert.equal(await fs.readFile(witness,'utf8'),'A\n') + await submit(store, server.port, command('A')) + await waitForExit(server.child).catch(error => { + throw new Error(server.getOutput(), { cause: error }) + }) + const before = await fs.readFile(store.commandStatus) + await unlockStoppedFixture(t, store) + const restarted = await launch(t, store, await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code, 1) + assert.deepEqual(await fs.readFile(store.commandStatus), before) + assert.equal(await fs.readFile(witness, 'utf8'), 'A\n') }) -for(const operation of ['sync','close:after']){ - test(`directory ${operation} failure after rename prevents done`,async t=>{ - const store=await fixture(t) - const server=await ready(t,store,{fault:{op:operation,file:path.basename(store.dir),action:'error',after:{op:'rename',file:'data.A.jsontag'}}}) - await submit(store,server.port,command('A')) - assert.equal((await waitForExit(server.child)).code,1) - assert.ok((await rows(store.trace)).some(e=>e.op==='injected'&&e.file===store.dir)) - assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) +for (const operation of ['sync', 'close:after']) { + test(`directory ${operation} failure after rename prevents done`, async t => { + const store = await fixture(t) + const server = await ready(t, store, { + fault: { + op: operation, + file: path.basename(store.dir), + action: 'error', + after: { op: 'rename', file: 'data.A.jsontag' } + } + }) + await submit(store, server.port, command('A')) + assert.equal((await waitForExit(server.child)).code, 1) + assert.ok( + (await rows(store.trace)).some( + e => e.op === 'injected' && e.file === store.dir + ) + ) + assert.equal( + (await readCommandStatusRecords(store)).some( + c => c.status === 'done' + ), + false + ) }) } -for(const operation of ['open','write','close:after']){ - test(`changeset ${operation} error is not masked or acknowledged`,async t=>{ - const store=await fixture(t) - const server=await ready(t,store,{fault:{op:operation,file:'data.A.jsontag.',action:'error',code:operation==='write'?'ENOSPC':'EIO'}}) - await submit(store,server.port,command('A')) - assert.equal((await waitForExit(server.child)).code,1) - assert.equal((await readCommandStatusRecords(store)).some(c=>c.status==='done'),false) - assert.match(server.getOutput(),/Storage outcome uncertain/) +for (const operation of ['open', 'write', 'close:after']) { + test(`changeset ${operation} error is not masked or acknowledged`, async t => { + const store = await fixture(t) + const server = await ready(t, store, { + fault: { + op: operation, + file: 'data.A.jsontag.', + action: 'error', + code: operation === 'write' ? 'ENOSPC' : 'EIO' + } + }) + await submit(store, server.port, command('A')) + assert.equal((await waitForExit(server.child)).code, 1) + assert.equal( + (await readCommandStatusRecords(store)).some( + c => c.status === 'done' + ), + false + ) + assert.match(server.getOutput(), /Storage outcome uncertain/) }) } -test('short command-log append preserves its entire JSONTag record',async t=>{ - const store=await fixture(t) - const server=await ready(t,store,{fault:{op:'write',file:'command-log.jsontag',contains:'"id":"A"',action:'short-write'}}) - await commit(store,server) - assert.deepEqual((await readCommandLogRecords(store))[0],command('A')) - assert.ok((await rows(store.trace)).some(e=>e.op==='injected')) +test('short command-log append preserves its entire JSONTag record', async t => { + const store = await fixture(t) + const server = await ready(t, store, { + fault: { + op: 'write', + file: 'command-log.jsontag', + contains: '"id":"A"', + action: 'short-write' + } + }) + await commit(store, server) + assert.deepEqual((await readCommandLogRecords(store))[0], command('A')) + assert.ok((await rows(store.trace)).some(e => e.op === 'injected')) }) diff --git a/test/power-loss-helpers.mjs b/test/power-loss-helpers.mjs index 0678925..7b2d390 100644 --- a/test/power-loss-helpers.mjs +++ b/test/power-loss-helpers.mjs @@ -3,25 +3,36 @@ import fs from 'node:fs/promises' import os from 'node:os' import path from 'node:path' import process from 'node:process' -import {spawn} from 'node:child_process' -import {fileURLToPath} from 'node:url' +import { spawn } from 'node:child_process' +import { fileURLToPath } from 'node:url' import JSONTag from '@muze-nl/jsontag' -import {makeServerFixture, rootDir, stopServer, waitForExit, getCommandStatus} from './durability-helpers.mjs' +import { + makeServerFixture, + rootDir, + stopServer, + waitForExit, + getCommandStatus +} from './durability-helpers.mjs' export async function fixture(t, options = {}) { const store = await makeServerFixture(t, options) - const audit = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-observer-')) - t.after(() => fs.rm(audit, {recursive: true, force: true})) + const audit = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-observer-') + ) + t.after(() => fs.rm(audit, { recursive: true, force: true })) const trace = path.join(audit, 'io.jsonl') const journal = path.join(audit, 'client.jsonl') const release = path.join(audit, 'release') await fs.writeFile(trace, '') await fs.writeFile(journal, '') - return {...store, audit, trace, journal, release} + return { ...store, audit, trace, journal, release } } export async function rows(file) { - return (await fs.readFile(file, 'utf8')).split('\n').filter(Boolean).map(line => JSON.parse(line)) + return (await fs.readFile(file, 'utf8')) + .split('\n') + .filter(Boolean) + .map(line => JSON.parse(line)) } export async function observe(store, event) { @@ -29,10 +40,15 @@ export async function observe(store, event) { } export async function submit(store, port, command, query = '') { - await observe(store, {event: 'issued', command, query}) + await observe(store, { event: 'issued', command, query }) const response = await fetch(`http://127.0.0.1:${port}/command${query}`, { - method: 'POST', headers: {accept: 'application/json', 'content-type': 'application/jsontag'}, - body: JSONTag.stringify(command), signal: AbortSignal.timeout(5000) + method: 'POST', + headers: { + accept: 'application/json', + 'content-type': 'application/jsontag' + }, + body: JSONTag.stringify(command), + signal: AbortSignal.timeout(5000) }) const text = await response.text() let body @@ -40,10 +56,15 @@ export async function submit(store, port, command, query = '') { body = JSON.parse(text) } catch { - body = {raw: text} + body = { raw: text } } - await observe(store, {event: 'response', id: command.id, http: response.status, body}) - return {http: response.status, body} + await observe(store, { + event: 'response', + id: command.id, + http: response.status, + body + }) + return { http: response.status, body } } export async function eventually(fn, description, timeout = 6000) { @@ -63,26 +84,54 @@ export async function completion(store, port, id) { const status = await getCommandStatus(port, id) return ['done', 'failed', 'unsafe'].includes(status.status) && status }, `completion of ${id}`) - await observe(store, {event: 'completion', id, body: result}) + await observe(store, { event: 'completion', id, body: result }) return result } export async function launch(t, store, port, options = {}) { - const config = path.join(store.audit, `config-${Math.random().toString(16).slice(2)}.json`) - await fs.writeFile(config, JSON.stringify({root: store.dir, trace: store.trace, - release: store.release, fault: options.fault})) - const child = spawn(process.execPath, ['--import', fileURLToPath(new URL('./power-loss-preload.mjs', import.meta.url)), store.runner], { - cwd: store.dir, - env: {...process.env, SIMPLYSTORE_ENV: 'test', SIMPLYSTORE_FAULT_POINT: '', - SIMPLYSTORE_BASELINE_IO: config, - SIMPLYSTORE_TEST_OPTIONS: JSON.stringify({ - port, datafile: store.datafile, commandsFile: store.commandsFile, - indexFile: store.indexFile, commandLog: store.commandLog, - commandStatus: store.commandStatus, wwwroot: path.join(rootDir, 'www'), - maxWorkers: 1, commandTimeout: 10000, ...options.server - })}, - stdio: ['ignore', 'pipe', 'pipe'] - }) + const config = path.join( + store.audit, + `config-${Math.random().toString(16).slice(2)}.json` + ) + await fs.writeFile( + config, + JSON.stringify({ + root: store.dir, + trace: store.trace, + release: store.release, + fault: options.fault + }) + ) + const child = spawn( + process.execPath, + [ + '--import', + fileURLToPath(new URL('./power-loss-preload.mjs', import.meta.url)), + store.runner + ], + { + cwd: store.dir, + env: { + ...process.env, + SIMPLYSTORE_ENV: 'test', + SIMPLYSTORE_FAULT_POINT: '', + SIMPLYSTORE_BASELINE_IO: config, + SIMPLYSTORE_TEST_OPTIONS: JSON.stringify({ + port, + datafile: store.datafile, + commandsFile: store.commandsFile, + indexFile: store.indexFile, + commandLog: store.commandLog, + commandStatus: store.commandStatus, + wwwroot: path.join(rootDir, 'www'), + maxWorkers: 1, + commandTimeout: 10000, + ...options.server + }) + }, + stdio: ['ignore', 'pipe', 'pipe'] + } + ) let output = '' child.stdout.on('data', data => { output += data @@ -91,7 +140,7 @@ export async function launch(t, store, port, options = {}) { output += data }) t.after(() => stopServer(child)) - return {child, getOutput: () => output} + return { child, getOutput: () => output } } export async function crash(child) { @@ -101,11 +150,20 @@ export async function crash(child) { } export async function copyImage(t, store) { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-crash-image-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) - await fs.cp(store.dir, dir, {recursive: true}) - const image = {...store, dir} - for (const key of ['datafile', 'commandsFile', 'indexFile', 'commandLog', 'commandStatus', 'runner']) { + const dir = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-crash-image-') + ) + t.after(() => fs.rm(dir, { recursive: true, force: true })) + await fs.cp(store.dir, dir, { recursive: true }) + const image = { ...store, dir } + for (const key of [ + 'datafile', + 'commandsFile', + 'indexFile', + 'commandLog', + 'commandStatus', + 'runner' + ]) { image[key] = path.join(dir, path.basename(store[key])) } return image @@ -113,11 +171,15 @@ export async function copyImage(t, store) { export async function loseUnpublishedName(store, image, original) { const trace = await rows(store.trace) - const publication = trace.findLastIndex(e => e.file === original && - (e.op === 'rename' || (e.op === 'open' && !e.existed))) + const publication = trace.findLastIndex( + e => + e.file === original && + (e.op === 'rename' || (e.op === 'open' && !e.existed)) + ) assert.ok(publication >= 0, `trace contains publication of ${original}`) - const durable = trace.slice(publication + 1).some(e => - e.file === path.dirname(original) && e.op === 'sync') + const durable = trace + .slice(publication + 1) + .some(e => e.file === path.dirname(original) && e.op === 'sync') if (!durable) { await fs.rm(path.join(image.dir, path.basename(original))) } @@ -125,7 +187,13 @@ export async function loseUnpublishedName(store, image, original) { } export async function requirement(t, id, actual, expected, message) { - await t.test(`${id}: ${message}`, { - todo: process.env.SIMPLYSTORE_BASELINE_STRICT === '1' ? false : 'Known baseline violation; run test:power-loss:strict' - }, () => assert.deepEqual(actual, expected, message)) + let todo = 'Known baseline violation; run test:power-loss:strict' + if (process.env.SIMPLYSTORE_BASELINE_STRICT === '1') { + todo = false + } + await t.test( + `${id}: ${message}`, + { todo }, + () => assert.deepEqual(actual, expected, message) + ) } diff --git a/test/power-loss-preload.mjs b/test/power-loss-preload.mjs index 8f16071..410e862 100644 --- a/test/power-loss-preload.mjs +++ b/test/power-loss-preload.mjs @@ -3,43 +3,78 @@ import fs from 'node:fs' import fsp from 'node:fs/promises' import path from 'node:path' import process from 'node:process' -import {threadId} from 'node:worker_threads' -import {syncBuiltinESMExports} from 'node:module' +import { threadId } from 'node:worker_threads' +import { syncBuiltinESMExports } from 'node:module' const configFile = process.env.SIMPLYSTORE_BASELINE_IO if (configFile) { const config = JSON.parse(fs.readFileSync(configFile, 'utf8')) const root = path.resolve(config.root) - const raw = Object.fromEntries(['open', 'write', 'fsync', 'close', 'rename', - 'appendFileSync', 'readFileSync', 'existsSync', 'statSync', 'writeFileSync'] - .map(key => [key, fs[key].bind(fs)])) + const raw = Object.fromEntries( + [ + 'open', + 'write', + 'fsync', + 'close', + 'rename', + 'appendFileSync', + 'readFileSync', + 'existsSync', + 'statSync', + 'writeFileSync' + ].map(key => [key, fs[key].bind(fs)]) + ) const open = fsp.open.bind(fsp) const rename = fsp.rename.bind(fsp) const handles = new Map() let injected = false - const scoped = file => typeof file === 'string' && + const scoped = file => + typeof file === 'string' && (file === root || file.startsWith(root + path.sep)) function record(op, file, extra = {}) { if (scoped(file)) { - raw.appendFileSync(config.trace, - JSON.stringify({op, file, threadId, ...extra}) + '\n') + raw.appendFileSync( + config.trace, + JSON.stringify({ op, file, threadId, ...extra }) + '\n' + ) } } function snapshot(file) { - return raw.statSync(file).isDirectory() ? {} : {bytes: raw.readFileSync(file).toString('base64')} + if (raw.statSync(file).isDirectory()) { + return {} + } + const bytes = raw.readFileSync(file).toString('base64') + return { bytes } } async function fault(op, file, content = '') { const rule = config.fault - if (injected || !rule || rule.op !== op || !scoped(file) || + if ( + injected || + !rule || + rule.op !== op || + !scoped(file) || !path.basename(file).startsWith(rule.file) || - (rule.contains && !content.includes(rule.contains))) { + (rule.contains && !content.includes(rule.contains)) + ) { return false } - if (rule.after && !raw.readFileSync(config.trace,'utf8').split('\n').filter(Boolean).map(JSON.parse).some(event => event.op === rule.after.op && path.basename(event.file) === rule.after.file)) { + if ( + rule.after && + !raw + .readFileSync(config.trace, 'utf8') + .split('\n') + .filter(Boolean) + .map(JSON.parse) + .some( + event => + event.op === rule.after.op && + path.basename(event.file) === rule.after.file + ) + ) { return false } injected = true - record('injected', file, {operation: op, action: rule.action}) + record('injected', file, { operation: op, action: rule.action }) if (rule.action === 'pause') { const deadline = Date.now() + 15000 while (!raw.existsSync(config.release)) { @@ -53,18 +88,21 @@ if (configFile) { return true } else { - throw Object.assign(new Error(`Injected ${rule.code || 'EIO'} at ${op}`), {code: rule.code || 'EIO'}) + throw Object.assign( + new Error(`Injected ${rule.code || 'EIO'} at ${op}`), + { code: rule.code || 'EIO' } + ) } return false } - fsp.rename = async function(from, to) { + fsp.rename = async function (from, to) { const file = path.resolve(String(to)) await fault('rename', file) await rename(from, to) - record('rename', file, {from:path.resolve(String(from))}) + record('rename', file, { from: path.resolve(String(from)) }) } - fsp.open = async function(file, ...args) { + fsp.open = async function (file, ...args) { const absolute = path.resolve(String(file)) const existed = raw.existsSync(absolute) await fault('open', absolute) @@ -72,71 +110,95 @@ if (configFile) { if (!scoped(absolute)) { return handle } - record('open', absolute, {existed, flags: args[0]}) + record('open', absolute, { existed, flags: args[0] }) let content = '' - for (const method of ['appendFile', 'write', 'datasync', 'sync', 'close']) { + for (const method of [ + 'appendFile', + 'write', + 'datasync', + 'sync', + 'close' + ]) { const original = handle[method].bind(handle) handle[method] = async (...params) => { if (method === 'appendFile') { content = String(params[0]) } if (method === 'write') { - content = String(params[0].subarray(params[1], params[1]+params[2])) + content = String( + params[0].subarray(params[1], params[1] + params[2]) + ) } const short = await fault(method, absolute, content) if (short && method === 'write') { - params[2] = Math.max(1, Math.floor(params[2]/2)) + params[2] = Math.max(1, Math.floor(params[2] / 2)) } const result = await original(...params) - record(method, absolute, method === 'datasync' || method === 'sync' - ? snapshot(absolute) : method === 'write' ? {bytesWritten:result.bytesWritten} : {}) + let extra = {} + if (method === 'datasync' || method === 'sync') { + extra = snapshot(absolute) + } + else if (method === 'write') { + extra = { bytesWritten: result.bytesWritten } + } + record(method, absolute, extra) await fault(method + ':after', absolute, content) return result } } return handle } - fs.open = function(file, ...args) { + fs.open = function (file, ...args) { const callback = args.pop() const absolute = path.resolve(String(file)) const existed = raw.existsSync(absolute) return raw.open(file, ...args, (error, fd) => { if (!error && scoped(absolute)) { handles.set(fd, absolute) - record('open', absolute, {existed, flags: args[0]}) + record('open', absolute, { existed, flags: args[0] }) } callback(error, fd) }) } - fs.write = function(fd, ...args) { + fs.write = function (fd, ...args) { const callback = args.pop() const file = handles.get(fd) - void fault('write', file).then(short => { - if (short) { - // The fixture targets write-file-atomic's Uint8Array overload. - if (!ArrayBuffer.isView(args[0]) || typeof args[2] !== 'number') { - throw new Error('Unsupported short-write fixture') - } - args[2] = Math.max(1, Math.floor(args[2] / 2)) - } - raw.write(fd, ...args, (error, bytes, buffer) => { - if (!error) { - record('write', file, {bytesWritten: bytes}) + void fault('write', file) + .then(short => { + if (short) { + // The fixture targets write-file-atomic's Uint8Array + // overload. + if ( + !ArrayBuffer.isView(args[0]) || + typeof args[2] !== 'number' + ) { + throw new Error('Unsupported short-write fixture') + } + args[2] = Math.max(1, Math.floor(args[2] / 2)) } - callback(error, bytes, buffer) + raw.write(fd, ...args, (error, bytes, buffer) => { + if (!error) { + record('write', file, { bytesWritten: bytes }) + } + callback(error, bytes, buffer) + }) }) - }).catch(callback) + .catch(callback) } - fs.fsync = function(fd, callback) { + fs.fsync = function (fd, callback) { const file = handles.get(fd) - void fault('fsync', file).then(() => raw.fsync(fd, error => { - if (!error && scoped(file)) { - record('sync', file, snapshot(file)) - } - callback(error) - })).catch(callback) + void fault('fsync', file) + .then(() => + raw.fsync(fd, error => { + if (!error && scoped(file)) { + record('sync', file, snapshot(file)) + } + callback(error) + }) + ) + .catch(callback) } - fs.close = function(fd, callback) { + fs.close = function (fd, callback) { const file = handles.get(fd) return raw.close(fd, error => { if (!error) { @@ -146,16 +208,22 @@ if (configFile) { callback(error) }) } - fs.rename = function(from, to, callback) { + fs.rename = function (from, to, callback) { const file = path.resolve(String(to)) - void fault('rename', file).then(() => raw.rename(from, to, error => { - if (!error) { - record('rename', file, {from: path.resolve(String(from))}) - } - callback(error) - })).catch(callback) + void fault('rename', file) + .then(() => + raw.rename(from, to, error => { + if (!error) { + record('rename', file, { + from: path.resolve(String(from)) + }) + } + callback(error) + }) + ) + .catch(callback) } - fs.writeFileSync = function(file, ...args) { + fs.writeFileSync = function (file, ...args) { const result = raw.writeFileSync(file, ...args) record('writeFileSync', path.resolve(String(file))) return result diff --git a/test/storage.mjs b/test/storage.mjs index 47e2bbc..8442784 100644 --- a/test/storage.mjs +++ b/test/storage.mjs @@ -1,44 +1,82 @@ -import {Buffer} from 'node:buffer' +import { Buffer } from 'node:buffer' import test from 'node:test' import assert from 'node:assert/strict' import fs from 'node:fs/promises' import os from 'node:os' import path from 'node:path' -import {writeAll,publishFile,publishFileSync,appendRecord,serialWriter} from '../src/storage.mjs' +import { + writeAll, + publishFile, + publishFileSync, + appendRecord, + serialWriter +} from '../src/storage.mjs' -test('complete-write loop preserves bytes through repeated short writes and rejects zero progress', async()=>{ - const bytes = Buffer.from('cafΓ© ζΌ’ε­— πŸ˜€'), written=[] - await writeAll({async write(buffer,offset,length){ - const n=Math.min(3,length);written.push(buffer.subarray(offset,offset+n));return {bytesWritten:n} - }},bytes) - assert.deepEqual(Buffer.concat(written),bytes) - for (const count of [0,-1,NaN,1000]) { - await assert.rejects(writeAll({async write(){ - return {bytesWritten:count} - }},bytes), /progress write/) +test('complete-write loop preserves bytes through repeated short writes and rejects zero progress', async () => { + const bytes = Buffer.from('cafΓ© ζΌ’ε­— πŸ˜€'), + written = [] + await writeAll( + { + async write(buffer, offset, length) { + const n = Math.min(3, length) + written.push(buffer.subarray(offset, offset + n)) + return { bytesWritten: n } + } + }, + bytes + ) + assert.deepEqual(Buffer.concat(written), bytes) + for (const count of [0, -1, NaN, 1000]) { + await assert.rejects( + writeAll( + { + async write() { + return { bytesWritten: count } + } + }, + bytes + ), + /progress write/ + ) } }) -test('serialized writer preserves order and stops after an uncertain operation',async()=>{ - const run=serialWriter(), seen=[] +test('serialized writer preserves order and stops after an uncertain operation', async () => { + const run = serialWriter(), + seen = [] let release - const gate=new Promise(resolve=>{ - release=resolve + const gate = new Promise(resolve => { + release = resolve }) - const first=run(async()=>{ - seen.push('A');await gate;throw new Error('uncertain') + const first = run(async () => { + seen.push('A') + await gate + throw new Error('uncertain') }) - const second=run(async()=>seen.push('B')) - release();await assert.rejects(first,/uncertain/);await assert.rejects(second,/uncertain/) - assert.deepEqual(seen,['A']) + const second = run(async () => seen.push('B')) + release() + await assert.rejects(first, /uncertain/) + await assert.rejects(second, /uncertain/) + assert.deepEqual(seen, ['A']) }) -test('durable publication replaces entire contents and concurrent append retains each record once',async t=>{ - const dir=await fs.mkdtemp(path.join(os.tmpdir(),'simplystore-storage-'));t.after(()=>fs.rm(dir,{recursive:true,force:true})) - const file=path.join(dir,'data') - publishFileSync(file,'old') - await publishFile(file,'new cafΓ©');assert.equal(await fs.readFile(file,'utf8'),'new cafΓ©') - await publishFile(file,'');assert.equal((await fs.stat(file)).size,0) - await Promise.all(Array.from({length:40},(_,i)=>appendRecord(file,String(i)))) - assert.equal(await fs.readFile(file,'utf8'),Array.from({length:40},(_,i)=>i+'\n').join('')) - await assert.rejects(publishFile(path.join(dir,'missing','x'),'x'),error=>error.code==='ENOENT' && error.storageFailure) - assert.deepEqual((await fs.readdir(dir)).sort(),['data']) +test('durable publication replaces entire contents and concurrent append retains each record once', async t => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-storage-')) + t.after(() => fs.rm(dir, { recursive: true, force: true })) + const file = path.join(dir, 'data') + publishFileSync(file, 'old') + await publishFile(file, 'new cafΓ©') + assert.equal(await fs.readFile(file, 'utf8'), 'new cafΓ©') + await publishFile(file, '') + assert.equal((await fs.stat(file)).size, 0) + await Promise.all( + Array.from({ length: 40 }, (_, i) => appendRecord(file, String(i))) + ) + assert.equal( + await fs.readFile(file, 'utf8'), + Array.from({ length: 40 }, (_, i) => i + '\n').join('') + ) + await assert.rejects( + publishFile(path.join(dir, 'missing', 'x'), 'x'), + error => error.code === 'ENOENT' && error.storageFailure + ) + assert.deepEqual((await fs.readdir(dir)).sort(), ['data']) }) From 8cd6e7e058decd840305277326787b4cb1b9d54e Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:23:06 +0200 Subject: [PATCH 33/37] Record conditional and width refactor verification --- .spiral/cycles/CYC-20260919-TTZ7C-17.md | 5 + .../EVD-20260919-TTZ7C-27-results.json | 49 ++++++++ .../evidence/EVD-20260919-TTZ7C-27-tests.txt | 113 ++++++++++++++++++ .spiral/evidence/EVD-20260919-TTZ7C-27.md | 28 +++++ .spiral/evidence/EVD-20260919-TTZ7C-27.ttl | 5 +- .spiral/project-context.md | 2 +- 6 files changed, 200 insertions(+), 2 deletions(-) diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md index 406d799..f20cf9c 100644 --- a/.spiral/cycles/CYC-20260919-TTZ7C-17.md +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -260,3 +260,8 @@ with named queries and comparison steps. Runtime behavior and plain-data public reports are preserved. The full relevant suite now passes 105/105, with the latest verification recorded in EVD-27. These refinements do not accept or integrate the cycle and do not replace the exact earlier filesystem evidence. + +The subsequent conditional/width revision expands dense ternaries, retains ten +short readable choices, and wraps the cycle code toward 80 columns with literal +exceptions. The 105-test suite and direct recovery/backup CLI checks pass; EVD-27 +records formatting and behavior-preservation checks. Acceptance is still pending. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json index dff000e..25717ee 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json @@ -245,5 +245,54 @@ ], "changedFileLint": "passed", "newVmRun": false + }, + "conditionalWidthRefactor": { + "implementation": "5d166a23b664ba9fbe34ae819feb1ed70959046c", + "passed": 105, + "failed": 0, + "skipped": 0, + "todo": 0, + "style": { + "files": 26, + "originalTernaries": 36, + "currentTernaries": 10, + "styleErrors": 0, + "styleWarnings": 0, + "fullLint": [ + { + "file": "src/index.id.mjs", + "messages": [ + { + "line": 49, + "rule": "no-undef", + "message": "'meta' is not defined." + } + ] + }, + { + "file": "src/index.offset.mjs", + "messages": [ + { + "line": 74, + "rule": "no-undef", + "message": "'meta' is not defined." + } + ] + } + ] + }, + "formatterSyntaxTreesMatched": 27, + "generatedVmModulesMatched": 4, + "storePathConfigurationsMatched": 16, + "cliSmoke": [ + "convert", + "inspect", + "apply A", + "backup", + "restore with source", + "restore without source", + "verify" + ], + "newVmRun": false } } diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt index e0622cd..aa438b0 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt @@ -249,3 +249,116 @@ JAQT refactor at 49fa48c123d0035df087b7a9d4c1460da92b85fa β„Ή skipped 0 β„Ή todo 0 β„Ή duration_ms 6076.58365 + +Conditional/width refactor at 5d166a23b664ba9fbe34ae819feb1ed70959046c +βœ” atomicity: command mutation followed by failure does not become visible or committed (770.328799ms) +βœ” isolation: mutating data inside a JavaScript query does not affect later queries (351.976338ms) +βœ” consistency: query during an active command sees the previous committed state (1059.750516ms) +βœ” idempotency: duplicate command ID is acknowledged but not applied twice (517.018137ms) +βœ” idempotency: retry during active command reports current status and does not enqueue twice (1019.480825ms) +βœ” administrator recovers trailing missing/waiting commands in log order on a copy (310.417052ms) +βœ” later accepted dataset blocks rerun, including zero-byte no-op (10.846552ms) +βœ” later done dataset blocks rerun, including zero-byte no-op (21.031532ms) +βœ” later unsafe dataset blocks rerun, including zero-byte no-op (10.38607ms) +βœ” new later file makes an approved plan stale before executing a handler (27.137195ms) +βœ” changed code and missing explicit approval reject apply (29.102662ms) +βœ” cooperative ownership excludes competing writer and path aliases (11.733063ms) +βœ” backup and restore validate saved bytes without executing commands (387.986745ms) +βœ” read-only inspection never imports handlers and rejects conflicting done order (23.627113ms) +βœ” historical status order never makes a later log dataset eligible for bypass (6.925995ms) +βœ” interrupted recovery preserves completed attempts and blocks an unfinished candidate (249.472331ms) +βœ” retained digest mismatch stops recovery before replacing integrity or appending done (94.715019ms) +βœ” malformed tails and overlapping paths cannot be accepted as valid stores (6.334512ms) +βœ” approved recovery resumes from completed candidate prefix without repeating external effects (314.392961ms) +βœ” finishing an interrupted completion report reconstructs data without executing again (132.623324ms) +βœ” multi-directory integrity paths and file permissions survive backup and restore (33.686382ms) +βœ” backup excludes unrelated secrets while retaining declared custom artifacts (15.969331ms) +βœ” a backup remains restorable after transfer to a different directory (69.563119ms) +βœ” promotion verification rejects code changed after completed recovery (79.133752ms) +βœ” verified A prefix permits only missing B, waiting C and missing D in log order (434.822874ms) +βœ” restore reports accepted commands absent from an older backup (18.754409ms) +βœ” identical historical duplicate retains first log position; conflicting duplicate blocks (4.386649ms) +βœ” backup coverage distinguishes changed history and data from missing source data (229.215826ms) +βœ” handler lifecycle: index update failure prevents command commit (666.692445ms) +βœ” handler lifecycle: derived index file can remain after failed command (375.946138ms) +βœ” handler lifecycle: final offset write failure prevents command commit (329.270472ms) +βœ” handler lifecycle: custom finalizer rejection prevents command commit (305.101047ms) +βœ” handler lifecycle: current index update can mutate canonical state before commit (332.934148ms) +βœ” command edit indexes only its changed record using changeset UTF-8 offsets (20.077215ms) +βœ” command addition indexes new entities and preserves interior skip numbers (10.47907ms) +βœ” command offsets include mutations and new records introduced by the index hook (6.912693ms) +βœ” successive command indexes are relative to their own changeset files (7.377323ms) +βœ” no-op commands replace any stale sidecar with an empty offset index (6.156606ms) +βœ” failure to write final offsets rejects the command before success (6.726363ms) +βœ” commands await custom finalization that delegates to the default index (45.640098ms) +βœ” custom finalization can replace the default without an unconditional offset write (7.550846ms) +βœ” custom finalization rejection propagates from the command worker (7.120895ms) +βœ” default conversion offsets match every final record (130.821568ms) +βœ” conversion finalizes offsets after custom indexing changes record sizes and adds records (107.953088ms) +βœ” conversion replaces stale offsets even when the custom hook writes no index (182.645488ms) +βœ” conversion awaits custom finalization with the original receiver and final bytes (118.508781ms) +βœ” conversion propagates rejection from custom finalization (153.766708ms) +βœ” conversion initializes existing-format logs and optional base integrity without overwriting a store (164.775744ms) +βœ” runtime environment defaults to production and ignores production fault points (1.444992ms) +βœ” fault points are inert outside test environment (0.190374ms) +βœ” crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (716.598156ms) +βœ” crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (425.24224ms) +βœ” crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (374.751707ms) +βœ” crash at before-command-changeset-write preserves evidence and never automatically reruns (423.374956ms) +βœ” crash at after-command-changeset-write preserves evidence and never automatically reruns (368.481424ms) +βœ” crash at before-command-done-status preserves evidence and never automatically reruns (369.830528ms) +βœ” crash after done status but before query update recovers committed state without replay (472.231201ms) +βœ” crash ownership cannot be stolen automatically (309.193466ms) +βœ” hanging command times out unsafe and later accepted command commits (349.149476ms) +βœ” hanging load worker fails startup explicitly (195.947203ms) +βœ” integrity-enabled command writes changeset digest before done status (320.734733ms) +βœ” normal restart preserves committed state according to reconstruction oracle (432.46088ms) +βœ” accepted command changeset is not treated as committed state on startup (78.677963ms) +βœ” done command with missing changeset refuses recovery instead of silently using base data (64.739072ms) +βœ” malformed base OD-JSONTag file refuses recovery (114.82201ms) +βœ” malformed committed changeset refuses recovery (195.38594ms) +βœ” truncated committed changeset payload refuses recovery before lazy parsing can hide it (70.943855ms) +βœ” malformed uncommitted changeset is ignored during committed startup reconstruction (111.229727ms) +βœ” durable status file selects only done command changesets for startup (84.72398ms) +βœ” durable command log replays only accepted commands (4.639563ms) +βœ” active command at attempt 1 remains unchanged pending administrator assessment (7.631507ms) +βœ” active command at attempt 2 remains unchanged pending administrator assessment (2.233418ms) +βœ” malformed durable status record refuses recovery with explicit integrity error (2.672729ms) +βœ” structurally invalid durable status record refuses recovery (2.157557ms) +βœ” malformed durable command log record refuses recovery with explicit integrity error (1.246495ms) +βœ” structurally invalid durable command log record refuses recovery (1.942975ms) +βœ” OD-JSONTag framing validation catches truncated lazy records (0.331445ms) +βœ” OD-JSONTag framing validation accepts changeset skip records (0.187469ms) +βœ” integrity manifest detects same-length altered base OD-JSONTag payload (61.989542ms) +βœ” integrity manifest detects same-length altered committed changeset payload (55.858695ms) +βœ” integrity enabled requires manifest entry for base data (73.109183ms) +βœ” PL01-05 durable publication survives modeled loss of unsynced names and restart (901.363066ms) +βœ” PL10 an actual short changeset write is completed before acknowledgment (377.270416ms) +βœ” PL07-09/11/18-20 log sync failure stops mutation without false terminal success (249.279553ms) +βœ” PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (243.886641ms) +βœ” PL07-09/11/18-20 log close failure stops mutation without false terminal success (204.041125ms) +βœ” PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (252.47621ms) +βœ” PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (253.389402ms) +βœ” PL07-09/11/18-20 done sync failure stops mutation without false terminal success (233.990209ms) +βœ” PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (247.363839ms) +βœ” PL16 delayed append cannot be overtaken (330.834357ms) +βœ” PL16 delayed append cannot be overtaken by duplicate ID (264.849904ms) +βœ” PL02-04/06 missing command-log.jsontag fails open rather than recreating history (286.264648ms) +βœ” PL02-04/06 missing command-status.jsontag fails open rather than recreating history (315.299591ms) +βœ” PL02-04/06 missing data.A.jsontag fails open rather than recreating history (315.321485ms) +βœ” PL12/17 startup preserves uncertain active work without repeating external effects (308.849112ms) +βœ” directory sync failure after rename prevents done (192.103111ms) +βœ” directory close:after failure after rename prevents done (189.438443ms) +βœ” changeset open error is not masked or acknowledged (191.130994ms) +βœ” changeset write error is not masked or acknowledged (210.408614ms) +βœ” changeset close:after error is not masked or acknowledged (214.214365ms) +βœ” short command-log append preserves its entire JSONTag record (204.016215ms) +βœ” complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.265931ms) +βœ” serialized writer preserves order and stops after an uncertain operation (0.736876ms) +βœ” durable publication replaces entire contents and concurrent append retains each record once (41.178696ms) +β„Ή tests 105 +β„Ή pass 105 +β„Ή fail 0 +β„Ή skipped 0 +β„Ή todo 0 +β„Ή duration_ms 6300.329225 diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md index 8b3bdee..298c8f3 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.md +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -163,3 +163,31 @@ ESLint passes all three changed production modules and the changed test file; Git whitespace checks pass. No persistence protocol changed, and no new VM power-cut run is claimed. The recorded filesystem experiment still refers to its exact original runtime revision. Cycle acceptance remains pending. + +## Conditional-expression and line-width verification + +Verified IMP-26 at `5d166a23b664ba9fbe34ae819feb1ed70959046c` after the maintainer +requested fewer ternaries and lines about 80 characters wide, with readability +exceptions. The cycle's 26 JavaScript modules now contain ten clear, short +conditional expressions, down from 36. Nested choices, optional fields and +parsing/conversion steps use explicit branches. Formatting also covers ESLint +configuration. The line-width rule is a warning with string/template/regex/URL +exceptions; it is not a hard prohibition that overrides readability. + +The formatting pass compared all 27 syntax trees before and after wrapping; +only source positions and literal raw spelling were ignored, and all matched. +Conditional rewrites were verified separately: the complete relevant suite passed +**105/105**, with no failures, skips or TODOs. A direct CLI smoke exercise passed +conversion, inspection, explicitly approved A recovery, backup, restore both with +and without source comparison, and promotion verification. Recovered command +coverage was A. Sixteen optional integrity/schema configurations produce identical +store paths/options to the previous implementation. All four VM scenario handler +modules generated by the rewritten branch selection are byte-identical to the +previous nested ternary's output. No new VM power-cut experiment is claimed. + +All 26 modules plus ESLint configuration pass brace and 80-column policy checks +with zero errors or warnings under the declared literal exceptions. Full lint +finds only the same two historical undefined-meta errors in index load methods, +now at their reformatted line positions. No dependency or lockfile changed; the +pinned formatter was installed only in /tmp. Existing newline block preferences, +canonical file formats and cycle acceptance status are preserved. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl index 1895dca..13861b3 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl @@ -11,4 +11,7 @@ project:EVD-20260919-TTZ7C-27 a sd:VerificationEvidence ; sd:gitCommit "13db3bf55926cf3a3a1b2769c422881a875193c8" ] ; sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; - sd:gitCommit "49fa48c123d0035df087b7a9d4c1460da92b85fa" ] . + sd:gitCommit "49fa48c123d0035df087b7a9d4c1460da92b85fa" ] ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "5d166a23b664ba9fbe34ae819feb1ed70959046c" ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 0ec4dde..a883826 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -24,7 +24,7 @@ Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offs Active-cycle progress on 2026-09-19: `IMP-20260919-TTZ7C-26` implements the corrected `DES-20260919-TTZ7C-24/25` following explicit authorization to complete the whole cycle. Existing command/status/integrity and dataset formats remain unchanged, with no migration. Durable complete-write and file/directory barriers, ownership, log-authoritative serial acceptance/execution, command-only workers, strict startup assessment, and source-bound administrator recovery/backup/restore are implemented. `EVD-20260919-TTZ7C-27` records 101 passing regression tests plus 22 expanded administrator tests (three additional cases), direct CLI checks, and a passing four-scenario ext4/QEMU power-cut exercise with device/filesystem negative controls and latency. `DURABILITY.md` and `docs/recovery.md` define the bounded behavior and runnable procedure. Historical baseline gaps remain at `EVD-20260919-TTZ7C-23`; PL15/16 expectations were corrected by the maintainer, not silently weakened. Full rebuild remains a documented separate contract. The cycle is ready for human acceptance but not Accepted or integrated. Broader filesystem/hardware and randomized testing, undeclared custom dependencies and historical input/effect uncertainty remain explicit limits. Two unchanged legacy index `load` methods have pre-existing undefined-meta lint errors outside this write-path cycle. -Acceptance-review refinements: explicit multiline blocks and selective JAQT queries now improve the cycle code’s readability. The latest full relevant regression suite passes 105/105; EVD-27 records preserved ordering, backup coverage and plain-data report contracts. Human cycle acceptance is still pending. +Acceptance-review refinements: explicit multiline blocks, selective JAQT queries, fewer ternaries, and lines near 80 columns now improve the cycle code’s readability. Short conditional choices and long literals remain where clearer. The latest full relevant regression suite passes 105/105; EVD-27 records preserved ordering, backup coverage and plain-data report contracts. Human cycle acceptance is still pending. ## Intake State From 8d6590fc4b616e0fb276ec5ca213e79b240e4648 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:41:11 +0200 Subject: [PATCH 34/37] Record example initialization and converter correction request --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index bfce2fc..0154ee7 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -145,3 +145,16 @@ obvious conditional choices when they are clearer. Wrap the cycle's JavaScript toward 80 columns, preserving exceptions for readability and literal contents. These are review guidelines with explicit exceptions, not blanket prohibitions. This authorizes a behavior-preserving readability revision, not cycle acceptance. + +## Example initialization correction β€” 2026-09-19 + +The maintainer reported that the example fails startup without its base and +command/status logs, and requested: + +> simplystore now provides a convert and build script. I think the example should use those, preferably by adding them in the package.json of the example. Then the convert script should work. An error in that is something we need to fix. + +This authorizes repairing the converter and exposing the example initialization +workflow through npm scripts. The checked-out converter already initializes empty +logs, but conversion of the actual Star Wars dataset fails on URL validation. +No separate SimplyStore build script exists in this checkout; its intended +meaning is being clarified. Existing store data must not be overwritten. From 9b4a340facd57f22edbead07a195700b0237c9ad Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:44:42 +0200 Subject: [PATCH 35/37] Clarify example build script scope --- .spiral/sources/SRC-20260919-TTZ7C-18.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md index 0154ee7..64883f1 100644 --- a/.spiral/sources/SRC-20260919-TTZ7C-18.md +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -158,3 +158,10 @@ workflow through npm scripts. The checked-out converter already initializes empt logs, but conversion of the actual Star Wars dataset fails on URL validation. No separate SimplyStore build script exists in this checkout; its intended meaning is being clarified. Existing store data must not be overwritten. + +The maintainer clarified: + +> curriculum-store has an import script and a build. The example only needs a build script + +Expose `npm run build` in the example, invoking SimplyStore's existing converter. +No separate import script or changed-code rebuild feature is requested. From d4136eaf7b0014726a1c90a0d0b8488ece2d6b27 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:45:20 +0200 Subject: [PATCH 36/37] Fix URL conversion and add example npm build --- .../implementations/IMP-20260919-TTZ7C-26.md | 18 ++++++++++++++++++ .../implementations/IMP-20260919-TTZ7C-26.ttl | 9 ++++++--- example/README.md | 8 ++++++-- example/package.json | 1 + scripts/convert.mjs | 8 ++++++-- test/conversion-offsets.mjs | 18 +++++++++++++++++- 6 files changed, 54 insertions(+), 8 deletions(-) diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md index d782905..025b556 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.md +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -101,3 +101,21 @@ existing brace rules; no package dependency or lockfile changed. The formatting step was checked with syntax-tree comparison. Conditional rewrites additionally receive the relevant regression and CLI verification recorded in EVD-27. Storage barriers, canonical formats and recovery policy are unchanged. + +## Example initialization correction + +The example now exposes `npm run build`, invoking the existing converter to +create the base, indexes and empty logs before startup. Its README documents +that one-time initialization and overwrite refusal, and uses the HTTP URL served +by the example. No import script or full command rebuild feature is added. + +The converter passed an empty base URL to the mutable OD-JSONTag parser. When +indexing forces lazy parsing of tagged URLs in the Star Wars dataset, URL +validation fails even for absolute URLs. Use the parser's default valid URL base +while preserving mutable indexing. Report caught conversion errors with a nonzero +exit status so thrown parser diagnostics remain visible. Failed conversions still +retain ownership for assessment; successful initialization releases it. + +Regression coverage converts the actual bundled dataset, verifies all final +record offsets and reads typed URL values, including a linked homeworld. No +canonical format, storage barrier or existing-store overwrite policy changes. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl index 4c469fa..718ba4f 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -17,6 +17,9 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ; sd:implementationLocation "src/index.id.mjs" ; sd:implementationLocation "src/recovery.mjs" ; sd:implementationLocation "scripts/convert.mjs" ; + sd:implementationLocation "example/package.json" ; + sd:implementationLocation "example/README.md" ; + sd:implementationLocation "test/conversion-offsets.mjs" ; sd:implementationLocation "scripts/recover.mjs" ; sd:implementationLocation "scripts/power-loss-vm/exercise.py" ; sd:implementationLocation "test/power-loss-baseline.mjs" ; @@ -30,6 +33,6 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; - sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "f14d13e2bf4b0f6ab3afeea7e9900ef60488590e" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "f14d13e2bf4b0f6ab3afeea7e9900ef60488590e" ] ; - sd:implementationChangeKind sd:BehaviorPreservingChange . + sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "9b4a340facd57f22edbead07a195700b0237c9ad" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "9b4a340facd57f22edbead07a195700b0237c9ad" ] ; + sd:implementationChangeKind sd:SemanticChange . diff --git a/example/README.md b/example/README.md index a258fb8..8a23b67 100644 --- a/example/README.md +++ b/example/README.md @@ -9,15 +9,19 @@ npm install Then convert the dataset `swdb.jsontag` to internal SimplyStore format: ```shell -node ../scripts/convert.mjs swdb.jsontag data.jsontag +npm run build ``` +This creates `data.jsontag`, its indexes, and empty command and status logs +for the first startup. Build requires a new store and refuses to overwrite +existing data or logs. Run it once before starting the example. + Then start the server: ```shell npm start ``` -Now go to `https://localhost:3000/query/` and you can enter queries, e.g.: +Now go to `http://localhost:3000/query/` and you can enter queries, e.g.: ```javascript from(data.people) diff --git a/example/package.json b/example/package.json index 48f013f..34c150e 100644 --- a/example/package.json +++ b/example/package.json @@ -2,6 +2,7 @@ "name": "Example SimplyStore", "main": "server.mjs", "scripts": { + "build": "node ../scripts/convert.mjs swdb.jsontag data.jsontag", "start": "node server.mjs" }, "dependencies": { diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 980c61d..cf0f0d9 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -81,7 +81,8 @@ async function main() { // indexes need the position data which is only available after // parsing the od-jsontag data - const parser = new Parser('', false) // allow mutations + // Keep the default URL base and allow index mutations. + const parser = new Parser(undefined, false) const odData = parser.parse(strData) let meta = { @@ -124,4 +125,7 @@ async function main() { console.log('Converted data written to ', outputFile) } -main() +main().catch(error => { + console.error('Conversion failed:', error) + process.exitCode = 1 +}) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs index 8878641..1b25ff9 100644 --- a/test/conversion-offsets.mjs +++ b/test/conversion-offsets.mjs @@ -71,9 +71,25 @@ async function verifyOffsets({ dir, output }) { payloads.length, 'no missing or extra offsets' ) - return { payloads, data: new Parser('').parse(bytes.toString('utf8')) } + return { payloads, data: new Parser().parse(bytes.toString('utf8')) } } +test('conversion supports the example dataset and its URL values', async t => { + const files = await fixture(t) + await fs.copyFile( + new URL('../example/swdb.jsontag', import.meta.url), + files.input + ) + await convert(files) + const { data } = await verifyOffsets(files) + assert.equal(String(data.people[0].name), 'Luke Skywalker') + assert.equal( + String(data.people[0].homeworld.url), + 'http://swapi.co/api/planets/1/' + ) + assert.equal(String(data.planets[0].url), 'http://swapi.co/api/planets/2/') +}) + test('default conversion offsets match every final record', async t => { const files = await fixture(t) await convert(files) From 8d2b7005c13ee9001cee47d02fb58a424b6fc4a7 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Sat, 19 Sep 2026 17:46:30 +0200 Subject: [PATCH 37/37] Record example build and startup verification --- .../EVD-20260919-TTZ7C-27-results.json | 15 ++ .../evidence/EVD-20260919-TTZ7C-27-tests.txt | 175 ++++++++++++++++++ .spiral/evidence/EVD-20260919-TTZ7C-27.md | 24 +++ .spiral/evidence/EVD-20260919-TTZ7C-27.ttl | 5 +- .spiral/project-context.md | 2 +- 5 files changed, 219 insertions(+), 2 deletions(-) diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json index 25717ee..6f47e26 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json @@ -294,5 +294,20 @@ "verify" ], "newVmRun": false + }, + "exampleBuildCorrection": { + "implementationCommit": "d4136eaf7b0014726a1c90a0d0b8488ece2d6b27", + "tests": 106, + "passed": 106, + "failed": 0, + "skipped": 0, + "converterTests": 7, + "lintErrors": 0, + "lintWarnings": 0, + "exampleNpmBuildStartQuery": "passed in disposable copy with available port", + "overwriteRefused": true, + "freshDependencyInstall": false, + "existingExampleLock": "preserved", + "newVmRun": false } } diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt index aa438b0..1a0f8fa 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt @@ -362,3 +362,178 @@ Conditional/width refactor at 5d166a23b664ba9fbe34ae819feb1ed70959046c β„Ή skipped 0 β„Ή todo 0 β„Ή duration_ms 6300.329225 + +Example conversion correction: d4136eaf7b0014726a1c90a0d0b8488ece2d6b27 +βœ” atomicity: command mutation followed by failure does not become visible or committed (530.649449ms) +βœ” isolation: mutating data inside a JavaScript query does not affect later queries (421.252932ms) +βœ” consistency: query during an active command sees the previous committed state (1070.76709ms) +βœ” idempotency: duplicate command ID is acknowledged but not applied twice (534.054773ms) +βœ” idempotency: retry during active command reports current status and does not enqueue twice (1017.610332ms) +task time 4 +task time 4 +βœ” administrator recovers trailing missing/waiting commands in log order on a copy (234.338869ms) +βœ” later accepted dataset blocks rerun, including zero-byte no-op (10.913853ms) +βœ” later done dataset blocks rerun, including zero-byte no-op (30.629165ms) +βœ” later unsafe dataset blocks rerun, including zero-byte no-op (15.335214ms) +βœ” new later file makes an approved plan stale before executing a handler (24.505838ms) +βœ” changed code and missing explicit approval reject apply (29.925745ms) +βœ” cooperative ownership excludes competing writer and path aliases (6.074581ms) +task time 4 +βœ” backup and restore validate saved bytes without executing commands (212.675254ms) +βœ” read-only inspection never imports handlers and rejects conflicting done order (8.706261ms) +βœ” historical status order never makes a later log dataset eligible for bypass (5.538214ms) +task time 4 +task error Error: external effect already happened + at Object.addPerson (file:///tmp/simplystore-crash-OXCuMb/commands.mjs:1:156) + at runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:116:38) + at MessagePort. (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker.mjs:8:24) +βœ” interrupted recovery preserves completed attempts and blocks an unfinished candidate (201.539075ms) +task time 8 +βœ” retained digest mismatch stops recovery before replacing integrity or appending done (176.973657ms) +βœ” malformed tails and overlapping paths cannot be accepted as valid stores (9.07078ms) +task time 6 +task error Error: connection lost after effect + at Object.addPerson (file:///tmp/simplystore-crash-PgoTAx/commands.mjs:5:33) + at runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:116:38) + at MessagePort. (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker.mjs:8:24) +task time 8 +βœ” approved recovery resumes from completed candidate prefix without repeating external effects (310.420671ms) +task time 5 +βœ” finishing an interrupted completion report reconstructs data without executing again (127.255539ms) +βœ” multi-directory integrity paths and file permissions survive backup and restore (54.723634ms) +βœ” backup excludes unrelated secrets while retaining declared custom artifacts (33.453306ms) +βœ” a backup remains restorable after transfer to a different directory (21.985454ms) +task time 4 +βœ” promotion verification rejects code changed after completed recovery (94.632834ms) +task time 8 +task time 3 +task time 3 +task time 3 +task time 3 +task time 5 +task time 3 +βœ” verified A prefix permits only missing B, waiting C and missing D in log order (452.363458ms) +βœ” restore reports accepted commands absent from an older backup (19.53747ms) +βœ” identical historical duplicate retains first log position; conflicting duplicate blocks (4.600562ms) +task time 4 +βœ” backup coverage distinguishes changed history and data from missing source data (195.325248ms) +βœ” handler lifecycle: index update failure prevents command commit (495.977502ms) +βœ” handler lifecycle: derived index file can remain after failed command (421.432538ms) +βœ” handler lifecycle: final offset write failure prevents command commit (308.179751ms) +βœ” handler lifecycle: custom finalizer rejection prevents command commit (302.869268ms) +βœ” handler lifecycle: current index update can mutate canonical state before commit (330.828754ms) +task time 4 +βœ” command edit indexes only its changed record using changeset UTF-8 offsets (19.576993ms) +task time 5 +βœ” command addition indexes new entities and preserves interior skip numbers (8.863262ms) +task time 2 +βœ” command offsets include mutations and new records introduced by the index hook (6.888152ms) +task time 1 +task time 2 +βœ” successive command indexes are relative to their own changeset files (8.108558ms) +task time 1 +βœ” no-op commands replace any stale sidecar with an empty offset index (5.530991ms) +task error Error: EISDIR: illegal operation on a directory, rename '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json.5bd60a71-efd9-4f0a-a14e-a6ecabd87119.tmp' -> '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json' + at async Object.rename (node:internal/fs/promises:787:10) + at async publishFile (file:///home/auke/git/slonl/spiral/simplystore/src/storage.mjs:137:9) + at async Object.writeSerialized (file:///home/auke/git/slonl/spiral/simplystore/src/index.offset.mjs:36:3) + at async runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:143:13) + at async waitForActual (node:assert:644:5) + at async strict.rejects (node:assert:767:25) + at async TestContext. (file:///home/auke/git/slonl/spiral/simplystore/test/command-offsets.mjs:131:2) + at async Test.run (node:internal/test_runner/test:1113:7) + at async Test.processPendingSubtests (node:internal/test_runner/test:788:7) { + errno: -21, + code: 'EISDIR', + syscall: 'rename', + path: '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json.5bd60a71-efd9-4f0a-a14e-a6ecabd87119.tmp', + dest: '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json', + storageFailure: true +} +βœ” failure to write final offsets rejects the command before success (5.732417ms) +task time 24 +βœ” commands await custom finalization that delegates to the default index (28.569013ms) +task time 4 +βœ” custom finalization can replace the default without an unconditional offset write (14.88495ms) +task error Error: custom finalization failed + at Object.finalize (file:///tmp/simplystore-command-offset-KrXVTr/index.mjs:1:55) + at finalizeIndex (file:///home/auke/git/slonl/spiral/simplystore/src/index.mjs:26:18) + at runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:143:19) + at async waitForActual (node:assert:644:5) + at async strict.rejects (node:assert:767:25) + at async TestContext. (file:///home/auke/git/slonl/spiral/simplystore/test/command-offsets.mjs:169:2) + at async Test.run (node:internal/test_runner/test:1113:7) + at async Test.processPendingSubtests (node:internal/test_runner/test:788:7) +βœ” custom finalization rejection propagates from the command worker (15.605834ms) +βœ” conversion supports the example dataset and its URL values (319.615459ms) +βœ” default conversion offsets match every final record (99.858631ms) +βœ” conversion finalizes offsets after custom indexing changes record sizes and adds records (82.948927ms) +βœ” conversion replaces stale offsets even when the custom hook writes no index (99.469857ms) +βœ” conversion awaits custom finalization with the original receiver and final bytes (140.26764ms) +βœ” conversion propagates rejection from custom finalization (90.291458ms) +βœ” conversion initializes existing-format logs and optional base integrity without overwriting a store (140.787504ms) +βœ” runtime environment defaults to production and ignores production fault points (1.70791ms) +βœ” fault points are inert outside test environment (0.239437ms) +βœ” crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (543.472158ms) +βœ” crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (466.784076ms) +βœ” crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (397.562307ms) +βœ” crash at before-command-changeset-write preserves evidence and never automatically reruns (415.927966ms) +βœ” crash at after-command-changeset-write preserves evidence and never automatically reruns (358.284123ms) +βœ” crash at before-command-done-status preserves evidence and never automatically reruns (349.615417ms) +βœ” crash after done status but before query update recovers committed state without replay (453.032633ms) +βœ” crash ownership cannot be stolen automatically (304.197048ms) +βœ” hanging command times out unsafe and later accepted command commits (353.503122ms) +βœ” hanging load worker fails startup explicitly (200.604231ms) +βœ” integrity-enabled command writes changeset digest before done status (326.050121ms) +βœ” normal restart preserves committed state according to reconstruction oracle (450.499881ms) +βœ” accepted command changeset is not treated as committed state on startup (89.709014ms) +βœ” done command with missing changeset refuses recovery instead of silently using base data (74.28066ms) +βœ” malformed base OD-JSONTag file refuses recovery (87.926958ms) +βœ” malformed committed changeset refuses recovery (116.383408ms) +βœ” truncated committed changeset payload refuses recovery before lazy parsing can hide it (69.881388ms) +βœ” malformed uncommitted changeset is ignored during committed startup reconstruction (52.813884ms) +βœ” durable status file selects only done command changesets for startup (63.021015ms) +βœ” durable command log replays only accepted commands (4.553718ms) +βœ” active command at attempt 1 remains unchanged pending administrator assessment (4.411881ms) +βœ” active command at attempt 2 remains unchanged pending administrator assessment (2.707677ms) +βœ” malformed durable status record refuses recovery with explicit integrity error (8.38551ms) +βœ” structurally invalid durable status record refuses recovery (2.233453ms) +βœ” malformed durable command log record refuses recovery with explicit integrity error (2.417322ms) +βœ” structurally invalid durable command log record refuses recovery (3.238178ms) +βœ” OD-JSONTag framing validation catches truncated lazy records (0.366872ms) +βœ” OD-JSONTag framing validation accepts changeset skip records (0.473898ms) +βœ” integrity manifest detects same-length altered base OD-JSONTag payload (76.421226ms) +βœ” integrity manifest detects same-length altered committed changeset payload (75.768513ms) +βœ” integrity enabled requires manifest entry for base data (72.482922ms) +βœ” PL01-05 durable publication survives modeled loss of unsynced names and restart (925.400434ms) +βœ” PL10 an actual short changeset write is completed before acknowledgment (337.850245ms) +βœ” PL07-09/11/18-20 log sync failure stops mutation without false terminal success (241.49423ms) +βœ” PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (208.023773ms) +βœ” PL07-09/11/18-20 log close failure stops mutation without false terminal success (204.181151ms) +βœ” PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (272.554329ms) +βœ” PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (235.896548ms) +βœ” PL07-09/11/18-20 done sync failure stops mutation without false terminal success (256.456294ms) +βœ” PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (267.047535ms) +βœ” PL16 delayed append cannot be overtaken (336.666777ms) +βœ” PL16 delayed append cannot be overtaken by duplicate ID (267.117791ms) +βœ” PL02-04/06 missing command-log.jsontag fails open rather than recreating history (323.819448ms) +βœ” PL02-04/06 missing command-status.jsontag fails open rather than recreating history (336.522485ms) +βœ” PL02-04/06 missing data.A.jsontag fails open rather than recreating history (330.718406ms) +βœ” PL12/17 startup preserves uncertain active work without repeating external effects (285.553404ms) +βœ” directory sync failure after rename prevents done (211.426983ms) +βœ” directory close:after failure after rename prevents done (213.647192ms) +βœ” changeset open error is not masked or acknowledged (212.155863ms) +βœ” changeset write error is not masked or acknowledged (188.578415ms) +βœ” changeset close:after error is not masked or acknowledged (191.1675ms) +βœ” short command-log append preserves its entire JSONTag record (237.012495ms) +βœ” complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.216848ms) +βœ” serialized writer preserves order and stops after an uncertain operation (0.450738ms) +βœ” durable publication replaces entire contents and concurrent append retains each record once (33.8786ms) +β„Ή tests 106 +β„Ή suites 0 +β„Ή pass 106 +β„Ή fail 0 +β„Ή cancelled 0 +β„Ή skipped 0 +β„Ή todo 0 +β„Ή duration_ms 6379.304191 diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md index 298c8f3..9a521b2 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.md +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -191,3 +191,27 @@ finds only the same two historical undefined-meta errors in index load methods, now at their reformatted line positions. No dependency or lockfile changed; the pinned formatter was installed only in /tmp. Existing newline block preferences, canonical file formats and cycle acceptance status are preserved. + +## Example build and URL conversion correction β€” 2026-09-19 + +Verified implementation `d4136eaf7b0014726a1c90a0d0b8488ece2d6b27`: + +- The actual bundled Star Wars input failed the old converter with a URL syntax + error. Lazy OD parsing initially returned a root, but indexing forced the + invalid empty-base URL validation. Using the default base permits conversion. +- `test/conversion-offsets.mjs`: 7/7 passing, including actual example conversion, + every final byte offset, typed URLs and linked homeworld data. +- Full relevant suite: 106/106 passing, zero skips/TODOs. Same nine test files as + the preceding review run; output appended to the companion tests file. +- Disposable example copy: `npm run build` created data, indexes and empty logs, + released ownership, and refused a second build. `npm start` then served an HTTP + query returning Luke Skywalker. Only the listening port was changed in the copy + to use an available local port. Existing installed dependencies were reused; + no fresh network installation was tested. +- ESLint passes for the changed converter and converter tests. +- The user's existing failed-attempt example lock and `package.json~` are retained. + The recorded PID is absent and no generated example store files were present; + this observation is not treated as automatic lock-release authorization. + +No new VM power-cut exercise is claimed: storage protocol and formats are +unchanged. Human cycle acceptance remains pending. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl index 13861b3..a16cc0d 100644 --- a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl @@ -14,4 +14,7 @@ project:EVD-20260919-TTZ7C-27 a sd:VerificationEvidence ; sd:gitCommit "49fa48c123d0035df087b7a9d4c1460da92b85fa" ] ; sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; - sd:gitCommit "5d166a23b664ba9fbe34ae819feb1ed70959046c" ] . + sd:gitCommit "5d166a23b664ba9fbe34ae819feb1ed70959046c" ] ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "d4136eaf7b0014726a1c90a0d0b8488ece2d6b27" ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index a883826..2eeaf78 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -24,7 +24,7 @@ Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offs Active-cycle progress on 2026-09-19: `IMP-20260919-TTZ7C-26` implements the corrected `DES-20260919-TTZ7C-24/25` following explicit authorization to complete the whole cycle. Existing command/status/integrity and dataset formats remain unchanged, with no migration. Durable complete-write and file/directory barriers, ownership, log-authoritative serial acceptance/execution, command-only workers, strict startup assessment, and source-bound administrator recovery/backup/restore are implemented. `EVD-20260919-TTZ7C-27` records 101 passing regression tests plus 22 expanded administrator tests (three additional cases), direct CLI checks, and a passing four-scenario ext4/QEMU power-cut exercise with device/filesystem negative controls and latency. `DURABILITY.md` and `docs/recovery.md` define the bounded behavior and runnable procedure. Historical baseline gaps remain at `EVD-20260919-TTZ7C-23`; PL15/16 expectations were corrected by the maintainer, not silently weakened. Full rebuild remains a documented separate contract. The cycle is ready for human acceptance but not Accepted or integrated. Broader filesystem/hardware and randomized testing, undeclared custom dependencies and historical input/effect uncertainty remain explicit limits. Two unchanged legacy index `load` methods have pre-existing undefined-meta lint errors outside this write-path cycle. -Acceptance-review refinements: explicit multiline blocks, selective JAQT queries, fewer ternaries, and lines near 80 columns now improve the cycle code’s readability. Short conditional choices and long literals remain where clearer. The latest full relevant regression suite passes 105/105; EVD-27 records preserved ordering, backup coverage and plain-data report contracts. Human cycle acceptance is still pending. +Acceptance-review refinements: explicit multiline blocks, selective JAQT queries, fewer ternaries, and lines near 80 columns now improve the cycle code’s readability. Short conditional choices and long literals remain where clearer. The latest full relevant regression suite passes 106/106; EVD-27 records preserved ordering, backup coverage and plain-data report contracts. The example now uses `npm run build` for explicit initialization; conversion of its tagged URLs is fixed and build/start/query smoke verification passes. Human cycle acceptance is still pending. ## Intake State