diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.md b/.spiral/cycles/CYC-20260919-TTZ7C-17.md new file mode 100644 index 0000000..f20cf9c --- /dev/null +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.md @@ -0,0 +1,267 @@ +# Cycle: Power-Loss Durability And Administrative Recovery + +Branch: `spiral/CYC-20260919-TTZ7C-17-power-loss-recovery`. +Created from authoritative `master` at +`44ea91717a328ad7b5e049c6c0845b5e3bcc1718` on 2026-09-19. +Branch creation and correspondence were verified with Git. The pre-existing +untracked `package.json~` is unrelated and remains untouched. + +Status: **Active — implementation and evaluation complete; awaiting human acceptance**. +The maintainer authorized the whole cycle with the corrected no-format-change +design. Engineering completion does not mark the cycle Accepted or authorize +integration. See Evaluate for the evidence and practical limits. + +## Analyze + +### Goal + +Make command acceptance and completion durable across the supported power-loss +model, and provide an administrator with a tested, evidence-preserving recovery +route when committed storage cannot be reconstructed automatically. + +### Baseline effective behavior and gap (at cycle opening) + +The project has crash/retry, malformed-storage, handler, timeout, integrity, and +persisted-offset evidence. The latest offset cycle is accepted and merged. +Log appends already await `datasync`; changesets/final offsets already use file +sync plus rename. Directory entries, initial base/ID-index writes, sync-error +handling, and administrator recovery remain incomplete. Existing retry evidence +does not establish safety for secondary effects. The detailed inspected baseline +and input-completeness caveat are in UND-20260919-TTZ7C-19. + +### Why this boundary and why now + +A narrower sync-only change would leave administrators stranded after explicit +startup failure and would retain unsafe replay assumptions. One cycle therefore +covers failure evidence, persistence ordering, recovery decisions, and backup +restoration. Full rebuild has a separate evaluation boundary: intentionally +changed code and cooperative suppression of effects need their own workflow. + +### Plan continuity: revise + +Re-read SRC-001, REQ-001, and DES-001. After accepted crash/retry/handler and +integrity slices and the offset correction, bring file publication and +administrative recovery ahead of general randomized/soak testing. The new human +direction strengthens failure-only recovery expectations and preserves the +existing small architecture. It does not replace the remaining roadmap. + +### Adopted profiles + +The project's Muze engineering and library stewardship profiles apply: preserve +inspectability, small replaceable mechanisms, and explicit compatibility. +WPF-HUMAN-001 applies to acknowledgment evidence, administrator authority, and +recovery options. No new profile or independent veto is introduced. The concrete +scope is grounded in the maintainer's requirements. + +## Plan + +### Committed requirements + +REQ-20260919-TTZ7C-20 is the acceptance matrix (P1-P12), interpreted from retained +human direction in SRC-20260919-TTZ7C-18 and UND-20260919-TTZ7C-19. + +- Complete command inputs must be logged; demonstrated omissions are bugs. +- Command-log order is authoritative for live execution, reconstruction, restart, + and the later-dataset recovery guard. Status/queue timing cannot redefine it. +- The subsequent maintainer clarification makes the command itself the complete + input beyond the dataset: remove transport context from command execution. + Explore this cycle using current store formats without migration. The earlier + design-only phase is superseded by authorization to complete the cycle; + source wording is retained in SRC-20260919-TTZ7C-18. +- Ordinary rerun requires administrator approval and no later accepted command + with a dataset. Recover the eligible trailing sequence in original order. +- Missing data is not proof that external effects did not happen. Existing + corrupt/ambiguous data is not silently classified as missing. +- Automatic retries and ordinary queue activity cannot bypass recovery rules. +- Preserve original evidence, work on a copy, validate and explicitly promote. +- Provide a tested consistent backup/restore path for unsafe/disallowed reruns. +- Retain separate rebuild semantics: selected code/history, new output store, + side-effect suppression context for commands/hooks; full workflow follows. + +### Work sequence + +1. **Failure matrix and baseline:** enumerate durable artifacts and dependencies, + establish command ordering/ownership, model lost/reordered/partial writes and + lost directory entries, and record acknowledgments independently of the store. + Preserve failing evidence before correcting runtime behavior. +2. **Design:** specify acceptance/completion barriers, required versus rebuildable + indexes, initial-file publication, I/O uncertainty handling, recovery preview + and apply behavior, input completeness, and compatibility. Create DES/IMP/EVD + artifacts when those decisions/results crystallize, not empty placeholders. +3. **Durable writes:** complete file/directory sync sequencing before acceptance + or completion becomes visible; ensure errors cannot falsely report success + or allow unsafe further mutations. +4. **Administrative recovery:** implement inspect/preview and explicitly authorized + ordered recovery on a copy; enforce later-dataset guards, stop on uncertain + effects, revalidate stale plans, and record outcomes without erasing history. + Cover interruption of recovery itself and normal/automatic queue interactions. +5. **Inputs and backup/restore:** test normal versus recovered invocation inputs, + remove unlogged request inputs, diagnose incomplete historical commands, and + exercise consistent backup creation and restoration with reported coverage. +6. **Filesystem validation and evaluation:** exercise a disposable block-level + power-loss setup on a documented Linux/local filesystem, measure commit + latency, run relevant regression suites and checks, and provide a runnable + administrator guide with bounded durability claims and known limitations. + +### Acceptance examples + +| Scenario | Required result | +|---|---| +| Power cut after observed `accepted` | Complete accepted command survives under the supported model; replay safety is assessed separately | +| Power cut after observed `done` | Recovered state includes that completed transition, with no partial state, duplication, or silent rollback | +| Write/sync/rename failure | No false acknowledgment; uncertain outcomes prevent further mutation pending recovery | +| A has data, B missing, C has data | Rerunning B is blocked, even if C is not currently `done` | +| A has data, B missing, C waiting, D missing | Administrator can authorize B/C/D in order after judging external effects | +| Later dataset appears after preview | Stale recovery plan is rejected/recomputed before execution | +| Corrupt artifact or incomplete log evidence | Explicit diagnosis; do not infer eligibility from absence of valid parsing | +| Rerun unsafe or prohibited | Demonstrate backup restoration, disclose verified coverage, loss, and uncertainty | +| Interrupted administrative recovery | Original evidence remains intact; subsequent inspection can explain the partial recovery attempt | + +P1-P12 were open until evidenced; the evaluation below supplies that evidence. P12 is documentation of a +follow-up contract, not acceptance of an implemented rebuild feature. Existing +startup/offset, handler, ACID, and crash suites must remain green with any +intentional retry-policy changes reflected by justified new tests. + +### Non-goals + +Full rebuild tooling, broad randomized/soak campaigns, arbitrary external-effect +rollback or exactly-once delivery, sandbox replacement, general database/message +frameworks, remote-filesystem promises, package release, and production recovery +or deployment. New canonical file/record formats and migration are excluded by +the current design constraint. External administrative plans/reports are tool +outputs, not required store metadata. The focused block-level fault exercise +remains in scope. The later maintainer instruction authorizes runtime changes within this boundary. + +### Material assumptions and replan conditions + +The first envelope is a named Linux/local filesystem with storage honoring +flushes and one writer; exact environment and ownership checks are design work. +Destructive tests use disposable storage only. Missing test infrastructure leaves +the required exercise outstanding; it does not authorize a weaker claim. + +Replan affected work if authoritative command order cannot be established, +necessary historical inputs cannot be recovered, a required behavior cannot be +achieved without a format change/migration, or the smallest safe solution +contradicts the stated architecture/administrator rules. Storage irrecoverability +must be reported honestly, not hidden by an invented replay outcome. + +## Act + +Cycle opening captured source, understanding, and request as separate immutable +causal commits, with companion Turtle references to prior versions. Project +context and repository instructions now point to this refinement. + +### Baseline slice — 2026-09-19 + +DES-20260919-TTZ7C-21 defines an independent observer and test-only I/O fault +harness. IMP-20260919-TTZ7C-22 implements 21 scenarios plus the matrix in +`docs/power-loss-baseline.md`; EVD-20260919-TTZ7C-23 records results and limits. + +`npm run test:power-loss` passes its 21 scenario/control tests and explicitly +reports 10 violated invariant assertions as TODO. Strict mode exposes those +failures with exit 1. Existing regression suites pass 59/59. Runtime behavior +is unchanged; this is evidence of gaps, not a power-loss safety claim. + +The failures cover lost log/status/changeset names, acknowledged partial writes, +repeated external effects, absent later-dataset rerun guards, configured-worker +request input loss, changed waiting-queue order after restart, and automatic +reclassification of uncertain work before failed startup. Positive directory-sync +and normal-restart controls distinguish the modeled barrier conditions. + +### Durable-write and recovery design — 2026-09-19 + +DES-20260919-TTZ7C-24 specifies complete-write loops, file and directory barriers, +initial publication, ownership, serialized acceptance/status/outcome handling, +and command-only execution. After the maintainer clarified the original command +contract and requested design without a new data format/migration, the design +removed its v2 envelope, stored sequence/profile fields, and store descriptor. +Use the existing command/status/integrity records, with command-log position as +the sole authoritative order. New acceptance is serialized so queue/execution +order agrees; contrary historical execution is diagnosed as a defect. + +DES-20260919-TTZ7C-25 specifies read-only inspection, source-bound recovery plans, +the exact later-accepted-dataset guard, explicit execution approval, preserved +source evidence, interruption handling, offline backup/restore, and separate +promotion. Its revised authorization/audit and backup reports live outside the +canonical store, while recovery appends existing status types/fields. It no longer +requires special recovery transitions or a migration. Rebuild remains a follow-up +contract. Both designs map baseline gaps +and remaining acceptance tests to concrete implementation responsibilities. + +The earlier design-only phase removed the proposed new format and migration. +The maintainer subsequently authorized completing the whole cycle. The corrected +PL15 test removes request context; corrected PL16 makes live/restarted execution +follow the command log. Historical failures remain recorded in EVD-23. + +### Durable runtime and recovery — completed + +IMP-20260919-TTZ7C-26 implements complete durable writes, file/directory barriers, +conversion initialization, ownership, serial acceptance through queue insertion, +command-only workers and strict read-only startup validation. Storage uncertainty +stops mutation; pending/active work is never automatically rerun. + +Administrative inspect/apply/finish/unlock/backup/restore/verify operates on +preserved evidence and new candidates, with explicit effect assessment, exact +suffix approval, later-dataset guards, source/code revalidation and deliberate +promotion. Backup copies preserve relative integrity paths and permissions and +can be transferred. Full rebuild remains the separate documented contract. + +The administrator guide, durability contract and packaged disposable VM harness +are complete. EVD-20260919-TTZ7C-27 records test outcomes, named filesystem/cache +conditions, negative controls, recovered state and measured latency. + +## Evaluate + +Engineering result: **ready for human cycle acceptance within the stated envelope**. +All P1-P12 rows have bounded evidence in EVD-20260919-TTZ7C-27 +at `14c640088420bba49c96333946f1e370cf0a5c60`. The relevant suite +passed 101/101; the expanded administrator suite passed 22/22, adding three exact +acceptance examples (104 distinct tests across those runs), with no skips/TODOs. +The direct CLI conversion/inspection/backup/restore/verification procedure passed. +Sixteen provenance-validator regression tests passed. + +| Criteria | Result | +|---|---| +| P1-P4 | Durable complete records/publications, fault propagation, independent observed acknowledgments, authoritative acceptance/execution order | +| P5-P8 | Read-only diagnosis, approved ordered suffix, no automatic replay, exact command bytes and removal of request context | +| P9-P10 | Consistent portable backup/restore, known-loss reporting, preserved originals/audit, resumable assessment and verified promotion preview | +| P11 | QEMU/ext4 cut/reboot after accepted, active and five done commands; filesystem/device negative controls; exact environment and latency retained | +| P12 | Separate cooperative rebuild contract documented; rebuild tooling explicitly remains a following cycle | + +The VM used Linux 7.0.0-31-generic, QEMU 10.2.1, ext4 ordered data/barriers, +virtio-blk/writeback and a volatile NBD backend honoring FLUSH/FUA. Completed +A/L0/L1/L2/L3 survived in order; accepted/active A remained pending/uncertain. +Unsynced filesystem and device writes were lost in negative controls. This is +real guest-filesystem recovery over a modeled block cache, not physical hardware +certification or exhaustive fault enumeration. Timings and test limits are in +EVD-27; emulated ext4 and tmpfs numbers are not production performance promises. + +Compatibility: no canonical format change or migration. Commands no longer get +HTTP request inputs. Existing pending work requires administrator assessment, +and older/noncooperating writers must not share a store. Historical completeness, +original code/undeclared dependencies and external effects require independent +evidence; irrecoverable stores may need an explicitly disclosed backup rollback. + +The unchanged legacy index `load` methods have two pre-existing lint errors; +new/changed behavior checks pass. They are outside this cycle's write-path work. +Randomized/soak testing, additional filesystems/hardware and full changed-code +rebuild remain follow-up work. There is no release or production deployment. + +Final staged, range and tree provenance validation must pass before handing over +for acceptance. Keep this cycle Active until the maintainer accepts it; then +record acceptance in a new commit and validate the exact prospective integration +against current master. Do not merge this Active branch outward. + +### Acceptance-review readability refinements + +At the maintainer's request, all blocks use explicit braces and multiline bodies. +The subsequent JAQT refactor replaces dense diagnostic/recovery/backup expressions +with named queries and comparison steps. Runtime behavior and plain-data public +reports are preserved. The full relevant suite now passes 105/105, with the latest +verification recorded in EVD-27. These refinements do not accept or integrate the +cycle and do not replace the exact earlier filesystem evidence. + +The subsequent conditional/width revision expands dense ternaries, retains ten +short readable choices, and wraps the cycle code toward 80 columns with literal +exceptions. The 105-test suite and direct recovery/backup CLI checks pass; EVD-27 +records formatting and behavior-preservation checks. Acceptance is still pending. diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl new file mode 100644 index 0000000..9cd3555 --- /dev/null +++ b/.spiral/cycles/CYC-20260919-TTZ7C-17.ttl @@ -0,0 +1,22 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:CYC-20260919-TTZ7C-17 + a sd:Cycle ; + dcterms:identifier "CYC-20260919-TTZ7C-17" ; + sd:repositoryPath ".spiral/cycles/CYC-20260919-TTZ7C-17.md" ; + sd:status sd:Active ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" + ] ; + sd:constrainedBy [ + a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "8a42ed15242c174a296eb90ed4a35b3354d79e8d" + ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-21.md b/.spiral/designs/DES-20260919-TTZ7C-21.md new file mode 100644 index 0000000..4a191dd --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-21.md @@ -0,0 +1,69 @@ +# Design: Power-Loss Baseline Harness + +## Purpose and boundary + +Implement the first work item of CYC-20260919-TTZ7C-17: discriminating baseline +tests and a failure matrix before persistence/recovery fixes. The design follows +REQ-20260919-TTZ7C-20. Product behavior remains unchanged in this slice. + +Use actual server/worker writes with a test-only Node preload. Trace completed +file operations and syncs outside the disposable store; capture synchronized +bytes for the small fixture logs. Independently record client-issued commands, +observed acceptance, and observed completion. The expected state comes from +fixture semantics and external observations, never solely from damaged logs. + +The preload is explicitly enabled only by a test configuration and is inherited +by worker threads. Scope interception to the disposable store. Capture filesystem +function references before wrapping so tracing cannot recurse. Controls must +prove that instrumentation observes real appends, file syncs, and renames. + +## Fault model + +- Trace file sync and namespace publication separately. After abrupt process + termination, remove a newly created/renamed filename only when its trace shows + no subsequent directory sync. Such image edits model a permitted persistence + outcome; they are not physical power cuts or measured filesystem behavior. +- At an explicit pause before a log sync, restore its last successfully synced + byte snapshot. Only unsynced data may be lost in this model. Preserve external + acknowledgments and side-effect witnesses outside the crash image. +- Inject I/O errors before a selected operation, and a real partial callback + write where applicable. Assert that fault injection was actually reached. +- Torn/corrupt durable artifacts and synthetic recovery-status combinations are + separate adversarial/policy fixtures. Do not present arbitrary mutation of + previously synced bytes as a valid cache-loss model. +- A controlled append-completion gate can exercise concurrent acceptance order + without relying on timing races. Test that subsequent recovery does not derive + its expected order from the same log it is checking. + +## Reporting and tests + +Ordinary assertions verify setup, interception, completed operations, observed +responses, and current classifications. Known requirement violations appear in +nested tests marked TODO. `SIMPLYSTORE_BASELINE_STRICT=1` removes only those TODO +annotations, exposing reproducible failing invariant assertions with a nonzero +exit. A fixture/harness failure outside that final invariant remains a real +failure in normal baseline mode. Remove TODO annotations as later fixes earn it. + +Cover durable acceptance/completion, missing directory entries, derived index +loss, interrupted status append, explicit sync/rename/write failures, automatic +replay with secondary effects, later-dataset rerun restrictions, command input +completeness, and concurrent queue order. Name untested matrix rows explicitly; +this baseline is not acceptance of the whole cycle or administrator tooling. + +Add an isolated `test:power-loss` command and a human-readable matrix with test +names, expected/current results, and limitations. Existing suites stay separate +and are run as regression evidence. Only new test files, package test scripts, +and Spiral/documentation artifacts belong to this implementation slice. + +## Independence and limits + +External observations and side-effect witnesses survive every modeled cut and +are not reconstructed from the store under test. Fixture command behavior is +simple and known. The OD-JSONTag parser remains shared for reading actual bytes; +compare its output against independently expected values/order. + +The trace covers the explicit filesystem APIs used by these fixtures, not all +Node/native/custom-module I/O. It cannot prove hardware flush honesty, torn-sector +semantics, mount behavior, cross-process writer exclusion, consistent backups, +or full administration/rebuild workflows. A disposable block-level exercise and +the remaining P1-P12 work are still required later in the active cycle. diff --git a/.spiral/designs/DES-20260919-TTZ7C-21.ttl b/.spiral/designs/DES-20260919-TTZ7C-21.ttl new file mode 100644 index 0000000..88b678e --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-21.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260919-TTZ7C-21 a sd:Design ; + dcterms:identifier "DES-20260919-TTZ7C-21" ; + sd:repositoryPath ".spiral/designs/DES-20260919-TTZ7C-21.md" ; + sd:status sd:Active ; + sd:satisfies [ a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md new file mode 100644 index 0000000..287f165 --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -0,0 +1,295 @@ +# Design: Durable Publication And Ordered Command Records + +## Status, scope, and evidence + +Design step of CYC-20260919-TTZ7C-17, following REQ-20260919-TTZ7C-20 and the +failures in EVD-20260919-TTZ7C-23. This document specifies intended behavior; +it does not claim implementation or cycle acceptance. It continues the active +cycle's agreed refinement of the durability roadmap. + +The maintainer's subsequent correction in SRC-20260919-TTZ7C-18 constrains this +earlier revision to design only, using the existing store formats without migration. +It supersedes this document's earlier v2 envelope/descriptor proposal. The subsequent authorization in SRC-20260919-TTZ7C-18 permits completing the +whole cycle with this corrected design; the no-format-change constraint remains. + +Retain OD-JSONTag base/changeset bytes, filenames, command records, status records, +and the optional integrity-record schema. No envelope, sequence field, execution +profile field, required store descriptor, log conversion, or mixed-format decoder +is introduced. Existing stores receive prospective write hardening after validated +opening; historical loss or uncertainty cannot be repaired by upgrading software. +Operational locks and external recovery plans/reports are described explicitly +below and in DES-20260919-TTZ7C-25; they are not a new canonical store format. + +## Durable file operations + +Introduce a small storage module, shared by runtime, conversion, and recovery: + +| Operation | Completion condition | +|---|---| +| `writeAll(handle, bytes)` | Loop over bytesWritten until complete; reject zero progress, invalid counts, and errors; never equate one successful write callback with a complete file | +| `publishFile(path, bytes)` | Create a unique temporary file in the same directory, write all bytes, sync file, close, rename to destination, sync parent directory, close directory | +| `appendRecord(path, bytes)` | Serialize writes per file; append a complete newline-terminated record, sync file data, close, sync parent directory, close directory | +| `syncDirectory(path)` | Open and sync the directory, await completion and close; propagate unsupported-operation and I/O errors | + +The initial implementation syncs the append parent on every completed append. +This avoids relying on a fragile existence check or process-local cache to infer +that a filename was previously durable. Optimize only after measurement and tests. +For nested new directories, make each containing entry durable up to an already +established directory. File permissions should follow existing behavior. + +File sync does not itself persist a directory entry: see the +[Linux fsync contract](https://man7.org/linux/man-pages/man2/fsync.2.html). +Use [Node FileHandle writes](https://nodejs.org/docs/latest-v24.x/api/fs.html#filehandlewritebuffer-offset-length-position) +and check their reported byte count. The design does not assume `writeFileSync` +or an atomic rename alone constitutes a persistence barrier. + +Temporary-file cleanup is best effort and must not obscure the first I/O error. +A failed open must not be masked by closing an undefined handle. Propagate close +failures as well as write/sync/rename failures. Preserve uncertain final artifacts +for inspection; do not delete a published file in an attempt to prove rollback. + +`publishFile` alone is not a multi-file transaction. Commit sequencing below +establishes which complete files belong to canonical state. Initial conversion +should publish into a new directory under ownership and validate all output before +releasing ownership; do not overwrite an operating store's base independently of +its other artifacts. No completion descriptor is added to the store. + +## Ownership and serialization + +All mutation runs under exclusive store ownership. For the first dependency-free +implementation use atomically created lock directories, one per canonical mutable +directory, acquired in sorted path order. This deliberately treats stores sharing +the default index/log directory as conflicting. Resolve existing parent paths; +do not allow symlink aliases to create independent locks on the same directory. + +Write owner diagnostics (random token, PID, host, start time) and sync the lock +entry before any store mutation. The token must match on release; clean shutdown +removes the lock and syncs its parent. Crash leftovers require an explicit offline +administrator release after confirming no writer remains. PID age, a timeout, or +a failed signal probe alone must never authorize automatic stealing. A crash +between mkdir and metadata publication is still a lock, not an empty slot. + +This is cooperative exclusion, not protection against arbitrary file writers. +Read-only inspection does not create source lock files. Backup and recovery apply +require a stopped/quiescent source plus ownership; verify source fingerprints +after acquiring it. Tests must demonstrate competing startup/apply exclusion and +crash-leftover handling. Kernel advisory locks are an alternative, but would need +a proven Node integration; no new native dependency is chosen in this step. + +Within the owner process use one acceptance FIFO and one serialized status writer. +Worker messages do not release the command-runner slot until the parent finishes +persisting the outcome. Only one command handler runs at a time. Acceptance may +continue during a healthy running command, but storage uncertainty closes both +acceptance and execution queues. + +## Complete commands; no transport execution inputs + +The logged command value is the complete invocation input beyond the preceding +dataset. Keep one JSONTag command per line, with existing `id`, `name`, and +application fields. Serialize once after validation; use that exact command +snapshot for logging and initial execution, and its retained value for recovery. +Preserve tagged values; do not add server metadata or HTTP fields to the command. + +Remove request context from the server's command-worker messages and remove the +request parameter from `runCommand`. The handler call becomes +`handler(dataspace, command, undefined, metaProxy)`: keep the fourth argument's +position for compatibility, leaving the former request slot empty. Dataset and +index access remain. Queries and HTTP admission/access checks can still use the +request; command execution cannot. Applications needing a value must supply it +as part of the complete command before acceptance/logging, not through a worker +request side channel. Never infer or silently inject omitted application inputs. + +The default worker already omits request when calling `runCommand`; PL15 exposes +a configured-worker escape route, not a requirement to preserve that route. +The revised P8 test must prove that HTTP query/method/URL context is unavailable +even through a configured worker, and that command fields and dataset/index +access are equivalent during initial execution and authorized recovery. Keep the +original baseline as historical evidence; its proposed request-retention assertion +is superseded by this corrected input contract, not accepted as a product need. + +Removing transport context does not sandbox arbitrary configured code. Required +timestamps, random choices, or external lookup results must also be command data +if they affect the intended repeatable transition; application validation and +tests establish that contract. The engine cannot detect every omitted semantic +input or certify arbitrary hooks. Historical request-dependent commands remain +defective and may be unrecoverable by rerun; this cycle will diagnose them, not +invent replacement inputs or rewrite their logs. Restoring verified data can still +avoid execution. A future rebuild-mode control is separate from business inputs. + +## Ordering with existing records + +**Command-log position is the sole authoritative command order**, as confirmed +by the maintainer in SRC-20260919-TTZ7C-18. Acceptance, live execution, committed +reconstruction, restart queues, and the administrative later-dataset predicate +must all follow that order. Status records describe outcomes and attempts; their +insertion order and client response timing cannot redefine the command sequence. + +The acceptance FIFO includes duplicate-ID checking, command-log append, +accepted-status append, in-memory status, and queue insertion. Both appends and +their required directory entries must be durable before HTTP 202. The resulting +log order is also first-acceptance order and handler execution order for new work. +An in-memory ordinal can aid checks; no sequence number is stored. Response +delivery order across clients is not the order of accepted commands. + +Retain existing status fields and values: `command`, `code`, `status`, and existing +attempt/message fields where applicable. Serialize all status appends, including +active and terminal outcomes. Reconstruct and queue by command-log position, +not incidental latest-status Map iteration. Failed/unsafe terminal +commands with no committed delta remain in that history without applying data. + +For existing history, compare first-acceptance and active/done execution evidence +against the command log to detect violations, never to choose another authority. +Older records may disagree (PL16). Keep repeated status/attempt records for +inspection. Identical duplicate command-log entries reference one logical ID at +its first occurrence; conflicting values for one ID require diagnosis. + +If only waiting-queue or acceptance-status order differs, the command order is +still known: use the complete log. That disagreement alone does not make the log +order ambiguous. Pending work still follows the administrator-assessment policy +below. If commands actually executed out of log order, their positional changesets +may encode an incompatible preceding state. Do not simply reorder those files +and claim a correct reconstruction. Diagnose the affected history, serve only a +verified unaffected prefix where appropriate, and use the guarded recovery or +backup path. Status evidence can reveal the defect; it cannot legitimize a +different command order. A missing/corrupt/incomplete command log is a separate +loss of authoritative evidence: other records cannot silently replace it. + +The revised ordering test must compare actual handler starts, completed state, +and restarted pending execution with command-log order. Exercise an A append +whose asynchronous completion is delayed while B is submitted; B must not overtake +A. Include a sequential control and the waiting-at-crash case. PL16's historical +assertion requiring restart to preserve the old B/A queue is superseded: with an +A/B log, restarting A/B is correct and the live B/A queue is the defect. Retain +that observation as evidence; do not fix the test by making restart follow B/A. + +Keep the existing client retry contract: an existing ID returns its known status +without another append or execution; fixing the concurrent duplicate race is in +scope. A new payload-conflict response policy is unnecessary for this cycle. +A log record without acceptance is an orphan; a status without its command is +inconsistent evidence. Neither may be silently executed, discarded, or reused. +Storage uncertainty halts admission even if acceptance might already be durable. + +## Opening and initialization without a descriptor + +Separate opening an existing store from explicit new-store initialization. +Opening requires the configured base, command log, and status log, plus the +manifest when integrity is configured as required. Missing files are errors, +never an invitation to create empty logs. Empty log/status pairs remain valid +for a fresh store when consistent with its other evidence; one empty log with +nonempty accepted history or unexplained changesets requires diagnosis. +Inventory configured artifact locations, including unexpected changesets. + +Initialization/conversion targets new output, acquires ownership before writing, +and writes the base, empty logs/status, required indexes and any configured +integrity manifest in their current formats. Sync and validate all output before +releasing ownership or serving requests. Crash leftovers retain the ownership +lock and require offline inspection. Release never makes an incomplete output +ready merely because some files parse. A pre-existing base with absent logs needs +an explicit administrator assessment; normal open cannot prove it has no history. + +For an existing validated store, establish file/directory barriers for the retained +base and required committed artifacts before admitting new writes, so subsequent +completion does not depend on old cached prerequisites. This syncs existing bytes; +it does not rewrite formats or prove no earlier data was lost. Custom required +artifacts must be covered. Startup cost is measurable work in P11. + +There is no on-disk flag proving a store was written by the hardened implementation. +The guarantee is prospective for operations performed by that implementation, +from a validated starting state, under its documented filesystem/ownership +envelope. Keep the deployed binary/configuration as operational evidence. An old +binary can still read these formats, but it does not provide the new ordering, +sync, or recovery safety policy; downgrading does not preserve those guarantees. + +Integrity must be explicitly configured if its continued presence is required; +the existing existence-based autodetection alone cannot detect a lost manifest. +Current digests detect covered byte mismatches, not loss of all history or a +coherent older snapshot. Without an independent retained witness, syntactically +valid truncation or replacement of a whole history may be undetectable. Neither +new software nor a static descriptor could reconstruct that missing information. + +## Command execution and completion + +1. Select the earliest eligible accepted command in established order. Append/sync + active before invoking a worker. Eligibility on restart is conservative as + specified below; absence of active is not retrospective proof of never started. +2. Execute command and current pre-serialization index mutations. Await required + hooks, including returned promises, before final serialization; test ordering + and rejection behavior rather than silently changing handler permissions. +3. Publish the complete changeset through the durable primitive. Preserve sparse + OD-JSONTag numbering and zero-byte no-op changesets. +4. Await index finalization; publish required built-in sidecars durably. The + current default loader can rebuild ID/offset indexes, as PL05 shows, but that + is not a reason to leave promised output files half-written. Custom finalizers + that promise required external files must use equivalent barriers; awaiting + a hook alone cannot certify arbitrary external I/O. +5. If integrity is enabled, append/sync its changeset digest, including the + manifest directory barrier. Validate manifest/base requirements at startup. +6. Append/sync done. Only then update live status, publish buffers to queries, + and release the runner slot. A query-publication failure cannot undo done; + stop/reload query service from committed state instead of appending failed. + +Any required persistence failure enters a storage-failed state: close admission, +terminate active execution if necessary, stop queue advancement, and report an +uncertain outcome. Do not append a misleading failed record after a possibly +durable done. The error path must not try to keep operating through the same +failed status writer. Safe reads require separately established committed state; +the simplest first implementation may exit after surfacing the diagnostic. + +Handler errors and timeouts remain distinct from storage faults. If terminal +failed/unsafe status can be durably recorded, no canonical transition is published +and normal subsequent work may continue under the existing timeout contract. +Such commands are never automatically rerun; existing uncommitted files and +external effects still require diagnosis before administrative action. + +## Startup and implementation order + +Parse and cross-check the input log, status transitions/order, required files, +framing, and configured integrity without changing command state or importing +mutation-capable modules. Reconstruct only established committed transitions. +Missing or inconsistent committed data fails before any recovery status append. +Active or artifact-bearing uncommitted commands are uncertain; no automatic +active-to-accepted rewrites, even below the former crash-attempt threshold. + +Because old and new records are indistinguishable, this design also requires +administrator assessment for pending accepted-only commands found on restart. +They may have run under the old retry protocol or lost old unsynced evidence. +They cannot be labeled certainly never-started solely from their current status. +The administrative plan includes waiting work explicitly and enforces the same +later-dataset predicate. Fully terminal, consistent stores can restart normally; +freshly accepted work in a healthy uninterrupted process runs normally. Recovery +of a pending suffix blocks new mutations that would overtake it. Automatic restart +resumption with durable proof of the protocol is deferred, not smuggled into a +new checkpoint format. This conservative availability tradeoff needs tests and +operator documentation; it does not require data migration. + +Implementation sequence when runtime work resumes: byte-complete publish/append +and directory barriers first (PL02/03/04/10 plus conversion/integrity), then parent +error handling/ownership, the acceptance FIFO and command-only worker boundary, +read-only startup validation, and administrative recovery under DES-20260919-TTZ7C-25. +The maintainer has now authorized that implementation sequence through evaluation. + +New tests must cover initial creation versus open, unchanged stored schemas, +existing-store operation without conversion, short/zero writes, close/open error +masking, every barrier, optional integrity loss, concurrent acceptance/duplicates, +terminal-status persistence failures, query-update failure after done, locks, +ambiguous historical ordering, and conservative pending-command startup. PL15's +replacement must test removal of unlogged inputs, not their retention. The harness +must trace newly used filesystem APIs before claiming coverage. Measure write and +startup sync latency and run the disposable filesystem exercise. None of these +future passing results is claimed by the design. + +## Reach within this cycle + +| Acceptance rows | Design achievable with existing store formats | Boundary | +|---|---|---| +| P1-P3 | Complete writes, ordered durable acceptance/completion, explicit uncertain errors | Prospective protocol guarantee; hardware must honor the tested barriers | +| P4, P11 | Independent witness, fault matrix, named filesystem exercise, latency | Implementation and actual filesystem evidence still required | +| P5-P7, P10 | Read-only inspection, source-bound suffix recovery, preserved evidence and deliberate promotion | Unknown history/order blocks rerun; restart pending work requires administrator assessment | +| P8 | Commands are the complete invocation input; remove worker request context | Arbitrary code needs conformance testing; omitted historical inputs are not repairable by assumption | +| P9 | Offline consistent backup and restore using unchanged store files | Report backup coverage and any unknown/lost history; no guarantee of lossless repair | +| P12 | Retain separate rebuild/effect-suppression contract | Full rebuild tooling remains follow-up | + +The P1-P12 goals need no new canonical data format on this design. Acceptance +remains contingent on implementation/evidence within these stated bounds. If a +required behavior cannot be achieved within them, record the gap and revisit the +cycle with the maintainer; do not quietly introduce a format extension/migration. diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.ttl b/.spiral/designs/DES-20260919-TTZ7C-24.ttl new file mode 100644 index 0000000..efe6ab7 --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-24.ttl @@ -0,0 +1,27 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . +@prefix culture: . + +project:DES-20260919-TTZ7C-24 a sd:Design ; + dcterms:identifier "DES-20260919-TTZ7C-24" ; + sd:repositoryPath ".spiral/designs/DES-20260919-TTZ7C-24.md" ; + sd:status sd:Active ; + sd:satisfies [ a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] ; + sd:derivedFrom [ a sd:ArtifactReference ; + sd:artifact project:EVD-20260919-TTZ7C-23 ; + sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ], + [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "8a42ed15242c174a296eb90ed4a35b3354d79e8d" ] ; + sd:shapedBy [ a sd:ArtifactReference ; + sd:artifact culture:CUL-MUZE-001 ; + sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ], + [ a sd:ArtifactReference ; + sd:artifact culture:CUL-MUZE-LIB-001 ; + sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.md b/.spiral/designs/DES-20260919-TTZ7C-25.md new file mode 100644 index 0000000..2a733ca --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-25.md @@ -0,0 +1,318 @@ +# Design: Administrative Recovery And Consistent Restore + +## Status and scope + +Step 2 of CYC-20260919-TTZ7C-17. Implements the recovery intent in +REQ-20260919-TTZ7C-20 at design level, using the durable operations and ordered +history of DES-20260919-TTZ7C-24. EVD-20260919-TTZ7C-23 supplies the baseline gaps. +No recovery tool or new runtime guarantee is implemented by this document. + +Ordinary recovery preserves the meaning/order of the existing history. Rebuild +is the separately retained option to execute selected history using changed code +into a new dataset. It must not become a `--force` bypass in ordinary recovery. +The maintainer's design-only correction retains current store files and schemas, +with no migration. This revision removes the earlier dependence on v2 records, +stored sequence/profile fields, and special recovery status types. + +Recovery plans, audit journals, and backup/completion reports are new outputs of +the administrative tools, kept outside the canonical store. Normal loading does +not need to parse them. They bind administrator actions and interrupted-tool +resumption, not ordinary dataset reconstruction. Temporary ownership locks are +operational exclusion, not a format-version marker. No required store descriptor, +new commit log, or runtime dependency on an administrative manifest is introduced. + +## Read-only inspection + +Proposed entry point: `node scripts/recover.mjs inspect --store --out `. +The store config identifies the base, log/status paths, integrity settings and +deployment information, so non-default paths are not silently replaced by CLI +defaults. Accept declarative configuration; do not execute server configuration +modules merely to discover paths. +This command parses data and metadata without invoking command handlers, custom +workers, custom index modules, or their top-level imports. Existing load-worker +behavior that imports a custom index is not an acceptable inspection mechanism. + +Source must be quiescent or an immutable copy for an actionable plan. Inspection +may diagnose a live store but must label that output non-actionable. It records +source identity, file inventory, sizes and cryptographic hashes, expected missing +paths, enabled integrity state, and code/profile identity. Check inventory and +hashes before and after inspection; that check detects change but is not a +substitute for quiescence. Write plans outside the source tree; do not create +lock files, rewrite statuses, truncate logs, or generate indexes during inspection. + +For each command report ID, authoritative order/evidence, original logged inputs, +acceptance and execution attempts, final status, expected dataset path, artifact +condition, and external-effect uncertainty. Report the last verified committed +prefix and commands whose effects cannot currently be reconstructed. Failed or +unsafe commands with a valid terminal outcome have no committed delta and do not +by themselves break the canonical prefix. Missing done data does break it. + +Validate complete newline-terminated records, required fields, duplicate-ID +consistency, acceptance/attempt history, established order, base identity, +OD-JSONTag framing, +and enabled integrity. A malformed tail remains evidence requiring intervention; +do not discard it merely because it is last. A syntactically valid log is not +proof that missing history never existed. If coverage cannot be established, +report unknown coverage and block assertions about rerun eligibility. + +Command-log position is authoritative, as specified in DES-20260919-TTZ7C-24. +Use acceptance and active/done records to detect deviations, not to select another +order. A complete log settles order even if an old waiting queue or acceptance +status sequence disagrees. If actual execution differed, diagnose whether retained +changesets depend on an incorrect preceding state; do not reorder positional +changesets and claim a valid repair. Allow committed reads only for verified +unaffected state. Independent backups can restore missing authoritative evidence +or valid data; they cannot make a conflicting status order authoritative. Missing +or damaged command-log evidence remains a separate recovery limitation. + +Reading a command cannot establish that its implementation used no hidden input. +The plan records the administrator's evidence about the chosen code and complete +command contract. Historical request-dependent handlers cannot be repaired by +supplying a transient request or modifying the logged command during ordinary +recovery. If verified saved data is unavailable, report that rerun is blocked. + +## Classification and decisions + +| Evidence | Default recovery decision | +|---|---| +| Done with verified dataset and prerequisites | Reconstruct from bytes; never execute the handler | +| Done with missing dataset | Needs recovery; assess later-dataset rule and external effects | +| Accepted only, no active attempt or dataset | Apparently waiting work; on restart require administrator assessment because the format cannot prove the write protocol or absence of prior execution | +| Active, any uncertain attempt, or uncommitted dataset present | Administrator diagnosis; no automatic retry or automatic promotion to done | +| Failed/unsafe terminal outcome | No automatic rerun; report attempts and any surviving uncommitted artifacts | +| Existing corrupt/partial dataset | Diagnose or restore verified artifacts; do not pretend the file is missing | +| Missing/inconsistent command inputs or order | Block rerun; independent evidence may establish order or confirm existing inputs, but this cycle does not inject missing inputs or rewrite a command | +| Valid data with missing default derived indexes | Rebuild default indexes from committed bytes on the destination; do not run mutation-capable legacy hooks | + +Restoring a missing file from a verified matching backup is not execution. It can +repair an interior gap when its identity, bytes, and corresponding commit evidence +match. A matching command ID alone is insufficient. Existing data without a +trustworthy commit outcome is not automatically committed just because it parses. + +## Exact ordinary-rerun predicate + +Use the preserved source's accepted history and complete artifact inventory, +not a target directory after files have been removed: + +1. Locate the command and every later accepted command by command-log position, + retaining acceptance history even if the latest status is failed/unsafe/active. + Never use acceptance-status insertion order or actual execution order to make + a later dataset appear earlier and thereby permit a rerun. +2. For an execution candidate with a missing dataset, **every later accepted + command must also have no dataset**. Presence includes zero-byte no-op files, + corrupt files, files belonging to accepted-but-not-done commands, and files + whose validity is uncertain. Unknown presence/order fails the predicate. +3. Establish a valid preceding committed state. Keep original order across the + selected trailing missing/waiting commands. Do not skip an unresolved earlier + command and execute a later command against a different starting state. +4. Require explicit approval identifying commands and assessed secondary effects. + The predicate establishes eligibility, not evidence that execution is safe. + +Thus A:data / B:missing / C:data blocks B regardless of C's current status. +A:data / B:missing / C:waiting / D:missing can admit B/C/D in order after approval. +Known terminal no-transition commands can remain terminal; an administrator must +explicitly select a rerun if reconsidering them. If an earlier failed command is +selected for rerun, the same later-dataset restriction applies. + +There is no ordinary option to delete/quarantine a later source dataset to make +an interior command eligible. Selecting an earlier backup is an explicit rollback +with its own known-loss report, not a claim to have recovered all existing history. +The full changed-code rebuild exception is described separately below. + +## Preview and apply + +The plan contains the source snapshot fingerprint, source/base and backup +identities, captured order and inputs, code/profile fingerprints, intended target, +commands to reconstruct/restore/rerun, preserved terminal commands, expected +artifacts, unresolved issues, and known/unknown loss. It states which decisions +require administrator judgment. It contains no assertion that side effects are +safe merely because a checkbox can be selected. + +Proposed application: + +```text +node scripts/recover.mjs apply --plan plan.json --to recovered-store --audit-dir recovery-audit \ + --approve-rerun B,C,D --operator --reason +``` + +1. Require a quiescent source and acquire the ownership locks specified by the + storage design. No live server queue may advance concurrently. Acquire target + ownership too; target must be new and outside the source. Reject symlink/path + overlap and destination overwrite. Plan paths do not grant arbitrary writes. +2. Rescan the full inventory, including absent expected paths and newly appearing + datasets, and verify source/backup/profile hashes. Any mismatch invalidates the + plan before command code is loaded. Recompute the later-dataset predicate; + an administrator approval cannot override it. + Recognized ownership locks are excluded from content fingerprints so acquiring + the tool's own lock is not a false mismatch; no data/log files are excluded. +3. Copy/preserve source evidence and construct the verified preceding state in + the target. Never hard-link mutable source logs/files into the target. Preserve + both original records and the plan rather than editing history to look clean. +4. Publish a durable recovery authorization in the external audit directory before + invoking any selected command. It binds plan fingerprint, operator/reason, + ordered IDs and command hashes, source/target locations, and deployment + evidence. No missing business inputs are supplied through the tool. Keep the + immutable source copy, plan, and audit together as the recovery evidence bundle; + make all these writes durable before execution, outside canonical store paths. +5. Execute sequentially with the same worker/handler inputs as normal execution. + Mark each recovery attempt durably active before invoking it, and commit using + DES-20260919-TTZ7C-24. Use the existing active/done/failed/unsafe status values, + command IDs and attempt fields; append outcomes without erasing earlier records. + Only the offline approved tool can reopen a terminal command. Link each attempt + to authorization and original order in the external audit, not through new + status fields/types. Record the planned attempt durably there before the active + append; if either write is uncertain, stop without invoking the command. No HTTP + server or ordinary queue runs on the candidate during apply. Since reopening + status is not a new schema, older readers may parse it but must not be used to + operate an unfinished candidate; they lack the new exclusion/recovery policy. +6. Stop on uncertain execution, I/O error, handler failure, or timeout. Leave the + candidate and audit record for inspection; never automatically resume an active + recovery attempt. A new plan and fresh external-effect decision are required + before a retry. Completed attempts can be reconstructed without executing them. +7. Verify that every selected restore/rerun has completed as approved, with no + unresolved commands or unexplained artifacts. Validate the candidate's complete + committed state and required files, sync them, + and publish the external completion report durably before releasing candidate + ownership. Unfinished candidates retain the ordinary ownership lock; normal + startup refuses that lock. Lock removal is never the generic stale-PID action: + it requires an offline assessment, and for a recovery candidate that includes + its bound plan/audit, remaining commands, and completion report. If that bundle + is lost, the tool cannot certify or automatically resume the attempt. Preserve + the candidate for diagnosis or start a new approved recovery from the source. + +The source inventory remains the guard throughout a plan. As B, C, D are recovered +in order, B's newly created target data is a predecessor, not a later source +dataset that would invalidate C. On interruption, revalidate the preserved source +and audit plus current candidate before any renewed approval. A durable target +done record with valid data can be reconstructed without rerun even if writing +its external audit outcome failed; an audit claim alone cannot establish commit. +Do not mark failed after a possibly durable done. Retrying an uncertain attempt +with an existing partial dataset remains blocked pending separate diagnosis; +automatic cleanup must not turn that artifact into apparent absence. + +Where a retained digest identifies the missing committed changeset, compare the +recreated bytes to that expectation before appending a new integrity record or +done status. A mismatch requires diagnosis; do not erase the old expectation by appending a new +digest and then claim the original state was recovered. Preserve the candidate +bytes and uncertainty. Without such a witness, structural reconstruction and a +complete-command contract cannot independently prove byte-identical historical +output; report that verification limit. + +Ordinary recovery uses the original command meaning and compatible deployment. +There is no execution-profile field in the command log. The administrator supplies +retained deployment/source/package evidence in the external plan, and the tool +binds the selected code to prevent changes after preview. A current hash proves +only what is selected now, not what executed originally. Unresolved code identity +blocks a confident ordinary rerun; a materially changed handler must not silently +become a changed-code rebuild. The engine's request-removal fix is compatible with +handlers that already obeyed the complete-command contract. It is not proof that +historical handlers which used request context can be rerun correctly. + +## Backup, restore, and promotion + +First implement offline consistent backup rather than live incremental snapshots: +stop the writer, acquire ownership, inventory configured base/log/status/integrity +and required artifacts, and copy them to a new destination. Record configuration +and code-profile identity without copying unrelated secrets. Durable-publish all +files, validate their hashes and committed reconstruction, and write a backup +completion manifest last in the external backup evidence bundle, syncing its +directory. The copied store retains its original format and does not require +that manifest for ordinary loading; the restore tool requires it to verify this +backup procedure completed. Reject incomplete backups for automatic restore. +A backup that merely parses may already contain data loss; +record its source coverage and integrity limits honestly. + +Restore accepts a verified completed backup, writes a separate candidate, validates +it, and reports its recovered commands and differences from retained source +history. Unknown source history yields explicit unknown loss. Additional surviving +committed changesets may be applied only when order, base lineage, commit evidence, +and required predecessors are verified; never guess that two backups share a base. + +Keep original command/status records as a byte-preserved prefix of a candidate +when they are complete and valid; recovery appends use existing schemas. Preserve +all originals separately in every case. Malformed logs, unresolved order, or +missing command inputs are diagnostic/backup-restore cases in this cycle, not an +automatic log-repair or migration facility. Selecting a completed older backup +can be a useful deliberate rollback, but cannot be called recovery of all later +accepted work. Missing or corrupt saved data need matching backup evidence or +an eligible rerun; the design offers no lossless solution when neither exists. + +Promotion remains a separate explicit administrator operation. Stop the existing +service, revalidate the completed candidate/report, and switch the deployment's +store configuration to the new location while preserving the old directory and +configuration. Document a reversible service-specific sequence; do not claim +that swapping several directories is one atomic operation. The new server must +validate the chosen store before listening; the promotion tool verifies the +external completion report. Normal loading does not need that report. After new +commands are accepted, rolling back to the old directory can lose those commands and requires +another explicit coverage/loss assessment. + +Lock release after a crash is offline and explicit; stale ownership does not +authorize removing data or re-executing commands. The administrator guide must +cover lock diagnostics alongside missing/corrupt artifacts and interrupted apply. + +## Rebuild exception retained for follow-up + +A future rebuild operation takes an explicit starting dataset, chosen history, +and chosen code profile, producing a new store and regenerated derived/integrity +artifacts. Its execution context communicates `mode: rebuild` and +`sideEffectsAllowed: false` to commands and relevant hooks. Ordinary recovery +does not silently set this flag, because skipping an effect that never happened +can be just as incorrect as repeating one that did. + +The flag is cooperative; audit commands/hooks and test effect suppression before +calling a profile rebuild-safe. External reads, clock/randomness, and omitted +inputs still affect reproducibility. Rebuild can deliberately differ under new +code. Full rebuild CLI, execution, and activation are outside this cycle. + +## Verification mapping and implementation sequence + +| Design boundary | Required evidence | +|---|---| +| Read-only inspection | Byte-identical source before/after, no module/handler effects, malformed-tail and missing-history diagnostics | +| Rerun predicate | PL13 both status variants, zero-byte/corrupt later files, failed/unsafe acceptance history, missing order, permitted B/C/D suffix | +| Apply binding | Changed source file, newly appearing dataset, changed code/backup, path alias/overlap, stale plan and competing writer all rejected before execution | +| Execution | PL12 effect witness unchanged until approval; approved sequence preserves inputs/order; no live queue overtake; every active/done barrier fault exercised | +| Interrupted apply | Originals intact, incomplete candidate blocked, completed attempts not repeated, active attempt requires renewed decision | +| Inputs/order | PL14; corrected PL16 expectations comparing live/restarted execution to the log; replacement for PL15 proving request context is unavailable; missing log evidence, out-of-order historical changesets, incomplete commands, and changed/unknown deployment identity | +| Existing format | Unchanged command/status/integrity schemas, no descriptor or conversion, prior command bytes retained, accepted-only restart requires assessment | +| Audit separation | Final store loads without tool reports; apply/promotion requires its external evidence; faults between audit/active/done/report writes stop safely | +| Recreated state | Existing expected digest matches or recovery stops; absent digest reports verification limits; incomplete approved plans cannot be promoted | +| Backup/restore | Consistent backup under exclusion, incomplete backup rejection, verified reconstruction, coverage/loss report, interrupted copying, restore without rerunning handlers | +| Promotion | Candidate validation, explicit configuration switch, retained original, failures before/after activation and honest rollback limits | + +Implement inspection/classification first, then plan validation and reconstruction +on a copy, then guarded execution and interruption records, then offline backup/ +restore and operator documentation. Reuse the same ordered record decoder and +durable operations as normal execution. Keep the reference oracle independent. +Do not convert expected failures to passing tests by weakening these decisions. +The exception is PL15's input-retention assertion: its historical failure remains +evidence, but its interpretation is corrected by the maintainer's complete-command +boundary. Document that change and add the discriminating removal/equivalence test. +PL16's assertion is also corrected: restart must follow the command log, not the +old in-memory queue. Add a predicate test with log A/B/C and conflicting status +order so C's dataset still blocks rerunning B. The live acceptance race remains +a defect to fix; the correction does not erase the reproduced ordering mismatch. + +Still required for acceptance: implement and execute the above tests, record +latency and the Linux/storage envelope, run the disposable block-level power-loss +exercise, and obtain human evaluation of the entire active cycle. The subsequent maintainer authorization permits implementation and evaluation +of this design. Unknown historical inputs/order, unavailable backup evidence, and +irrecoverable history remain explicit limits, not reasons to introduce migration. + +## Concrete operational choices + +Copy canonical files and explicitly declared required artifacts, preserving their +relative directory layout and permissions so existing integrity paths retain their +meaning. Do not sweep unrelated application files or secrets into a backup. Code +is separately selected and hashed; deployments and undeclared dependencies still +need administrator assessment. Backups can be transferred before restoration; +validate and rebase their external inventory without rewriting canonical bytes. + +Recovery protects incomplete destination layouts with a root ownership lock. +An interrupted final report can be finished from validated retained authorization +and completed data without executing commands. A remaining active suffix requires +fresh assessment and a new candidate. Backup coverage comparisons require a +quiescent retained source and compare base, command, status and surviving data. +Promotion remains an explicit deployment configuration operation after validation; +this cycle supplies a verified configuration preview, not a deployment integration. diff --git a/.spiral/designs/DES-20260919-TTZ7C-25.ttl b/.spiral/designs/DES-20260919-TTZ7C-25.ttl new file mode 100644 index 0000000..7104b43 --- /dev/null +++ b/.spiral/designs/DES-20260919-TTZ7C-25.ttl @@ -0,0 +1,20 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260919-TTZ7C-25 a sd:Design ; + dcterms:identifier "DES-20260919-TTZ7C-25" ; + sd:repositoryPath ".spiral/designs/DES-20260919-TTZ7C-25.md" ; + sd:status sd:Active ; + sd:satisfies [ a sd:ArtifactReference ; + sd:artifact project:REQ-20260919-TTZ7C-20 ; + sd:gitCommit "b1f15b5964986a9e3cab9025f08829b88294974c" ] ; + sd:derivedFrom [ a sd:ArtifactReference ; + sd:artifact project:EVD-20260919-TTZ7C-23 ; + sd:gitCommit "40f87397774e08b6549fc95dcd966922ddbaa7e9" ] ; + sd:constrainedBy [ a sd:ArtifactReference ; + sd:artifact project:DES-20260919-TTZ7C-24 ; + sd:gitCommit "088e389bfb2e2fa33c77c3c8d6c6b3f8af110e9d" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-23.md b/.spiral/evidence/EVD-20260919-TTZ7C-23.md new file mode 100644 index 0000000..fa820da --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-23.md @@ -0,0 +1,107 @@ +# Evidence: Power-Loss Baseline And Recovery Gaps + +## Claim and implementation + +IMP-20260919-TTZ7C-22 at `0766186d24e973c47bc78865c3b71a6272effde0` +implements the independent baseline harness and fault matrix. This verifies the +first work item of CYC-20260919-TTZ7C-17, not completion of the durability cycle. +The unchanged runtime baseline remains +`44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. + +## Method and environment + +Executed on 2026-09-19 using Node v24.7.0 on Linux with the installed dependencies. +Tests bind localhost on ephemeral ports and use disposable store/observer +directories. The server preloader traces actual current file API calls in both +main and worker threads. Client observations and side-effect witnesses remain +outside the copied crash images. No production data was changed. + +The tests model selected permitted cache/namespace-loss outcomes after SIGKILL, +inject actual API failures/partial writes, and exercise explicit adversarial and +policy fixtures. They do not cut physical power or test the deployment filesystem. +`docs/power-loss-baseline.md` records each distinction and uncovered matrix rows. + +## Commands and results + +| Check | Result | +|---|---| +| `npm run test:power-loss` | Exit 0; 21 scenario tests pass, 10 nested invariant assertions fail as explicit TODO; 31 total, no unexpected failures | +| `npm run test:power-loss:strict` | Expected exit 1; the same 10 invariant failures become ordinary failures; 9 enclosing scenarios also fail, giving Node totals of 12 pass / 19 fail / 31 tests | +| Existing startup/offset, ACID, crash, and handler suites | 59/59 pass | +| ESLint on all three new test files | Pass | +| JavaScript syntax checks and `git diff --check` | Pass | +| Staged provenance validation before design and implementation commits | Pass | + +The regression command was: + +```sh +node --test --test-isolation=none --test-reporter=tap \ + test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs \ + test/acid-baseline.mjs test/durability-crash.mjs test/after-change-handlers.mjs +``` + +Strict-mode failures were inspected: all top-level failures are `subtestsFailed` +from the named requirement assertions, not fixture/instrumentation failures. +The baseline and strict final runs took about 5.9 seconds each; this is test +duration, not a production commit-latency measurement. + +## Reproduced gaps + +1. **PL02 / P2:** after external observation of `done`, dropping the unsynced new + status filename makes restart serve `[]` instead of independently expected + `['A']`. The surviving log alone does not detect this silent rollback. +2. **PL03 / P1:** dropping the unsynced new command-log filename leaves accepted + work without a reconstructible task; server still starts. +3. **PL04 / P2:** losing an unsynced changeset rename after `done` produces an + explicit missing-changeset startup failure rather than recovery of the commit. +4. **PL10 / P3:** a callback write that writes only half the changeset succeeds; + the installed atomic-write dependency does not complete the remaining bytes. + The live server receives intact memory bytes and reports `done`, but startup + subsequently rejects the truncated disk file. This is actual short-write + injection, not post-commit tampering with a complete file. +5. **PL12 / P7:** after an external effect and a pause before done-status sync, + restore the last synced status bytes and lose the unpublished changeset name. + Restart automatically repeats the effect: witness `['A', 'A']` while the + dataset contains one transition. No administrator authorized the second call. +6. **PL13 / P6 (two assertions):** synthetic B-accepted/missing and C-with-dataset + policy fixtures still queue B whether C's current status is accepted or done. +7. **PL15 / P8:** a configured worker forwards request query input to its handler; + live execution uses `name=from-request`, while reconstructed task request is + null. The ordinary JSONTag command body itself passes the PL14 round-trip + control. This qualifies the earlier FIXME speculation with actual evidence. +8. **PL16 / P4/P6:** controlled concurrent acceptance produces external acceptance + and waiting-queue order B/A, but log order A/B. After a crash, recovery executes + A/B. The observer's order is independent of the log being checked. +9. **PL17 / P7:** startup reclassifies uncertain active A as accepted before + failing on missing committed C. Original records remain intact as an append + prefix; the finding is unauthorized replay eligibility, not erased history. + +## Passing controls and useful bounds + +- PL01 traces real append/data-sync and worker temp-file-sync/rename order and + verifies normal restart against independent expected values. +- PL21 supplies an explicit directory sync in a test-only finalizer. The trace + observes it, the model preserves the published filenames, and restart matches + the external expectation. The loss model distinguishes presence of a barrier. +- PL05 shows current default startup reconstructs without ID/offset sidecars; + this does not classify custom consumers or future lazy-index loading. +- PL06 rejects a torn status tail while preserving the untouched original store. +- PL07/08/09 reject changeset fsync, rename, and write errors before completion. +- PL11 exits on done-status sync failure while written done bytes remain visible: + an I/O failure is not proof that the command rolled back. +- PL18/19/20 return HTTP 500 rather than acceptance after log/status sync or + post-close errors. + +## Consequences and remaining work + +Baseline investigation is complete for the named matrix rows. Design the durable +publication/error-handling changes and ordered administrator recovery against +these failures before altering runtime behavior. Retain immutable failure +evidence and convert TODO invariants into ordinary passing tests as fixes land. + +Initial conversion publication, integrity combinations, additional I/O cases, +single-writer enforcement, administrator preview/apply and stale-plan guards, +allowed trailing recovery, backup/restore, latency, and the disposable block-level +filesystem exercise remain open. Full rebuild remains the recorded follow-up. +Neither passing controls nor expected failures establish actual power-loss +resilience or cycle acceptance. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-23.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-23.ttl new file mode 100644 index 0000000..b6dd11a --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-23.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260919-TTZ7C-23 a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260919-TTZ7C-23" ; + sd:repositoryPath ".spiral/evidence/EVD-20260919-TTZ7C-23.md" ; + sd:status sd:Active ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-22 ; + sd:gitCommit "0766186d24e973c47bc78865c3b71a6272effde0" ] . diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json new file mode 100644 index 0000000..6f47e26 --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-results.json @@ -0,0 +1,313 @@ +{ + "implementation": "049c45642fe60146bd511bb87abb4b032f8df36a", + "additionalAcceptanceTests": "13db3bf55926cf3a3a1b2769c422881a875193c8", + "regression": { + "passed": 101, + "failed": 0, + "skipped": 0, + "todo": 0 + }, + "administratorTestsAfterThreeAdditions": { + "passed": 22, + "failed": 0, + "skipped": 0, + "todo": 0 + }, + "vmEnvironment": { + "qemu": "QEMU emulator version 10.2.1 (Debian 1:10.2.1+ds-1ubuntu3.2)\nCopyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers\n", + "node": "v24.7.0\n", + "kernelSha256": "19fd789cd5e6b4adfe18c9c6b92a1ce8c8b2eef6ff88456cef80ee2dca9b3f72", + "repositoryHead": "049c45642fe60146bd511bb87abb4b032f8df36a", + "worktreeDiffSha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "filesystem": "ext4 data=ordered,barrier=1", + "device": "virtio-blk, QEMU cache=writeback, volatile NBD backend honoring FLUSH/FUA", + "acceleration": "TCG, 2 CPUs, 1024 MiB", + "physicalPowerCut": false + }, + "guestEnvelope": { + "kernel": "Linux version 7.0.0-31-generic (buildd@lcy02-amd64-091) (x86_64-linux-gnu-gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, GNU ld (GNU Binutils for Ubuntu) 2.46) #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 04:26:38 UTC 2026\n", + "mounts": "rootfs / rootfs rw,size=178920k,nr_inodes=44730,inode64 0 0\nproc /proc proc rw,relatime 0 0\nsysfs /sys sysfs rw,relatime 0 0\ndevtmpfs /dev devtmpfs rw,relatime,size=178920k,nr_inodes=44730,mode=755,inode64 0 0\n/dev/vda /store ext4 rw,relatime,data=ordered 0 0\n", + "writeCache": "write back\n" + }, + "vmObserver": [ + { + "deviceControl": "flushed A survives; unflushed B lost", + "discardedDirtyPages": 1 + }, + { + "scenario": "done", + "recover": false, + "marker": "CUT_DONE {\"samples\":[{\"id\":\"A\",\"doneMs\":2520.205792},{\"id\":\"L0\",\"acceptMs\":44.68866600000001,\"doneMs\":868.762804},{\"id\":\"L1\",\"acceptMs\":33.01929300000029,\"doneMs\":872.86582},{\"id\":\"L2\",\"acceptMs\":31.05912999999964,\"doneMs\":755.6233089999996},{\"id\":\"L3\",\"acceptMs\":31.991556999999375,\"doneMs\":829.8371459999998}]}", + "flushes": 101, + "writes": 134, + "discardedDirtyPages": 0 + }, + { + "scenario": "done", + "recover": true, + "marker": "VERIFY {\"ready\":true,\"errors\":[],\"commands\":[{\"id\":\"A\",\"status\":\"done\",\"present\":true},{\"id\":\"L0\",\"status\":\"done\",\"present\":true},{\"id\":\"L1\",\"status\":\"done\",\"present\":true},{\"id\":\"L2\",\"status\":\"done\",\"present\":true},{\"id\":\"L3\",\"status\":\"done\",\"present\":true}],\"persons\":[{\"name\":\"A\"},{\"name\":\"L0\"},{\"name\":\"L1\"},{\"name\":\"L2\"},{\"name\":\"L3\"}],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 8, + "discardedDirtyPages": 0 + }, + { + "scenario": "accepted", + "recover": false, + "marker": "CUT_ACCEPTED", + "flushes": 25, + "writes": 30, + "discardedDirtyPages": 0 + }, + { + "scenario": "accepted", + "recover": true, + "marker": "VERIFY {\"ready\":false,\"errors\":[],\"commands\":[{\"id\":\"A\",\"status\":\"accepted\",\"present\":false}],\"persons\":[],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 8, + "discardedDirtyPages": 0 + }, + { + "scenario": "active", + "recover": false, + "marker": "CUT_ACTIVE", + "flushes": 27, + "writes": 33, + "discardedDirtyPages": 0 + }, + { + "scenario": "active", + "recover": true, + "marker": "VERIFY {\"ready\":false,\"errors\":[],\"commands\":[{\"id\":\"A\",\"status\":\"active\",\"present\":false}],\"persons\":[],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 8, + "discardedDirtyPages": 0 + }, + { + "scenario": "unsynced", + "recover": false, + "marker": "CUT_UNSYNCED", + "flushes": 13, + "writes": 17, + "discardedDirtyPages": 0 + }, + { + "scenario": "unsynced", + "recover": true, + "marker": "VERIFY {\"ready\":true,\"errors\":[],\"commands\":[],\"persons\":[],\"unsyncedPresent\":false}", + "flushes": 4, + "writes": 7, + "discardedDirtyPages": 0 + } + ], + "vmResults": [ + { + "scenario": "done", + "cut": "CUT_DONE {\"samples\":[{\"id\":\"A\",\"doneMs\":2520.205792},{\"id\":\"L0\",\"acceptMs\":44.68866600000001,\"doneMs\":868.762804},{\"id\":\"L1\",\"acceptMs\":33.01929300000029,\"doneMs\":872.86582},{\"id\":\"L2\",\"acceptMs\":31.05912999999964,\"doneMs\":755.6233089999996},{\"id\":\"L3\",\"acceptMs\":31.991556999999375,\"doneMs\":829.8371459999998}]}", + "recovered": { + "ready": true, + "errors": [], + "commands": [ + { + "id": "A", + "status": "done", + "present": true + }, + { + "id": "L0", + "status": "done", + "present": true + }, + { + "id": "L1", + "status": "done", + "present": true + }, + { + "id": "L2", + "status": "done", + "present": true + }, + { + "id": "L3", + "status": "done", + "present": true + } + ], + "persons": [ + { + "name": "A" + }, + { + "name": "L0" + }, + { + "name": "L1" + }, + { + "name": "L2" + }, + { + "name": "L3" + } + ], + "unsyncedPresent": false + } + }, + { + "scenario": "accepted", + "cut": "CUT_ACCEPTED", + "recovered": { + "ready": false, + "errors": [], + "commands": [ + { + "id": "A", + "status": "accepted", + "present": false + } + ], + "persons": [], + "unsyncedPresent": false + } + }, + { + "scenario": "active", + "cut": "CUT_ACTIVE", + "recovered": { + "ready": false, + "errors": [], + "commands": [ + { + "id": "A", + "status": "active", + "present": false + } + ], + "persons": [], + "unsyncedPresent": false + } + }, + { + "scenario": "unsynced", + "cut": "CUT_UNSYNCED", + "recovered": { + "ready": true, + "errors": [], + "commands": [], + "persons": [], + "unsyncedPresent": false + } + } + ], + "tmpfsLatency": { + "samples": 30, + "startupMs": 127.17863, + "acceptance": { + "p50Ms": 0.9827060000000074, + "p95Ms": 2.0263760000000275, + "maxMs": 35.388232999999985 + }, + "completion": { + "p50Ms": 39.87711200000001, + "p95Ms": 44.079794999999876, + "maxMs": 79.74672899999999 + }, + "filesystem": "tmpfs", + "node": "v24.7.0", + "notes": "Single localhost client; completion includes a fresh worker and polling; no production performance claim" + }, + "cliSmoke": [ + "convert --integrity", + "inspect --quiescent", + "backup --quiescent", + "restore --store --quiescent", + "verify" + ], + "jaqtRefactor": { + "implementation": "49fa48c123d0035df087b7a9d4c1460da92b85fa", + "passed": 105, + "failed": 0, + "skipped": 0, + "todo": 0, + "changedProductionFiles": [ + "src/server.mjs", + "src/store-inspection.mjs", + "src/admin-recovery.mjs" + ], + "plainDataCloning": "passed", + "backupCoverageVariations": [ + "same", + "command", + "status", + "data", + "base", + "missing" + ], + "changedFileLint": "passed", + "newVmRun": false + }, + "conditionalWidthRefactor": { + "implementation": "5d166a23b664ba9fbe34ae819feb1ed70959046c", + "passed": 105, + "failed": 0, + "skipped": 0, + "todo": 0, + "style": { + "files": 26, + "originalTernaries": 36, + "currentTernaries": 10, + "styleErrors": 0, + "styleWarnings": 0, + "fullLint": [ + { + "file": "src/index.id.mjs", + "messages": [ + { + "line": 49, + "rule": "no-undef", + "message": "'meta' is not defined." + } + ] + }, + { + "file": "src/index.offset.mjs", + "messages": [ + { + "line": 74, + "rule": "no-undef", + "message": "'meta' is not defined." + } + ] + } + ] + }, + "formatterSyntaxTreesMatched": 27, + "generatedVmModulesMatched": 4, + "storePathConfigurationsMatched": 16, + "cliSmoke": [ + "convert", + "inspect", + "apply A", + "backup", + "restore with source", + "restore without source", + "verify" + ], + "newVmRun": false + }, + "exampleBuildCorrection": { + "implementationCommit": "d4136eaf7b0014726a1c90a0d0b8488ece2d6b27", + "tests": 106, + "passed": 106, + "failed": 0, + "skipped": 0, + "converterTests": 7, + "lintErrors": 0, + "lintWarnings": 0, + "exampleNpmBuildStartQuery": "passed in disposable copy with available port", + "overwriteRefused": true, + "freshDependencyInstall": false, + "existingExampleLock": "preserved", + "newVmRun": false + } +} diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt new file mode 100644 index 0000000..1a0f8fa --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27-tests.txt @@ -0,0 +1,539 @@ +simplystore-acceptance-tests.log +✔ atomicity: command mutation followed by failure does not become visible or committed (458.010713ms) +✔ isolation: mutating data inside a JavaScript query does not affect later queries (320.714073ms) +✔ consistency: query during an active command sees the previous committed state (1049.932589ms) +✔ idempotency: duplicate command ID is acknowledged but not applied twice (504.230852ms) +✔ idempotency: retry during active command reports current status and does not enqueue twice (1013.644938ms) +✔ administrator recovers trailing missing/waiting commands in log order on a copy (218.649497ms) +✔ later accepted dataset blocks rerun, including zero-byte no-op (12.996066ms) +✔ later done dataset blocks rerun, including zero-byte no-op (10.972497ms) +✔ later unsafe dataset blocks rerun, including zero-byte no-op (17.307829ms) +✔ new later file makes an approved plan stale before executing a handler (15.721993ms) +✔ changed code and missing explicit approval reject apply (17.297063ms) +✔ cooperative ownership excludes competing writer and path aliases (11.828539ms) +✔ backup and restore validate saved bytes without executing commands (279.629517ms) +✔ read-only inspection never imports handlers and rejects conflicting done order (19.357133ms) +✔ historical status order never makes a later log dataset eligible for bypass (17.97534ms) +✔ interrupted recovery preserves completed attempts and blocks an unfinished candidate (187.880494ms) +✔ retained digest mismatch stops recovery before replacing integrity or appending done (78.7803ms) +✔ malformed tails and overlapping paths cannot be accepted as valid stores (7.249875ms) +✔ approved recovery resumes from completed candidate prefix without repeating external effects (266.76745ms) +✔ finishing an interrupted completion report reconstructs data without executing again (98.479939ms) +✔ multi-directory integrity paths and file permissions survive backup and restore (29.938398ms) +✔ backup excludes unrelated secrets while retaining declared custom artifacts (27.105721ms) +✔ a backup remains restorable after transfer to a different directory (55.572793ms) +✔ promotion verification rejects code changed after completed recovery (79.208975ms) +✔ handler lifecycle: index update failure prevents command commit (432.338152ms) +✔ handler lifecycle: derived index file can remain after failed command (397.476824ms) +✔ handler lifecycle: final offset write failure prevents command commit (289.95578ms) +✔ handler lifecycle: custom finalizer rejection prevents command commit (264.197927ms) +✔ handler lifecycle: current index update can mutate canonical state before commit (276.577028ms) +✔ command edit indexes only its changed record using changeset UTF-8 offsets (17.352132ms) +✔ command addition indexes new entities and preserves interior skip numbers (9.110173ms) +✔ command offsets include mutations and new records introduced by the index hook (7.558886ms) +✔ successive command indexes are relative to their own changeset files (8.252224ms) +✔ no-op commands replace any stale sidecar with an empty offset index (5.626016ms) +✔ failure to write final offsets rejects the command before success (5.18124ms) +✔ commands await custom finalization that delegates to the default index (28.165176ms) +✔ custom finalization can replace the default without an unconditional offset write (4.302294ms) +✔ custom finalization rejection propagates from the command worker (5.008679ms) +✔ default conversion offsets match every final record (100.442045ms) +✔ conversion finalizes offsets after custom indexing changes record sizes and adds records (98.071236ms) +✔ conversion replaces stale offsets even when the custom hook writes no index (130.345011ms) +✔ conversion awaits custom finalization with the original receiver and final bytes (99.235133ms) +✔ conversion propagates rejection from custom finalization (60.646362ms) +✔ conversion initializes existing-format logs and optional base integrity without overwriting a store (202.842664ms) +✔ runtime environment defaults to production and ignores production fault points (1.555028ms) +✔ fault points are inert outside test environment (0.183785ms) +✔ crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (518.498482ms) +✔ crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (396.358861ms) +✔ crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (364.429233ms) +✔ crash at before-command-changeset-write preserves evidence and never automatically reruns (379.091026ms) +✔ crash at after-command-changeset-write preserves evidence and never automatically reruns (338.042732ms) +✔ crash at before-command-done-status preserves evidence and never automatically reruns (357.359279ms) +✔ crash after done status but before query update recovers committed state without replay (463.445396ms) +✔ crash ownership cannot be stolen automatically (301.306069ms) +✔ hanging command times out unsafe and later accepted command commits (322.625473ms) +✔ hanging load worker fails startup explicitly (193.730659ms) +✔ integrity-enabled command writes changeset digest before done status (329.91289ms) +✔ normal restart preserves committed state according to reconstruction oracle (410.676844ms) +✔ accepted command changeset is not treated as committed state on startup (89.67717ms) +✔ done command with missing changeset refuses recovery instead of silently using base data (61.171916ms) +✔ malformed base OD-JSONTag file refuses recovery (79.053377ms) +✔ malformed committed changeset refuses recovery (92.273027ms) +✔ truncated committed changeset payload refuses recovery before lazy parsing can hide it (49.439526ms) +✔ malformed uncommitted changeset is ignored during committed startup reconstruction (54.649112ms) +✔ durable status file selects only done command changesets for startup (63.786153ms) +✔ durable command log replays only accepted commands (3.713886ms) +✔ active command at attempt 1 remains unchanged pending administrator assessment (6.586364ms) +✔ active command at attempt 2 remains unchanged pending administrator assessment (2.793342ms) +✔ malformed durable status record refuses recovery with explicit integrity error (4.069897ms) +✔ structurally invalid durable status record refuses recovery (5.262939ms) +✔ malformed durable command log record refuses recovery with explicit integrity error (1.916061ms) +✔ structurally invalid durable command log record refuses recovery (2.074256ms) +✔ OD-JSONTag framing validation catches truncated lazy records (0.327636ms) +✔ OD-JSONTag framing validation accepts changeset skip records (0.241231ms) +✔ integrity manifest detects same-length altered base OD-JSONTag payload (75.395083ms) +✔ integrity manifest detects same-length altered committed changeset payload (89.357652ms) +✔ integrity enabled requires manifest entry for base data (78.988138ms) +✔ PL01-05 durable publication survives modeled loss of unsynced names and restart (834.651879ms) +✔ PL10 an actual short changeset write is completed before acknowledgment (290.104519ms) +✔ PL07-09/11/18-20 log sync failure stops mutation without false terminal success (213.085435ms) +✔ PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (202.657879ms) +✔ PL07-09/11/18-20 log close failure stops mutation without false terminal success (180.136444ms) +✔ PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (255.984928ms) +✔ PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (254.343941ms) +✔ PL07-09/11/18-20 done sync failure stops mutation without false terminal success (266.18276ms) +✔ PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (249.577537ms) +✔ PL16 delayed append cannot be overtaken (317.039566ms) +✔ PL16 delayed append cannot be overtaken by duplicate ID (271.982054ms) +✔ PL02-04/06 missing command-log.jsontag fails open rather than recreating history (319.898933ms) +✔ PL02-04/06 missing command-status.jsontag fails open rather than recreating history (310.826762ms) +✔ PL02-04/06 missing data.A.jsontag fails open rather than recreating history (316.136257ms) +✔ PL12/17 startup preserves uncertain active work without repeating external effects (278.472846ms) +✔ directory sync failure after rename prevents done (189.041135ms) +✔ directory close:after failure after rename prevents done (186.196454ms) +✔ changeset open error is not masked or acknowledged (188.168142ms) +✔ changeset write error is not masked or acknowledged (185.555103ms) +✔ changeset close:after error is not masked or acknowledged (190.574387ms) +✔ short command-log append preserves its entire JSONTag record (195.428932ms) +✔ complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.23133ms) +✔ serialized writer preserves order and stops after an uncertain operation (0.477868ms) +✔ durable publication replaces entire contents and concurrent append retains each record once (33.390561ms) +ℹ tests 101 +ℹ pass 101 +ℹ fail 0 +ℹ skipped 0 +ℹ todo 0 +ℹ duration_ms 5928.321784 + +simplystore-acceptance-examples-detail.log +✔ administrator recovers trailing missing/waiting commands in log order on a copy (121.693187ms) +✔ later accepted dataset blocks rerun, including zero-byte no-op (3.134899ms) +✔ later done dataset blocks rerun, including zero-byte no-op (2.581528ms) +✔ later unsafe dataset blocks rerun, including zero-byte no-op (3.826567ms) +✔ new later file makes an approved plan stale before executing a handler (4.010908ms) +✔ changed code and missing explicit approval reject apply (5.358755ms) +✔ cooperative ownership excludes competing writer and path aliases (1.982249ms) +✔ backup and restore validate saved bytes without executing commands (75.292536ms) +✔ read-only inspection never imports handlers and rejects conflicting done order (4.477334ms) +✔ historical status order never makes a later log dataset eligible for bypass (2.365027ms) +✔ interrupted recovery preserves completed attempts and blocks an unfinished candidate (90.496786ms) +✔ retained digest mismatch stops recovery before replacing integrity or appending done (50.347143ms) +✔ malformed tails and overlapping paths cannot be accepted as valid stores (2.83327ms) +✔ approved recovery resumes from completed candidate prefix without repeating external effects (150.275693ms) +✔ finishing an interrupted completion report reconstructs data without executing again (57.460823ms) +✔ multi-directory integrity paths and file permissions survive backup and restore (19.356728ms) +✔ backup excludes unrelated secrets while retaining declared custom artifacts (9.276232ms) +✔ a backup remains restorable after transfer to a different directory (13.518402ms) +✔ promotion verification rejects code changed after completed recovery (53.110458ms) +✔ verified A prefix permits only missing B, waiting C and missing D in log order (309.225479ms) +✔ restore reports accepted commands absent from an older backup (11.018172ms) +✔ identical historical duplicate retains first log position; conflicting duplicate blocks (6.571817ms) +ℹ tests 22 +ℹ pass 22 +ℹ fail 0 +ℹ skipped 0 +ℹ todo 0 +ℹ duration_ms 1023.688896 + +JAQT refactor at 49fa48c123d0035df087b7a9d4c1460da92b85fa +✔ atomicity: command mutation followed by failure does not become visible or committed (532.107676ms) +✔ isolation: mutating data inside a JavaScript query does not affect later queries (324.200054ms) +✔ consistency: query during an active command sees the previous committed state (1054.010803ms) +✔ idempotency: duplicate command ID is acknowledged but not applied twice (530.143222ms) +✔ idempotency: retry during active command reports current status and does not enqueue twice (998.609118ms) +✔ administrator recovers trailing missing/waiting commands in log order on a copy (241.327474ms) +✔ later accepted dataset blocks rerun, including zero-byte no-op (16.185536ms) +✔ later done dataset blocks rerun, including zero-byte no-op (16.552248ms) +✔ later unsafe dataset blocks rerun, including zero-byte no-op (7.222186ms) +✔ new later file makes an approved plan stale before executing a handler (18.110875ms) +✔ changed code and missing explicit approval reject apply (45.191118ms) +✔ cooperative ownership excludes competing writer and path aliases (26.89224ms) +✔ backup and restore validate saved bytes without executing commands (266.833759ms) +✔ read-only inspection never imports handlers and rejects conflicting done order (8.947992ms) +✔ historical status order never makes a later log dataset eligible for bypass (6.309472ms) +✔ interrupted recovery preserves completed attempts and blocks an unfinished candidate (177.417264ms) +✔ retained digest mismatch stops recovery before replacing integrity or appending done (89.767431ms) +✔ malformed tails and overlapping paths cannot be accepted as valid stores (7.603056ms) +✔ approved recovery resumes from completed candidate prefix without repeating external effects (328.172436ms) +✔ finishing an interrupted completion report reconstructs data without executing again (91.850579ms) +✔ multi-directory integrity paths and file permissions survive backup and restore (91.720318ms) +✔ backup excludes unrelated secrets while retaining declared custom artifacts (17.910951ms) +✔ a backup remains restorable after transfer to a different directory (22.998871ms) +✔ promotion verification rejects code changed after completed recovery (87.006294ms) +✔ verified A prefix permits only missing B, waiting C and missing D in log order (460.71018ms) +✔ restore reports accepted commands absent from an older backup (20.271611ms) +✔ identical historical duplicate retains first log position; conflicting duplicate blocks (9.887978ms) +✔ backup coverage distinguishes changed history and data from missing source data (205.716463ms) +✔ handler lifecycle: index update failure prevents command commit (519.483947ms) +✔ handler lifecycle: derived index file can remain after failed command (376.97463ms) +✔ handler lifecycle: final offset write failure prevents command commit (298.168032ms) +✔ handler lifecycle: custom finalizer rejection prevents command commit (298.790672ms) +✔ handler lifecycle: current index update can mutate canonical state before commit (332.522035ms) +✔ command edit indexes only its changed record using changeset UTF-8 offsets (32.151269ms) +✔ command addition indexes new entities and preserves interior skip numbers (9.547552ms) +✔ command offsets include mutations and new records introduced by the index hook (6.824472ms) +✔ successive command indexes are relative to their own changeset files (7.807516ms) +✔ no-op commands replace any stale sidecar with an empty offset index (5.904184ms) +✔ failure to write final offsets rejects the command before success (6.037079ms) +✔ commands await custom finalization that delegates to the default index (27.552478ms) +✔ custom finalization can replace the default without an unconditional offset write (7.920356ms) +✔ custom finalization rejection propagates from the command worker (4.951147ms) +✔ default conversion offsets match every final record (85.67897ms) +✔ conversion finalizes offsets after custom indexing changes record sizes and adds records (70.588717ms) +✔ conversion replaces stale offsets even when the custom hook writes no index (89.021922ms) +✔ conversion awaits custom finalization with the original receiver and final bytes (169.155138ms) +✔ conversion propagates rejection from custom finalization (59.819893ms) +✔ conversion initializes existing-format logs and optional base integrity without overwriting a store (166.155078ms) +✔ runtime environment defaults to production and ignores production fault points (1.643785ms) +✔ fault points are inert outside test environment (0.199473ms) +✔ crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (598.803859ms) +✔ crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (389.509455ms) +✔ crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (390.5932ms) +✔ crash at before-command-changeset-write preserves evidence and never automatically reruns (414.629954ms) +✔ crash at after-command-changeset-write preserves evidence and never automatically reruns (357.3841ms) +✔ crash at before-command-done-status preserves evidence and never automatically reruns (361.370497ms) +✔ crash after done status but before query update recovers committed state without replay (457.528028ms) +✔ crash ownership cannot be stolen automatically (305.466252ms) +✔ hanging command times out unsafe and later accepted command commits (344.438148ms) +✔ hanging load worker fails startup explicitly (200.290418ms) +✔ integrity-enabled command writes changeset digest before done status (323.951706ms) +✔ normal restart preserves committed state according to reconstruction oracle (430.876834ms) +✔ accepted command changeset is not treated as committed state on startup (54.887938ms) +✔ done command with missing changeset refuses recovery instead of silently using base data (44.489354ms) +✔ malformed base OD-JSONTag file refuses recovery (53.9879ms) +✔ malformed committed changeset refuses recovery (62.875247ms) +✔ truncated committed changeset payload refuses recovery before lazy parsing can hide it (57.170068ms) +✔ malformed uncommitted changeset is ignored during committed startup reconstruction (56.035862ms) +✔ durable status file selects only done command changesets for startup (74.028892ms) +✔ durable command log replays only accepted commands (11.255528ms) +✔ active command at attempt 1 remains unchanged pending administrator assessment (4.023563ms) +✔ active command at attempt 2 remains unchanged pending administrator assessment (5.108949ms) +✔ malformed durable status record refuses recovery with explicit integrity error (2.544773ms) +✔ structurally invalid durable status record refuses recovery (2.119582ms) +✔ malformed durable command log record refuses recovery with explicit integrity error (4.181299ms) +✔ structurally invalid durable command log record refuses recovery (2.684668ms) +✔ OD-JSONTag framing validation catches truncated lazy records (2.12683ms) +✔ OD-JSONTag framing validation accepts changeset skip records (0.27993ms) +✔ integrity manifest detects same-length altered base OD-JSONTag payload (72.635582ms) +✔ integrity manifest detects same-length altered committed changeset payload (85.968267ms) +✔ integrity enabled requires manifest entry for base data (55.965339ms) +✔ PL01-05 durable publication survives modeled loss of unsynced names and restart (824.722012ms) +✔ PL10 an actual short changeset write is completed before acknowledgment (345.526846ms) +✔ PL07-09/11/18-20 log sync failure stops mutation without false terminal success (238.53111ms) +✔ PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (238.050442ms) +✔ PL07-09/11/18-20 log close failure stops mutation without false terminal success (185.767947ms) +✔ PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (287.340691ms) +✔ PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (227.624115ms) +✔ PL07-09/11/18-20 done sync failure stops mutation without false terminal success (249.948147ms) +✔ PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (246.860237ms) +✔ PL16 delayed append cannot be overtaken (321.306723ms) +✔ PL16 delayed append cannot be overtaken by duplicate ID (267.407714ms) +✔ PL02-04/06 missing command-log.jsontag fails open rather than recreating history (317.57945ms) +✔ PL02-04/06 missing command-status.jsontag fails open rather than recreating history (312.701103ms) +✔ PL02-04/06 missing data.A.jsontag fails open rather than recreating history (326.792291ms) +✔ PL12/17 startup preserves uncertain active work without repeating external effects (287.859862ms) +✔ directory sync failure after rename prevents done (186.775761ms) +✔ directory close:after failure after rename prevents done (185.850535ms) +✔ changeset open error is not masked or acknowledged (202.392105ms) +✔ changeset write error is not masked or acknowledged (210.718358ms) +✔ changeset close:after error is not masked or acknowledged (185.063299ms) +✔ short command-log append preserves its entire JSONTag record (192.086459ms) +✔ complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.346819ms) +✔ serialized writer preserves order and stops after an uncertain operation (0.464406ms) +✔ durable publication replaces entire contents and concurrent append retains each record once (36.437087ms) +ℹ tests 105 +ℹ pass 105 +ℹ fail 0 +ℹ skipped 0 +ℹ todo 0 +ℹ duration_ms 6076.58365 + +Conditional/width refactor at 5d166a23b664ba9fbe34ae819feb1ed70959046c +✔ atomicity: command mutation followed by failure does not become visible or committed (770.328799ms) +✔ isolation: mutating data inside a JavaScript query does not affect later queries (351.976338ms) +✔ consistency: query during an active command sees the previous committed state (1059.750516ms) +✔ idempotency: duplicate command ID is acknowledged but not applied twice (517.018137ms) +✔ idempotency: retry during active command reports current status and does not enqueue twice (1019.480825ms) +✔ administrator recovers trailing missing/waiting commands in log order on a copy (310.417052ms) +✔ later accepted dataset blocks rerun, including zero-byte no-op (10.846552ms) +✔ later done dataset blocks rerun, including zero-byte no-op (21.031532ms) +✔ later unsafe dataset blocks rerun, including zero-byte no-op (10.38607ms) +✔ new later file makes an approved plan stale before executing a handler (27.137195ms) +✔ changed code and missing explicit approval reject apply (29.102662ms) +✔ cooperative ownership excludes competing writer and path aliases (11.733063ms) +✔ backup and restore validate saved bytes without executing commands (387.986745ms) +✔ read-only inspection never imports handlers and rejects conflicting done order (23.627113ms) +✔ historical status order never makes a later log dataset eligible for bypass (6.925995ms) +✔ interrupted recovery preserves completed attempts and blocks an unfinished candidate (249.472331ms) +✔ retained digest mismatch stops recovery before replacing integrity or appending done (94.715019ms) +✔ malformed tails and overlapping paths cannot be accepted as valid stores (6.334512ms) +✔ approved recovery resumes from completed candidate prefix without repeating external effects (314.392961ms) +✔ finishing an interrupted completion report reconstructs data without executing again (132.623324ms) +✔ multi-directory integrity paths and file permissions survive backup and restore (33.686382ms) +✔ backup excludes unrelated secrets while retaining declared custom artifacts (15.969331ms) +✔ a backup remains restorable after transfer to a different directory (69.563119ms) +✔ promotion verification rejects code changed after completed recovery (79.133752ms) +✔ verified A prefix permits only missing B, waiting C and missing D in log order (434.822874ms) +✔ restore reports accepted commands absent from an older backup (18.754409ms) +✔ identical historical duplicate retains first log position; conflicting duplicate blocks (4.386649ms) +✔ backup coverage distinguishes changed history and data from missing source data (229.215826ms) +✔ handler lifecycle: index update failure prevents command commit (666.692445ms) +✔ handler lifecycle: derived index file can remain after failed command (375.946138ms) +✔ handler lifecycle: final offset write failure prevents command commit (329.270472ms) +✔ handler lifecycle: custom finalizer rejection prevents command commit (305.101047ms) +✔ handler lifecycle: current index update can mutate canonical state before commit (332.934148ms) +✔ command edit indexes only its changed record using changeset UTF-8 offsets (20.077215ms) +✔ command addition indexes new entities and preserves interior skip numbers (10.47907ms) +✔ command offsets include mutations and new records introduced by the index hook (6.912693ms) +✔ successive command indexes are relative to their own changeset files (7.377323ms) +✔ no-op commands replace any stale sidecar with an empty offset index (6.156606ms) +✔ failure to write final offsets rejects the command before success (6.726363ms) +✔ commands await custom finalization that delegates to the default index (45.640098ms) +✔ custom finalization can replace the default without an unconditional offset write (7.550846ms) +✔ custom finalization rejection propagates from the command worker (7.120895ms) +✔ default conversion offsets match every final record (130.821568ms) +✔ conversion finalizes offsets after custom indexing changes record sizes and adds records (107.953088ms) +✔ conversion replaces stale offsets even when the custom hook writes no index (182.645488ms) +✔ conversion awaits custom finalization with the original receiver and final bytes (118.508781ms) +✔ conversion propagates rejection from custom finalization (153.766708ms) +✔ conversion initializes existing-format logs and optional base integrity without overwriting a store (164.775744ms) +✔ runtime environment defaults to production and ignores production fault points (1.444992ms) +✔ fault points are inert outside test environment (0.190374ms) +✔ crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (716.598156ms) +✔ crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (425.24224ms) +✔ crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (374.751707ms) +✔ crash at before-command-changeset-write preserves evidence and never automatically reruns (423.374956ms) +✔ crash at after-command-changeset-write preserves evidence and never automatically reruns (368.481424ms) +✔ crash at before-command-done-status preserves evidence and never automatically reruns (369.830528ms) +✔ crash after done status but before query update recovers committed state without replay (472.231201ms) +✔ crash ownership cannot be stolen automatically (309.193466ms) +✔ hanging command times out unsafe and later accepted command commits (349.149476ms) +✔ hanging load worker fails startup explicitly (195.947203ms) +✔ integrity-enabled command writes changeset digest before done status (320.734733ms) +✔ normal restart preserves committed state according to reconstruction oracle (432.46088ms) +✔ accepted command changeset is not treated as committed state on startup (78.677963ms) +✔ done command with missing changeset refuses recovery instead of silently using base data (64.739072ms) +✔ malformed base OD-JSONTag file refuses recovery (114.82201ms) +✔ malformed committed changeset refuses recovery (195.38594ms) +✔ truncated committed changeset payload refuses recovery before lazy parsing can hide it (70.943855ms) +✔ malformed uncommitted changeset is ignored during committed startup reconstruction (111.229727ms) +✔ durable status file selects only done command changesets for startup (84.72398ms) +✔ durable command log replays only accepted commands (4.639563ms) +✔ active command at attempt 1 remains unchanged pending administrator assessment (7.631507ms) +✔ active command at attempt 2 remains unchanged pending administrator assessment (2.233418ms) +✔ malformed durable status record refuses recovery with explicit integrity error (2.672729ms) +✔ structurally invalid durable status record refuses recovery (2.157557ms) +✔ malformed durable command log record refuses recovery with explicit integrity error (1.246495ms) +✔ structurally invalid durable command log record refuses recovery (1.942975ms) +✔ OD-JSONTag framing validation catches truncated lazy records (0.331445ms) +✔ OD-JSONTag framing validation accepts changeset skip records (0.187469ms) +✔ integrity manifest detects same-length altered base OD-JSONTag payload (61.989542ms) +✔ integrity manifest detects same-length altered committed changeset payload (55.858695ms) +✔ integrity enabled requires manifest entry for base data (73.109183ms) +✔ PL01-05 durable publication survives modeled loss of unsynced names and restart (901.363066ms) +✔ PL10 an actual short changeset write is completed before acknowledgment (377.270416ms) +✔ PL07-09/11/18-20 log sync failure stops mutation without false terminal success (249.279553ms) +✔ PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (243.886641ms) +✔ PL07-09/11/18-20 log close failure stops mutation without false terminal success (204.041125ms) +✔ PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (252.47621ms) +✔ PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (253.389402ms) +✔ PL07-09/11/18-20 done sync failure stops mutation without false terminal success (233.990209ms) +✔ PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (247.363839ms) +✔ PL16 delayed append cannot be overtaken (330.834357ms) +✔ PL16 delayed append cannot be overtaken by duplicate ID (264.849904ms) +✔ PL02-04/06 missing command-log.jsontag fails open rather than recreating history (286.264648ms) +✔ PL02-04/06 missing command-status.jsontag fails open rather than recreating history (315.299591ms) +✔ PL02-04/06 missing data.A.jsontag fails open rather than recreating history (315.321485ms) +✔ PL12/17 startup preserves uncertain active work without repeating external effects (308.849112ms) +✔ directory sync failure after rename prevents done (192.103111ms) +✔ directory close:after failure after rename prevents done (189.438443ms) +✔ changeset open error is not masked or acknowledged (191.130994ms) +✔ changeset write error is not masked or acknowledged (210.408614ms) +✔ changeset close:after error is not masked or acknowledged (214.214365ms) +✔ short command-log append preserves its entire JSONTag record (204.016215ms) +✔ complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.265931ms) +✔ serialized writer preserves order and stops after an uncertain operation (0.736876ms) +✔ durable publication replaces entire contents and concurrent append retains each record once (41.178696ms) +ℹ tests 105 +ℹ pass 105 +ℹ fail 0 +ℹ skipped 0 +ℹ todo 0 +ℹ duration_ms 6300.329225 + +Example conversion correction: d4136eaf7b0014726a1c90a0d0b8488ece2d6b27 +✔ atomicity: command mutation followed by failure does not become visible or committed (530.649449ms) +✔ isolation: mutating data inside a JavaScript query does not affect later queries (421.252932ms) +✔ consistency: query during an active command sees the previous committed state (1070.76709ms) +✔ idempotency: duplicate command ID is acknowledged but not applied twice (534.054773ms) +✔ idempotency: retry during active command reports current status and does not enqueue twice (1017.610332ms) +task time 4 +task time 4 +✔ administrator recovers trailing missing/waiting commands in log order on a copy (234.338869ms) +✔ later accepted dataset blocks rerun, including zero-byte no-op (10.913853ms) +✔ later done dataset blocks rerun, including zero-byte no-op (30.629165ms) +✔ later unsafe dataset blocks rerun, including zero-byte no-op (15.335214ms) +✔ new later file makes an approved plan stale before executing a handler (24.505838ms) +✔ changed code and missing explicit approval reject apply (29.925745ms) +✔ cooperative ownership excludes competing writer and path aliases (6.074581ms) +task time 4 +✔ backup and restore validate saved bytes without executing commands (212.675254ms) +✔ read-only inspection never imports handlers and rejects conflicting done order (8.706261ms) +✔ historical status order never makes a later log dataset eligible for bypass (5.538214ms) +task time 4 +task error Error: external effect already happened + at Object.addPerson (file:///tmp/simplystore-crash-OXCuMb/commands.mjs:1:156) + at runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:116:38) + at MessagePort. (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker.mjs:8:24) +✔ interrupted recovery preserves completed attempts and blocks an unfinished candidate (201.539075ms) +task time 8 +✔ retained digest mismatch stops recovery before replacing integrity or appending done (176.973657ms) +✔ malformed tails and overlapping paths cannot be accepted as valid stores (9.07078ms) +task time 6 +task error Error: connection lost after effect + at Object.addPerson (file:///tmp/simplystore-crash-PgoTAx/commands.mjs:5:33) + at runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:116:38) + at MessagePort. (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker.mjs:8:24) +task time 8 +✔ approved recovery resumes from completed candidate prefix without repeating external effects (310.420671ms) +task time 5 +✔ finishing an interrupted completion report reconstructs data without executing again (127.255539ms) +✔ multi-directory integrity paths and file permissions survive backup and restore (54.723634ms) +✔ backup excludes unrelated secrets while retaining declared custom artifacts (33.453306ms) +✔ a backup remains restorable after transfer to a different directory (21.985454ms) +task time 4 +✔ promotion verification rejects code changed after completed recovery (94.632834ms) +task time 8 +task time 3 +task time 3 +task time 3 +task time 3 +task time 5 +task time 3 +✔ verified A prefix permits only missing B, waiting C and missing D in log order (452.363458ms) +✔ restore reports accepted commands absent from an older backup (19.53747ms) +✔ identical historical duplicate retains first log position; conflicting duplicate blocks (4.600562ms) +task time 4 +✔ backup coverage distinguishes changed history and data from missing source data (195.325248ms) +✔ handler lifecycle: index update failure prevents command commit (495.977502ms) +✔ handler lifecycle: derived index file can remain after failed command (421.432538ms) +✔ handler lifecycle: final offset write failure prevents command commit (308.179751ms) +✔ handler lifecycle: custom finalizer rejection prevents command commit (302.869268ms) +✔ handler lifecycle: current index update can mutate canonical state before commit (330.828754ms) +task time 4 +✔ command edit indexes only its changed record using changeset UTF-8 offsets (19.576993ms) +task time 5 +✔ command addition indexes new entities and preserves interior skip numbers (8.863262ms) +task time 2 +✔ command offsets include mutations and new records introduced by the index hook (6.888152ms) +task time 1 +task time 2 +✔ successive command indexes are relative to their own changeset files (8.108558ms) +task time 1 +✔ no-op commands replace any stale sidecar with an empty offset index (5.530991ms) +task error Error: EISDIR: illegal operation on a directory, rename '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json.5bd60a71-efd9-4f0a-a14e-a6ecabd87119.tmp' -> '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json' + at async Object.rename (node:internal/fs/promises:787:10) + at async publishFile (file:///home/auke/git/slonl/spiral/simplystore/src/storage.mjs:137:9) + at async Object.writeSerialized (file:///home/auke/git/slonl/spiral/simplystore/src/index.offset.mjs:36:3) + at async runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:143:13) + at async waitForActual (node:assert:644:5) + at async strict.rejects (node:assert:767:25) + at async TestContext. (file:///home/auke/git/slonl/spiral/simplystore/test/command-offsets.mjs:131:2) + at async Test.run (node:internal/test_runner/test:1113:7) + at async Test.processPendingSubtests (node:internal/test_runner/test:788:7) { + errno: -21, + code: 'EISDIR', + syscall: 'rename', + path: '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json.5bd60a71-efd9-4f0a-a14e-a6ecabd87119.tmp', + dest: '/tmp/simplystore-command-offset-ZWilwl/index.offset.blocked.json', + storageFailure: true +} +✔ failure to write final offsets rejects the command before success (5.732417ms) +task time 24 +✔ commands await custom finalization that delegates to the default index (28.569013ms) +task time 4 +✔ custom finalization can replace the default without an unconditional offset write (14.88495ms) +task error Error: custom finalization failed + at Object.finalize (file:///tmp/simplystore-command-offset-KrXVTr/index.mjs:1:55) + at finalizeIndex (file:///home/auke/git/slonl/spiral/simplystore/src/index.mjs:26:18) + at runCommand (file:///home/auke/git/slonl/spiral/simplystore/src/command-worker-module.mjs:143:19) + at async waitForActual (node:assert:644:5) + at async strict.rejects (node:assert:767:25) + at async TestContext. (file:///home/auke/git/slonl/spiral/simplystore/test/command-offsets.mjs:169:2) + at async Test.run (node:internal/test_runner/test:1113:7) + at async Test.processPendingSubtests (node:internal/test_runner/test:788:7) +✔ custom finalization rejection propagates from the command worker (15.605834ms) +✔ conversion supports the example dataset and its URL values (319.615459ms) +✔ default conversion offsets match every final record (99.858631ms) +✔ conversion finalizes offsets after custom indexing changes record sizes and adds records (82.948927ms) +✔ conversion replaces stale offsets even when the custom hook writes no index (99.469857ms) +✔ conversion awaits custom finalization with the original receiver and final bytes (140.26764ms) +✔ conversion propagates rejection from custom finalization (90.291458ms) +✔ conversion initializes existing-format logs and optional base integrity without overwriting a store (140.787504ms) +✔ runtime environment defaults to production and ignores production fault points (1.70791ms) +✔ fault points are inert outside test environment (0.239437ms) +✔ crash at after-command-log-before-accepted-status preserves evidence and never automatically reruns (543.472158ms) +✔ crash at after-command-accepted-status-before-response preserves evidence and never automatically reruns (466.784076ms) +✔ crash at after-active-status-before-command-worker preserves evidence and never automatically reruns (397.562307ms) +✔ crash at before-command-changeset-write preserves evidence and never automatically reruns (415.927966ms) +✔ crash at after-command-changeset-write preserves evidence and never automatically reruns (358.284123ms) +✔ crash at before-command-done-status preserves evidence and never automatically reruns (349.615417ms) +✔ crash after done status but before query update recovers committed state without replay (453.032633ms) +✔ crash ownership cannot be stolen automatically (304.197048ms) +✔ hanging command times out unsafe and later accepted command commits (353.503122ms) +✔ hanging load worker fails startup explicitly (200.604231ms) +✔ integrity-enabled command writes changeset digest before done status (326.050121ms) +✔ normal restart preserves committed state according to reconstruction oracle (450.499881ms) +✔ accepted command changeset is not treated as committed state on startup (89.709014ms) +✔ done command with missing changeset refuses recovery instead of silently using base data (74.28066ms) +✔ malformed base OD-JSONTag file refuses recovery (87.926958ms) +✔ malformed committed changeset refuses recovery (116.383408ms) +✔ truncated committed changeset payload refuses recovery before lazy parsing can hide it (69.881388ms) +✔ malformed uncommitted changeset is ignored during committed startup reconstruction (52.813884ms) +✔ durable status file selects only done command changesets for startup (63.021015ms) +✔ durable command log replays only accepted commands (4.553718ms) +✔ active command at attempt 1 remains unchanged pending administrator assessment (4.411881ms) +✔ active command at attempt 2 remains unchanged pending administrator assessment (2.707677ms) +✔ malformed durable status record refuses recovery with explicit integrity error (8.38551ms) +✔ structurally invalid durable status record refuses recovery (2.233453ms) +✔ malformed durable command log record refuses recovery with explicit integrity error (2.417322ms) +✔ structurally invalid durable command log record refuses recovery (3.238178ms) +✔ OD-JSONTag framing validation catches truncated lazy records (0.366872ms) +✔ OD-JSONTag framing validation accepts changeset skip records (0.473898ms) +✔ integrity manifest detects same-length altered base OD-JSONTag payload (76.421226ms) +✔ integrity manifest detects same-length altered committed changeset payload (75.768513ms) +✔ integrity enabled requires manifest entry for base data (72.482922ms) +✔ PL01-05 durable publication survives modeled loss of unsynced names and restart (925.400434ms) +✔ PL10 an actual short changeset write is completed before acknowledgment (337.850245ms) +✔ PL07-09/11/18-20 log sync failure stops mutation without false terminal success (241.49423ms) +✔ PL07-09/11/18-20 acceptance sync failure stops mutation without false terminal success (208.023773ms) +✔ PL07-09/11/18-20 log close failure stops mutation without false terminal success (204.181151ms) +✔ PL07-09/11/18-20 changeset sync failure stops mutation without false terminal success (272.554329ms) +✔ PL07-09/11/18-20 changeset rename failure stops mutation without false terminal success (235.896548ms) +✔ PL07-09/11/18-20 done sync failure stops mutation without false terminal success (256.456294ms) +✔ PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip (267.047535ms) +✔ PL16 delayed append cannot be overtaken (336.666777ms) +✔ PL16 delayed append cannot be overtaken by duplicate ID (267.117791ms) +✔ PL02-04/06 missing command-log.jsontag fails open rather than recreating history (323.819448ms) +✔ PL02-04/06 missing command-status.jsontag fails open rather than recreating history (336.522485ms) +✔ PL02-04/06 missing data.A.jsontag fails open rather than recreating history (330.718406ms) +✔ PL12/17 startup preserves uncertain active work without repeating external effects (285.553404ms) +✔ directory sync failure after rename prevents done (211.426983ms) +✔ directory close:after failure after rename prevents done (213.647192ms) +✔ changeset open error is not masked or acknowledged (212.155863ms) +✔ changeset write error is not masked or acknowledged (188.578415ms) +✔ changeset close:after error is not masked or acknowledged (191.1675ms) +✔ short command-log append preserves its entire JSONTag record (237.012495ms) +✔ complete-write loop preserves bytes through repeated short writes and rejects zero progress (2.216848ms) +✔ serialized writer preserves order and stops after an uncertain operation (0.450738ms) +✔ durable publication replaces entire contents and concurrent append retains each record once (33.8786ms) +ℹ tests 106 +ℹ suites 0 +ℹ pass 106 +ℹ fail 0 +ℹ cancelled 0 +ℹ skipped 0 +ℹ todo 0 +ℹ duration_ms 6379.304191 diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.md b/.spiral/evidence/EVD-20260919-TTZ7C-27.md new file mode 100644 index 0000000..9a521b2 --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.md @@ -0,0 +1,217 @@ +# Evidence: Power-Loss Durability And Administrative Recovery + +Evaluation of IMP-20260919-TTZ7C-26 at +`13db3bf55926cf3a3a1b2769c422881a875193c8`. Runtime and VM harness are unchanged +from `049c45642fe60146bd511bb87abb4b032f8df36a`; the later commit adds three +specific administrator acceptance examples. The VM captured the latter runtime +revision with an empty tracked worktree diff. Human cycle acceptance is pending. + +Machine-readable environment, observations, recovered data and timings are in +[EVD-27-results.json](EVD-20260919-TTZ7C-27-results.json); individual test outcomes +are retained in [EVD-27-tests.txt](EVD-20260919-TTZ7C-27-tests.txt). + +## Automated regression and policy evidence + +```sh +node --test test/durability-startup.mjs test/conversion-offsets.mjs \ + test/command-offsets.mjs test/acid-baseline.mjs test/durability-crash.mjs \ + test/after-change-handlers.mjs test/storage.mjs test/admin-recovery.mjs \ + test/power-loss-baseline.mjs +node --test --test-isolation=none test/admin-recovery.mjs +node --test test/durability-latency.mjs +``` + +The complete relevant suite passed 101/101, zero failures/skips/TODOs. After three +additional acceptance examples, the administrator suite passed 22/22. This covers +104 distinct tests across those runs; 19 administrator tests were rerun. The +latency test passed separately. Tests use disposable stores and bounded localhost +listeners. Conversion/inspect/backup/restore/verify also passed a direct CLI smoke +exercise with an integrity-enabled store. Python harness syntax parsing passed. + +The earlier baseline remains immutable in EVD-23. PL15's request-retention +expectation was explicitly corrected to request removal with complete logged +JSONTag inputs; PL16's old waiting-queue-preservation expectation was corrected to +command-log authority. Other changes replace unsafe automatic-replay assertions +with byte-preserving refusal, explicit assessment and guarded recovery evidence. +A failed sync can leave visible done bytes: tests assert no false public success +and conservative termination, not that failed sync proves rollback. + +| Requirement | Evidence and disposition | +|---|---| +| P1 acceptance | Complete short appends; command/accepted sync and close failures; serialized concurrent A/B/duplicate acceptance; ext4 cut after observed 202 retains accepted A | +| P2 completion | Changeset file-sync/rename/directory-sync trace precedes done; default offset/conversion/integrity regressions; ext4 cut retains all five completed transitions | +| P3 I/O uncertainty | Short/zero/invalid writes; open/write/ENOSPC/file sync/rename/directory sync/close injection; poisoned writer and stopped mutation, preserved uncertainty | +| P4 independent oracle | Client commands/status witnesses outside damaged stores; namespace-loss controls; host serial observer expects A/L0/L1/L2/L3 independently of recovered logs | +| P5 inspection | No user module import or source-status mutation; framing/tails, missing files, conflicting IDs/order, integrity and existing uncertain data diagnosed | +| P6 ordered rerun | Later accepted/done/unsafe zero-byte data blocks; contrary status order cannot bypass guard; verified A prefix selects only missing B, waiting C, missing D and recovers A/B/C/D | +| P7 no bypass | Crash locks retained, alias exclusion, pending startup refusal without rerun, source/code/later-file changes rejected, partial recovery requires new assessment | +| P8 inputs | Configured worker cannot access request; third handler argument undefined and fourth metadata retained; nested/date/Unicode JSONTag body survives; exact original command bytes retained in recovered copy | +| P9 backup | Consistent integrity-enabled multi-directory copy, permissions and portable backup; no command execution on restore; changed backup rejected; older-backup comparison reports missing accepted command | +| P10 evidence/promotion | Original source/log unchanged, external authorization and attempt records; completed prefix never rerun on resume; finish missing report without execution; changed candidate/code fails verification; CLI emits activation preview only | +| P11 filesystem | Four disposable ext4 VM cut/reboot scenarios plus independent volatile-device negative control; exact envelope and timing recorded below | +| P12 rebuild | Separate new-store selected-code/history and cooperative effect-suppression contract retained in DES-25 and administrator guide; no ordinary recovery override or implemented rebuild claim | + +## Filesystem experiment + +Run the command in `docs/power-loss-filesystem.md` with the packaged harness. +This evaluation used QEMU 10.2.1, Linux 7.0.0-31-generic, Node v24.7.0, ext4 +`data=ordered,barrier=1`, virtio-blk with QEMU `cache=writeback`, and a purpose-built +volatile NBD backend honoring FLUSH/FUA. TCG used two CPUs and 1024 MiB RAM; no +KVM or host block-device writes. QEMU/kernel files were extracted under `/tmp`, +without installing host packages. Exact kernel digest and observed mounts/cache +are retained in the JSON evidence. + +The host kills the whole guest on an observed boundary, discards volatile block +writes, then boots the same surviving image for ext4 recovery and engine +inspection. Results: + +- Done: five completed commands A/L0/L1/L2/L3 survive in order; ready=true. +- Accepted: A's complete accepted record survives; no dataset; ready=false. +- Active: A remains active with no dataset; ready=false; inspection executes no handler. +- Unsynced filesystem control: an ordinary unsynced file disappears after the cut. +- Independent device control: flushed A persists, unflushed B is lost (one dirty page discarded). + +At the engine boundaries no dirty device pages remained: completed flushes had +drained them. The separate device control verifies that the backend really loses +unflushed writes; the filesystem control demonstrates guest-cache loss. The +test does not fabricate deletion of acknowledged synced data. Output images, +initramfs and full serial logs were retained locally under +`/tmp/simplystore-vm-acceptance`; compact evidence is committed here, not images. + +## Latency and practical bounds + +On the ext4/TCG guest, the first cold command completed in 2520 ms. Four subsequent +commands accepted in 31.1–44.7 ms and completed in 755.6–872.9 ms. These include +emulation, fresh workers and polling and are too few for throughput or percentile +claims. Flush correctness, not production performance, is this experiment's aim. + +A separate 30-command localhost/tmpfs sample measured startup 127.2 ms; +acceptance p50/p95 0.98/2.03 ms; completion p50/p95 39.88/44.08 ms. Maximums were +35.39/79.75 ms respectively. These are tmpfs timings, not disk measurements or a +before/after performance comparison. A deployment should measure its own storage. + +Evidence is bounded: no physical power removal, random exhaustive campaign, +remote-filesystem promise, lying-device tolerance, or universal custom-hook +certification. The preload instruments the newly used async paths; synchronous +index helpers have functional/conversion coverage, not exhaustive injected faults +at every native call. Interrupted recovery boundaries combine real worker effects +with explicit fixtures; deployment promotion is documented and previewed, not +performed against a service manager. These are test limits, not stronger claims. + +No canonical format change or migration is required. Pending/uncertain historical +work deliberately reduces automatic availability. Surviving files cannot prove +complete old history, original code/dependencies, missing business inputs or +external effects. Missing/corrupt unrecoverable state may require a disclosed +backup rollback. Full rebuild tooling and wider stress/platform testing remain +follow-up work within their separately recorded boundaries. + +ESLint passed the new modules, changed server/worker/recovery/converter paths, +harness and revised tests. Full changed-file lint also exposes two pre-existing +undefined `meta` references in the untouched `load` methods of `index.id.mjs` and +`index.offset.mjs`, confirmed identical on master; this cycle changes their write +paths only. This is not represented as a clean repository-wide lint result. + +## Braces correction during review + +After the maintainer's style instruction recorded at +`80a1633e1791ed33eb644439bdea511578de21dd`, checked all 26 JavaScript modules +changed in this cycle with ESLint's `curly: ["error", "all"]` rule. Nineteen +files needed braces or accompanying block spacing; the check reports zero brace +violations. Added the rule to ESLint configuration and retained the preference +in AGENTS.md. Unrelated legacy files were not reformatted. + +Parsed all 26 before/after files with Espree and compared their syntax trees, +ignoring source positions, literal raw formatting and single-statement blocks +in control-flow bodies. All trees match after that normalization. Embedded fixture +source strings were unchanged. This establishes that the edit adds control-flow +blocks/spacing without changing the contained expressions, statements or branches. +The runtime regression and VM evidence above remains tied to its exact original +revision; no new power-cut experiment is claimed for this formatting correction. + +The subsequent block-layout clarification expands all statement/function blocks +in the same 26 JavaScript files onto multiple lines, including empty blocks and +callbacks. ESLint uses `brace-style: ["error", "stroustrup", {allowSingleLine: +false}]` alongside `curly`; repository instructions also explicitly cover empty +blocks. `else`, `catch` and `finally` follow the closing-brace line. Existing +file indentation conventions are retained and trailing whitespace removed. + +Compared the resulting Espree syntax trees with the files at +`941a96ee4158034e611018a03839d2b1e5c0c861`, ignoring only source positions and +literal raw spelling. All 26 trees are identical; 728 block statements were +checked, with zero inline blocks. Curly/brace-style/indentation/trailing-whitespace +checks pass. No runtime tests or VM experiments were repeated for this verified +whitespace-only adjustment. + +## JAQT readability verification + +Verified IMP-26 at `49fa48c123d0035df087b7a9d4c1460da92b85fa` after the maintainer +approved the JAQT example and requested review of the remaining code. Changes +are limited to startup diagnostics, store inspection, administrator recovery and +backup coverage, plus a focused regression test. The review retained simple +array operations and ordered I/O loops where JAQT added no clarity. + +The complete relevant suite listed above passed **105/105**, with zero failures, +skips or TODOs. This includes the earlier recovery/order/short-write/fault cases, +plus a new backup-coverage case comparing unchanged backup state against changed +command input, status, data and base bytes, and missing source data. Structured +clone assertions confirm that inspection command lists, recovery plans and restore +reports remain ordinary data, with no exposed JAQT proxies. Startup diagnostics +were also compared against the earlier expression across 50 representative inputs +during the preceding read-only proposal; messages and ordering matched. + +ESLint passes all three changed production modules and the changed test file; +Git whitespace checks pass. No persistence protocol changed, and no new VM +power-cut run is claimed. The recorded filesystem experiment still refers to its +exact original runtime revision. Cycle acceptance remains pending. + +## Conditional-expression and line-width verification + +Verified IMP-26 at `5d166a23b664ba9fbe34ae819feb1ed70959046c` after the maintainer +requested fewer ternaries and lines about 80 characters wide, with readability +exceptions. The cycle's 26 JavaScript modules now contain ten clear, short +conditional expressions, down from 36. Nested choices, optional fields and +parsing/conversion steps use explicit branches. Formatting also covers ESLint +configuration. The line-width rule is a warning with string/template/regex/URL +exceptions; it is not a hard prohibition that overrides readability. + +The formatting pass compared all 27 syntax trees before and after wrapping; +only source positions and literal raw spelling were ignored, and all matched. +Conditional rewrites were verified separately: the complete relevant suite passed +**105/105**, with no failures, skips or TODOs. A direct CLI smoke exercise passed +conversion, inspection, explicitly approved A recovery, backup, restore both with +and without source comparison, and promotion verification. Recovered command +coverage was A. Sixteen optional integrity/schema configurations produce identical +store paths/options to the previous implementation. All four VM scenario handler +modules generated by the rewritten branch selection are byte-identical to the +previous nested ternary's output. No new VM power-cut experiment is claimed. + +All 26 modules plus ESLint configuration pass brace and 80-column policy checks +with zero errors or warnings under the declared literal exceptions. Full lint +finds only the same two historical undefined-meta errors in index load methods, +now at their reformatted line positions. No dependency or lockfile changed; the +pinned formatter was installed only in /tmp. Existing newline block preferences, +canonical file formats and cycle acceptance status are preserved. + +## Example build and URL conversion correction — 2026-09-19 + +Verified implementation `d4136eaf7b0014726a1c90a0d0b8488ece2d6b27`: + +- The actual bundled Star Wars input failed the old converter with a URL syntax + error. Lazy OD parsing initially returned a root, but indexing forced the + invalid empty-base URL validation. Using the default base permits conversion. +- `test/conversion-offsets.mjs`: 7/7 passing, including actual example conversion, + every final byte offset, typed URLs and linked homeworld data. +- Full relevant suite: 106/106 passing, zero skips/TODOs. Same nine test files as + the preceding review run; output appended to the companion tests file. +- Disposable example copy: `npm run build` created data, indexes and empty logs, + released ownership, and refused a second build. `npm start` then served an HTTP + query returning Luke Skywalker. Only the listening port was changed in the copy + to use an available local port. Existing installed dependencies were reused; + no fresh network installation was tested. +- ESLint passes for the changed converter and converter tests. +- The user's existing failed-attempt example lock and `package.json~` are retained. + The recorded PID is absent and no generated example store files were present; + this observation is not treated as automatic lock-release authorization. + +No new VM power-cut exercise is claimed: storage protocol and formats are +unchanged. Human cycle acceptance remains pending. diff --git a/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl new file mode 100644 index 0000000..a16cc0d --- /dev/null +++ b/.spiral/evidence/EVD-20260919-TTZ7C-27.ttl @@ -0,0 +1,20 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260919-TTZ7C-27 a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260919-TTZ7C-27" ; + sd:repositoryPath ".spiral/evidence/EVD-20260919-TTZ7C-27.md" ; + sd:status sd:Active ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "13db3bf55926cf3a3a1b2769c422881a875193c8" ] ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "49fa48c123d0035df087b7a9d4c1460da92b85fa" ] ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "5d166a23b664ba9fbe34ae819feb1ed70959046c" ] ; + sd:verifies [ a sd:ArtifactReference ; + sd:artifact project:IMP-20260919-TTZ7C-26 ; + sd:gitCommit "d4136eaf7b0014726a1c90a0d0b8488ece2d6b27" ] . diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-22.md b/.spiral/implementations/IMP-20260919-TTZ7C-22.md new file mode 100644 index 0000000..abc85bf --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-22.md @@ -0,0 +1,30 @@ +# Implementation: Independent Power-Loss Baseline + +Implements DES-20260919-TTZ7C-21 as a new isolated test suite. No `src/` or +production script behavior changes are included. + +- `test/power-loss-preload.mjs`: explicitly enabled, store-scoped I/O tracing and + faults, inherited by disposable server workers; synced-byte snapshots and + namespace publication observations live outside the failed storage domain. +- `test/power-loss-helpers.mjs`: external client observations, bounded waits, + abrupt shutdown, copied crash images, trace-constrained filename loss, and + isolated expected-failure invariant reporting. +- `test/power-loss-baseline.mjs`: 21 scenarios covering normal persistence, + missing publications, corruption, I/O errors and short writes, effects on + replay, later-dataset guards, input completeness, concurrent order, and a + positive directory-sync model control. +- `package.json`: `test:power-loss` and `test:power-loss:strict` scripts. Strict + mode exposes requirement failures; normal baseline uses explicit nested TODO + assertions while keeping harness/control failures fatal. +- `docs/power-loss-baseline.md`: fault matrix, commands, current observations, + independence limits, and remaining cycle work. + +The test-only custom finalizer in PL21 supplies a directory sync as a positive +control; it is not a production fix. Policy fixtures and torn-record corruption +are explicitly distinguished from permitted unsynced-data/name-loss models. +The partial-write injection actually writes a shorter buffer through the current +dependency; it does not merely fabricate an error return. + +This new suite adds a separate evidence mechanism without changing the existing +durability helper or runtime implementation units. Verification results belong +in EVD-20260919-TTZ7C-23 after this implementation has a committed version. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-22.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-22.ttl new file mode 100644 index 0000000..e34c9ff --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-22.ttl @@ -0,0 +1,17 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260919-TTZ7C-22 a sd:Implementation ; + dcterms:identifier "IMP-20260919-TTZ7C-22" ; + sd:repositoryPath ".spiral/implementations/IMP-20260919-TTZ7C-22.md" ; + sd:status sd:Active ; + sd:implementationLocation "test/power-loss-preload.mjs" ; + sd:implementationLocation "test/power-loss-helpers.mjs" ; + sd:implementationLocation "test/power-loss-baseline.mjs" ; + sd:implementationLocation "package.json#scripts.test:power-loss" ; + sd:implementationLocation "package.json#scripts.test:power-loss:strict" ; + sd:implementationLocation "docs/power-loss-baseline.md" ; + sd:implements [ a sd:ArtifactReference ; + sd:artifact project:DES-20260919-TTZ7C-21 ; + sd:gitCommit "88fbfadffe40474590b777cd735996314a8b59c0" ] . diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md new file mode 100644 index 0000000..025b556 --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -0,0 +1,121 @@ +# Implementation: Durable Ordered Writes And Administrative Recovery + +Implements the corrected DES-24/25 within existing canonical formats, following +the maintainer's authorization to complete CYC-17. It transforms the effective +server/worker, converter/index and baseline-test implementations identified in +the companion Turtle. Existing timeout containment, awaited hooks, final offsets, +framing and optional integrity semantics remain; automatic uncertain replay and +transport-derived command inputs are deliberately removed. + +`src/storage.mjs` centralizes complete writes, durable publication/appends, file +and directory barriers, permission preservation, and first-error propagation. +Default indexes retain synchronous hook compatibility with equivalent barriers. +Conversion finishes a new base, indexes, empty logs and optional integrity under +ownership, refusing overwrite. `requiredFiles` explicitly includes custom +recovery dependencies in startup checks, completion syncing and copied inventory. + +`src/server.mjs` serializes acceptance through durable command/status writes and +queue insertion, then executes sequentially in log order. It passes the exact +logged command to workers, with no request object. Awaited handler/index work, +required-file barriers and done persistence precede query publication. Uncertain +storage failures stop mutation. `src/execute-worker.mjs` retains worker timeout +containment. Healthy shutdown drains accepted work and releases ownership. + +`src/store-inspection.mjs` validates canonical history and reconstructs its verified +prefix without importing user modules. It distinguishes missing, corrupt and +uncommitted data, records ever-accepted history, and checks later datasets in log +order. `src/store-ownership.mjs` excludes cooperating writers using canonical +paths and durable owner records; crash locks are never automatically stolen. +Normal startup refuses missing canonical files and pending/uncertain work without +rewriting statuses or rerunning handlers. + +`src/admin-recovery.mjs` and `scripts/recover.mjs` implement inspect, preview, +explicitly approved suffix recovery into a new candidate, preserved original +copies and external audit, fresh source/code validation, completion without +reexecution after an interrupted report, consistent backup/restore, offline +unlock and promotion verification. Existing status schemas/IDs are retained. +A later ever-accepted dataset blocks earlier rerun even when empty/corrupt/unsafe. +Backup copy preserves relative artifact paths and permissions, excludes unrelated +secrets, and remains portable. Restore reports known coverage and uncertainty. +Code and undeclared dependencies retain the documented administrator obligations. + +Tests cover injected short writes/errors/barrier ordering, authoritative concurrent +acceptance, JSONTag command inputs without request context, process-crash policy, +read-only inspection, guards, staleness, interruption/effect witnesses, retained +digests, portable backups and code-aware promotion verification. PL15 and PL16 +expectations change only as explicitly corrected by the maintainer; historical +baseline evidence remains at EVD-23. New tests have no TODO exceptions. + +`scripts/power-loss-vm/` packages a disposable Linux/ext4/QEMU exercise with a +volatile NBD cache, flush/FUA persistence, independent serial observer, device +and filesystem negative controls, accepted/active/done cuts and commit timings. +It never touches a host block device. Results belong in EVD-27 after this +implementation has an exact committed revision. + +Operational documentation is in `DURABILITY.md`, `docs/recovery.md`, and +`docs/power-loss-filesystem.md`. Full rebuild remains contract-only. Physical +hardware certification, exhaustive faults, arbitrary effect rollback, deployment, +release and integration are outside this implementation. + +The final acceptance examples additionally exercise exactly the verified A / missing +B / waiting C / missing D suffix, disclosed loss from an older backup, and +identical versus conflicting historical duplicate IDs. These add verification +coverage without changing the committed runtime or VM exercise. + +## JAQT readability refactor during review + +The maintainer approved using JAQT and named intermediate results where they +clarify the cycle code. Reviewed source/scripts for data-selection expressions; +the useful changes are in server startup diagnostics, inspection's later-dataset +and readiness queries, and administrator recovery/backup comparisons. Existing +simple array operations, sequential I/O and mutation loops remain direct code. + +JAQT 0.10.6 is already a dependency. Query patterns operate on inspected command +records with established string statuses/IDs and boolean flags; truthy diagnostic +fields use Boolean predicates. Query order remains command-log order. Public +plan/report collections and later-dataset IDs are materialized as ordinary arrays +so JAQT proxies do not change cloning or public data contracts. Optional report +properties keep their prior undefined/omitted behavior. + +The backup-coverage predicate now names command/status matching separately from +surviving-byte comparison. Named stages also replace dense canonical-path and +code-hash expressions where JAQT would not improve the operation. Tests cover +backup differences and unchanged plain-data output alongside the existing suite. +This is a behavior-preserving refactor; persistence boundaries, formats, recovery +eligibility and administrator authority are unchanged. + +## Conditional expressions and 80-column review + +Expanded optional configuration fields, recovery metadata loading, CLI argument +handling, index buffer conversion and nested fault/VM fixture choices into named +steps and explicit branches. Retained ten short conditional expressions where a +simple two-value choice is clearer; the reviewed files originally contained 36. +Wrapped the cycle's 26 JavaScript files toward 80 columns while preserving the +maintainer's multiline braces and each file's indentation convention. Literal +contents remain intact when splitting would reduce readability. ESLint now warns +on line width with explicit literal/URL exceptions; ternaries remain a review +judgment rather than a blanket prohibited construct. + +Formatting used pinned Prettier 3.6.2 installed only under /tmp, followed by the +existing brace rules; no package dependency or lockfile changed. The formatting +step was checked with syntax-tree comparison. Conditional rewrites additionally +receive the relevant regression and CLI verification recorded in EVD-27. Storage +barriers, canonical formats and recovery policy are unchanged. + +## Example initialization correction + +The example now exposes `npm run build`, invoking the existing converter to +create the base, indexes and empty logs before startup. Its README documents +that one-time initialization and overwrite refusal, and uses the HTTP URL served +by the example. No import script or full command rebuild feature is added. + +The converter passed an empty base URL to the mutable OD-JSONTag parser. When +indexing forces lazy parsing of tagged URLs in the Star Wars dataset, URL +validation fails even for absolute URLs. Use the parser's default valid URL base +while preserving mutable indexing. Report caught conversion errors with a nonzero +exit status so thrown parser diagnostics remain visible. Failed conversions still +retain ownership for assessment; successful initialization releases it. + +Regression coverage converts the actual bundled dataset, verifies all final +record offsets and reads typed URL values, including a linked homeworld. No +canonical format, storage barrier or existing-store overwrite policy changes. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl new file mode 100644 index 0000000..718ba4f --- /dev/null +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -0,0 +1,38 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260919-TTZ7C-26 a sd:Implementation ; + dcterms:identifier "IMP-20260919-TTZ7C-26" ; + sd:repositoryPath ".spiral/implementations/IMP-20260919-TTZ7C-26.md" ; + sd:status sd:Active ; + sd:implementationLocation "src/storage.mjs" ; + sd:implementationLocation "src/server.mjs" ; + sd:implementationLocation "src/command-worker-module.mjs" ; + sd:implementationLocation "src/execute-worker.mjs" ; + sd:implementationLocation "src/store-inspection.mjs" ; + sd:implementationLocation "src/store-ownership.mjs" ; + sd:implementationLocation "src/admin-recovery.mjs" ; + sd:implementationLocation "src/index.offset.mjs" ; + sd:implementationLocation "src/index.id.mjs" ; + sd:implementationLocation "src/recovery.mjs" ; + sd:implementationLocation "scripts/convert.mjs" ; + sd:implementationLocation "example/package.json" ; + sd:implementationLocation "example/README.md" ; + sd:implementationLocation "test/conversion-offsets.mjs" ; + sd:implementationLocation "scripts/recover.mjs" ; + sd:implementationLocation "scripts/power-loss-vm/exercise.py" ; + sd:implementationLocation "test/power-loss-baseline.mjs" ; + sd:implementationLocation "test/admin-recovery.mjs" ; + sd:implementationLocation "test/storage.mjs" ; + sd:implementationLocation "docs/recovery.md" ; + sd:implementationLocation "DURABILITY.md" ; + sd:implementationLocation "eslint.config.js" ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-25 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; sd:gitCommit "9b4a340facd57f22edbead07a195700b0237c9ad" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "9b4a340facd57f22edbead07a195700b0237c9ad" ] ; + sd:implementationChangeKind sd:SemanticChange . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 9c333d1..2eeaf78 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,7 +18,13 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Current active Spiral cycle: none. Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18. Conversion and command changesets now finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. Integration must still be validated against the current target. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16. +Current active Spiral cycle: `.spiral/cycles/CYC-20260919-TTZ7C-17.md` (`Power-Loss Durability And Administrative Recovery`), committed by the maintainer on 2026-09-19 and opened on `spiral/CYC-20260919-TTZ7C-17-power-loss-recovery` from `master` at `44ea91717a328ad7b5e049c6c0845b5e3bcc1718`. The cycle is Active; implementation and evaluation are complete and human acceptance is pending. Its source, understanding, and acceptance matrix are `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20`. + +Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18 and merged into `master` at `0444094`. Conversion and command changesets finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16 and merged at `e59c6f1`. + +Active-cycle progress on 2026-09-19: `IMP-20260919-TTZ7C-26` implements the corrected `DES-20260919-TTZ7C-24/25` following explicit authorization to complete the whole cycle. Existing command/status/integrity and dataset formats remain unchanged, with no migration. Durable complete-write and file/directory barriers, ownership, log-authoritative serial acceptance/execution, command-only workers, strict startup assessment, and source-bound administrator recovery/backup/restore are implemented. `EVD-20260919-TTZ7C-27` records 101 passing regression tests plus 22 expanded administrator tests (three additional cases), direct CLI checks, and a passing four-scenario ext4/QEMU power-cut exercise with device/filesystem negative controls and latency. `DURABILITY.md` and `docs/recovery.md` define the bounded behavior and runnable procedure. Historical baseline gaps remain at `EVD-20260919-TTZ7C-23`; PL15/16 expectations were corrected by the maintainer, not silently weakened. Full rebuild remains a documented separate contract. The cycle is ready for human acceptance but not Accepted or integrated. Broader filesystem/hardware and randomized testing, undeclared custom dependencies and historical input/effect uncertainty remain explicit limits. Two unchanged legacy index `load` methods have pre-existing undefined-meta lint errors outside this write-path cycle. + +Acceptance-review refinements: explicit multiline blocks, selective JAQT queries, fewer ternaries, and lines near 80 columns now improve the cycle code’s readability. Short conditional choices and long literals remain where clearer. The latest full relevant regression suite passes 106/106; EVD-27 records preserved ordering, backup coverage and plain-data report contracts. The example now uses `npm run build` for explicit initialization; conversion of its tagged URLs is fixed and build/start/query smoke verification passes. Human cycle acceptance is still pending. ## Intake State @@ -96,7 +102,11 @@ The first production-readiness priority is proving more of SimplyStore's ACID cl The current durability/extensibility source is `.spiral/sources/SRC-001.md`. The first ordered slice is baseline archaeology, followed by a crash/fault-injection harness, an independent reconstruction oracle, adversarial storage tests, retry/idempotency hardening, after-change handler contracts, handler failure evidence, optional integrity roots, integrity acceptance tests, property/randomized durability tests, destructive soak tests, a minimal post-commit extension seam, a demonstration derived store, and `DURABILITY.md`. -Process continuity rule: before proposing the next durability/production-readiness cycle, re-read `.spiral/sources/SRC-001.md`, `.spiral/requests/REQ-001.md`, and `.spiral/designs/DES-001.md`; identify the current position in that ordered plan; compare the latest evidence with it; and explicitly classify the proposal as continue, revise, or deliberate deviation. +**2026-09-19 refinement — revise immediate order:** `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20` bring durable file publication and administrative recovery ahead of general randomized/soak work. These refine the earlier failure-only allowance: explicit startup refusal must be accompanied by a tested recovery route. Accepted/completed acknowledgments need complete file/directory persistence barriers; storage-fault evidence precedes hardening, followed by recovery tooling, command-input completeness, consistent backup/restore, latency measurements, and a disposable filesystem power-loss exercise. The rest of the original roadmap remains. + +**Recovery contract (required, not yet implemented):** missing datasets do not prove that external effects never happened. Ordinary rerun requires administrator approval and no later accepted command with a dataset; execute eligible trailing missing/waiting commands in original order from verified preceding state. Automatic retries must not bypass this rule. Preserve original evidence, preview recovery on a copy, and validate before explicit promotion. Required command inputs must be logged; omissions are bugs. Separate full rebuild may replay selected history with changed code into a new store while instructing commands/hooks to suppress external side effects. Record that cooperative contract in this cycle; full rebuild tooling is follow-up work. + +Process continuity rule: before proposing the next durability/production-readiness cycle, re-read `.spiral/sources/SRC-001.md`, `.spiral/requests/REQ-001.md`, `.spiral/designs/DES-001.md`, and the 2026-09-19 refinement in `.spiral/sources/SRC-20260919-TTZ7C-18.md` / `.spiral/requests/REQ-20260919-TTZ7C-20.md`; identify the current position in that ordered plan; compare the latest evidence with it; and explicitly classify the proposal as continue, revise, or deliberate deviation. Current Spiral collaboration rule: treat planning, evaluation, and ambiguous human suggestions as discourse until a sufficiently explicit commitment exists. Human confirmation of a cycle goal is the commitment boundary for execution within that goal; later tentative comments should not silently become scope expansion. Before commitment, surface material ambiguities, contradictions, unsupported premises, or alternative framings when resolving them differently could materially change downstream work. @@ -269,7 +279,7 @@ This is not yet a claim that SimplyStore is production-safe for all workloads. I - Continue guided brownfield intake as concrete durability, security, API, and downstream questions arise. - Add a developer/operator script for resetting or cleaning local/example datasets after explicit recovery failures. -- Decide later whether SimplyStore should ever attempt durable artifact recovery/repair, or only provide inspection and reset tooling. +- Administrative inspection, authorized ordered recovery, and consistent backup/restore are now committed in CYC-20260919-TTZ7C-17. More general automatic repair remains a later decision. - Characterize the VM2 replacement risk as a candidate early Spiral cycle. - Extend process-level fault tests around command acceptance, changeset write, status append, and restart recovery. - Define larger-system integration options after the durability priority is better bounded. diff --git a/.spiral/requests/REQ-20260919-TTZ7C-20.md b/.spiral/requests/REQ-20260919-TTZ7C-20.md new file mode 100644 index 0000000..b4bc552 --- /dev/null +++ b/.spiral/requests/REQ-20260919-TTZ7C-20.md @@ -0,0 +1,87 @@ +--- +id: REQ-20260919-TTZ7C-20 +--- + +# Request: Power-Loss Durability And Administrative Recovery + +## Outcome and audience + +Developers and administrators need a bounded, executable power-loss durability +claim and a practical recovery route when committed storage cannot be loaded. +Complete this as one cycle, from fault evidence through sync hardening, +administrative recovery, and a verified backup/restore exercise. + +This supplements REQ-001 and refines its failure-only allowance: refusing unsafe +startup remains necessary, but a tested administrative route is also required. +It does not retrospectively invalidate the earlier bounded test results. + +## Required behavior and acceptance evidence + +| ID | Requirement | Discriminating evidence | +|---|---|---| +| P1 | Durable command acceptance | The complete execution record and acceptance survive supported power-loss cuts after `202`; a retained command is not mistaken for proof of safe re-execution | +| P2 | Durable completion | Before `done` is observable, the changeset and every required recovery artifact, including new/replaced directory entries, are durable; acknowledged completed state survives modeled cache loss | +| P3 | Explicit I/O uncertainty | Write, short-write, sync, rename, and close failures never report false acceptance/completion; uncertain commit failures halt further mutation pending recovery | +| P4 | Independent crash oracle | Keep issued-command order and observed acknowledgments outside the damaged store; independently reconstruct surviving committed data and detect silent rollback, duplication, and partial state | +| P5 | Read-only recovery inspection | Identify relevant command order/status/attempts, missing/corrupt/ambiguous artifacts, verified preceding state, and known/unknown external effects; produce a preview without changing original evidence | +| P6 | Administrator-authorized ordered recovery | Rerun only when explicitly authorized and no later accepted command has a dataset; execute eligible trailing missing/waiting commands in original order from verified preceding state; do not infer the rule from `done` alone | +| P7 | Recovery cannot be bypassed | Automatic retry and normal queue processing cannot bypass P6 or overtake recovery; revalidate evidence before applying a plan; corrupt/unknown files are not treated as absent | +| P8 | Complete execution inputs | Verify original and recovered invocations have all required logged inputs beyond the preceding dataset, including relevant configured-worker behavior; fix actual omissions, and diagnose incomplete legacy records rather than inventing values | +| P9 | Consistent backup and restore | Provide and test a usable consistent backup procedure and restoration into a separate store; show recovered command coverage, known loss, and uncertainty when reruns are prohibited or unsafe | +| P10 | Preserved evidence and deliberate promotion | Recovery preserves the original store/log and records administrator decisions and execution outcomes; validation precedes explicit promotion; recovery interruption itself must leave an inspectable safe state | +| P11 | Bounded filesystem claim | Name the tested Linux filesystem, mount options, storage/cache assumptions, and writer ownership; validate with a disposable block-level power-loss exercise, distinguish simulation from physical hardware evidence, and measure commit latency | +| P12 | Rebuild exception retained | Document separate new-store rebuild using selected code/history and context instructing commands/hooks to suppress external effects; this cannot become an ordinary recovery override | + +## Mandatory recovery examples + +In established command order, `A:dataset, B:missing, C:dataset` blocks rerunning +B, including when C is accepted but not currently marked done. + +`A:dataset, B:missing, C:waiting, D:missing` permits an administrator-authorized +ordered recovery of B/C/D after checking secondary effects. It is permission +eligibility, not a claim that replay is safe or automatically approved. + +A file that exists but is corrupt requires diagnosis. A later dataset appearing +after preview invalidates that plan. Missing command-log evidence also prevents +asserting eligibility from filenames alone. Failed/unsafe/duplicate records and +concurrent acceptance must receive explicit design/test dispositions, not a +silent choice of a convenient map iteration order. + +## Implementation direction and compatibility + +Start with a failure matrix and baseline tests. Reuse the current file-based +architecture and reconstruction oracle; choose small durable-write primitives +and explicit sequencing over a new storage framework. Determine which indexes +must persist before commit and which can be rebuilt safely from canonical data. +Include initial base/log creation and recovery writes in the persistence audit. + +The supported first envelope assumes a single writer and storage that honors +flushes on a named Linux/local-filesystem configuration. If that envelope cannot +be established in the test environment, report the missing evidence and keep +the power-loss claim outstanding. Do not relabel process-kill tests as proof. + +Preserve public REST/query APIs and disk formats by default. Any necessary log +or recovery format extension must explicitly address old stores, migration, +failure behavior, and downstream compatibility before implementation. + +Full rebuild tooling is follow-up work under P12's retained contract. Side-effect +suppression is cooperative unless separately enforced; actual command and hook +implementations need verification before being called rebuild-safe. Changes to +command code are explicit rebuild inputs, not ordinary recovery assumptions. + +## Scope exclusions + +- General randomized/destructive soak campaigns beyond the focused fault matrix + and disposable filesystem acceptance exercise. +- Full rebuild CLI/workflow implementation in this cycle. +- Automatic rollback or exactly-once guarantees for arbitrary external effects. +- General message buses, transaction frameworks, remote-filesystem guarantees, + sandbox replacement, package publishing, or production data recovery/deployment. + +## Evaluation + +Each P1-P12 row needs traceable implementation/documentation and verification +evidence appropriate to its claim. P12 is a recorded contract, not a tested +rebuild feature. Run relevant existing startup/offset, handler, ACID, and crash +suites as regression checks. A failed or unavailable required exercise remains +an open acceptance item; creation of the cycle is not acceptance of its outcome. diff --git a/.spiral/requests/REQ-20260919-TTZ7C-20.ttl b/.spiral/requests/REQ-20260919-TTZ7C-20.ttl new file mode 100644 index 0000000..22f1303 --- /dev/null +++ b/.spiral/requests/REQ-20260919-TTZ7C-20.ttl @@ -0,0 +1,23 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260919-TTZ7C-20 + a sd:Request ; + dcterms:identifier "REQ-20260919-TTZ7C-20" ; + sd:repositoryPath ".spiral/requests/REQ-20260919-TTZ7C-20.md" ; + sd:status sd:Active ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:UND-20260919-TTZ7C-19 ; + sd:gitCommit "d522359dee1621267df8e550237a9767cb72fccd" + ] ; + sd:constrainedBy [ + a sd:ArtifactReference ; + sd:artifact project:REQ-001 ; + sd:gitCommit "44ea91717a328ad7b5e049c6c0845b5e3bcc1718" + ], [ + a sd:ArtifactReference ; + sd:artifact project:DES-001 ; + sd:gitCommit "44ea91717a328ad7b5e049c6c0845b5e3bcc1718" + ] . diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.md b/.spiral/sources/SRC-20260919-TTZ7C-18.md new file mode 100644 index 0000000..64883f1 --- /dev/null +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.md @@ -0,0 +1,167 @@ +--- +id: SRC-20260919-TTZ7C-18 +--- + +# Source: Power-Loss Recovery And Administrative Replay + +## Origin and availability + +Maintainer conversation on 2026-09-19. The operative human statements below +are retained verbatim; the proposed cycle boundary is summarized separately. +There is no external conversation locator. This source supplements SRC-001. + +## Retained human direction + +> If we fail to recover the committed state, we need to have a plausible way for an administrator to recover. + +> We have a command log, if that is complete, shows an accepted command, but its data files are missing.... we can't just assume we can run the command again, it might have secondary effects. But an administrator should be able to judge whether it can be rerun. + +> The commands stored in the log should have all required information. If that is not so, that is a bug. Another thing, I don't think we should allow a command to be rerun, unless there are no accepted commands with their datasets after it. If all commands after it are either waiting or also miss their datasets, then the administrator can run them again. + +> There is one exception. I designed the system to theoretically be able to rerun all commands, useful if you alter the code the command executes and want to create a new dataset consistent with that new code. So I think we should have an option the adminstrator could set to signal to the commands not to trigger side effects, only update the data. + +After a proposed next-cycle plan, the maintainer committed: + +> Lets make a cycle to do all this work. + +## Proposal to which the commitment responded + +The assistant proposed one cycle for durable commits and an administrative +recovery path, explicitly revising the immediate roadmap order: + +- failure matrix and baseline tests before runtime hardening; +- durable acceptance and completion, including file/directory sync ordering; +- simulated lost/reordered/partial writes and sync failures, with acknowledgment + evidence outside the damaged store; +- inspection and administrator-authorized ordered recovery on a copy, preserving + the original store and preventing automatic replay from bypassing the rules; +- command-log input completeness and fixes for demonstrated omissions; +- a tested consistent backup/restore procedure; +- a specified Linux/local-filesystem envelope, latency measurements, and a + disposable block-level power-loss exercise before claiming that envelope; +- recording the rebuild-mode contract now, with the full rebuild workflow in + a following cycle. That mode produces a new store using selected code and + communicates that external side effects must be suppressed. + +This boundary is the assistant's proposal confirmed by the final human message; +it is not a quotation authored by the maintainer. The side-effect flag is a +cooperative command/hook contract, not proof that arbitrary code is sandboxed. + +## Subsequent design correction — 2026-09-19 + +After the first design proposed a versioned log envelope and migration, the +maintainer clarified the intended input boundary: + +> The original design goal was that commands would be the complete set of information. I think we might fix the issue by removing extra information in the command worker, instead of adding it to the log. + +The maintainer then authorized a design revision: + +> Lets see how far we can get in this cycle without a new dataformat and migration. Design only. + +This authorizes revising the cycle design, not runtime implementation in this +step. The interpretation to explore is complete command values as the execution +inputs beyond the existing dataset, removal of transport request context from +command execution, and retention of the current store file/record formats. +The assistant's earlier envelope, sequence fields, and mandatory v2 activation +were design choices, not explicit maintainer requirements. New recovery-tool +plans/reports are administrative output; they must not become a disguised +required store format or migration. Remaining limits must be stated explicitly. + +## Command-log authority clarification — 2026-09-19 + +The maintainer requested verification of the intended ordering rule: + +> The current system should have a single sequence of commands being processed according to the order in the command log. You should be able to rely on that order. Unless I have missed something. Please verify. + +After verification exposed a concurrency gap between append and queue insertion, +the maintainer confirmed: + +> That is a problem, the command log should be authoritive + +The command log defines the order. Status records, client responses, and queue +timing are not alternative authorities. Their disagreement is evidence of a bug +or damaged history. The previous design-only and no-migration boundary remains; +the immediate work is to correct the designs and future verification expectations. + +## Authorization to complete implementation — 2026-09-19 + +Following explanation of serialized acceptance through queue insertion, the +maintainer authorized continuing the entire cycle: + +> Ok, that sounds like a good fix, I'll let you continue the whole cycle with this design. Only stop when I need to accept the cycle, or you cannot resolve a question in a sensible direction yourself. + +This supersedes the earlier design-only execution limit. Continue implementation, +tests, recovery tooling, documentation and evaluation under the current designs, +with unchanged store formats and command-log authority. Cycle acceptance remains +the maintainer's decision. Routine implementation choices require no new approval. + +## Remaining acceptance evidence + +Exact recovery CLI syntax, durable command ordering, log-format implications, +filesystem test configuration, and handling of ambiguous/corrupt evidence need +design and executable evidence. No runtime outcome or cycle acceptance is +claimed by this source. Creating the cycle is distinct from completing it. + +## Code-style correction during acceptance review — 2026-09-19 + +The maintainer has not yet accepted the cycle. After reviewing the net server +code growth, they requested: + +> That is a good number. One thing: I prefer to always use block { and }, even if there is only one line in an if or for statement. + +Use braces consistently in the cycle's JavaScript control flow and retain this +preference in repository instructions. This is a formatting correction, not a +change to the durability/recovery design or cycle acceptance. + +The maintainer clarified the block layout: + +> After a block { always start on a new line, and a closing } needs its own line as well + +Expand block bodies, including empty blocks and callbacks, onto separate lines; +place `else`, `catch`, and `finally` after the closing-brace line. Object literals +and destructuring braces are not statement blocks. This remains formatting-only. + +## Readability and JAQT review — 2026-09-19 + +The maintainer identified the dense startup diagnostic expression, requested a +read-only JAQT feasibility check, and approved the example with named diagnostic +groups using the installed JAQT 0.10.6. They then instructed: + +> That is a lot more readable. Please check the rest of the code to see if jaqt could improve it, if so, update it. + +Review the remaining code for clearer query expressions and named intermediate +results; use JAQT where it improves readability. Preserve behavior, ordering and +public result types. This authorizes the refactor within the still-active cycle; +it does not constitute cycle acceptance. + +## Conditional expressions and line width — 2026-09-19 + +The maintainer added these readability preferences: + +> More small changes: I prefer not to use ?: unless it is a clear win in readability. I prefer to keep lines about 80 chars wide at most, again, unless readability suffers. + +Expand dense/nested ternaries into named steps or `if`/`else`; retain short, +obvious conditional choices when they are clearer. Wrap the cycle's JavaScript +toward 80 columns, preserving exceptions for readability and literal contents. +These are review guidelines with explicit exceptions, not blanket prohibitions. +This authorizes a behavior-preserving readability revision, not cycle acceptance. + +## Example initialization correction — 2026-09-19 + +The maintainer reported that the example fails startup without its base and +command/status logs, and requested: + +> simplystore now provides a convert and build script. I think the example should use those, preferably by adding them in the package.json of the example. Then the convert script should work. An error in that is something we need to fix. + +This authorizes repairing the converter and exposing the example initialization +workflow through npm scripts. The checked-out converter already initializes empty +logs, but conversion of the actual Star Wars dataset fails on URL validation. +No separate SimplyStore build script exists in this checkout; its intended +meaning is being clarified. Existing store data must not be overwritten. + +The maintainer clarified: + +> curriculum-store has an import script and a build. The example only needs a build script + +Expose `npm run build` in the example, invoking SimplyStore's existing converter. +No separate import script or changed-code rebuild feature is requested. diff --git a/.spiral/sources/SRC-20260919-TTZ7C-18.ttl b/.spiral/sources/SRC-20260919-TTZ7C-18.ttl new file mode 100644 index 0000000..c56f5b3 --- /dev/null +++ b/.spiral/sources/SRC-20260919-TTZ7C-18.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:SRC-20260919-TTZ7C-18 + a sd:Source ; + dcterms:identifier "SRC-20260919-TTZ7C-18" ; + sd:repositoryPath ".spiral/sources/SRC-20260919-TTZ7C-18.md" ; + sd:status sd:Active ; + sd:sourceAvailability sd:Retained ; + sd:provenanceConfidence sd:Explicit . diff --git a/.spiral/understandings/UND-20260919-TTZ7C-19.md b/.spiral/understandings/UND-20260919-TTZ7C-19.md new file mode 100644 index 0000000..20aa607 --- /dev/null +++ b/.spiral/understandings/UND-20260919-TTZ7C-19.md @@ -0,0 +1,84 @@ +--- +id: UND-20260919-TTZ7C-19 +--- + +# Understanding: Durable Commits With Recoverable Failure + +## My understanding + +The durability objective includes both preventing acknowledged data loss under +the supported power-loss model and giving administrators a practical recovery +route when automatic reconstruction fails. Refusing startup remains appropriate +for ambiguous storage, but refusal alone no longer satisfies the outcome. + +Reconstruction applies surviving committed changesets without executing command +code. Rerunning a command is different: its external effects may already have +happened. Missing data does not authorize automatic re-execution. + +For ordinary recovery, a command may be rerun only with administrator approval +and only when no later accepted command has a dataset. The eligible trailing +commands are executed in their original order from the verified preceding +state. This predicate must consider all accepted commands, not just current +`done` statuses. Corrupted or ambiguous files are not silently treated as absent. + +Logged commands must preserve all execution inputs required beyond the preceding +dataset. Any demonstrated omission is a defect, not an acceptable recovery +limitation. Missing inputs in historical records cannot be invented by tooling. + +Full rebuild is a separate operation: execute selected history using selected +code against a starting dataset and write a new store. It may deliberately +produce different data and bypass ordinary recovery's later-dataset restriction. +Commands and hooks must receive a context instructing them to suppress external +side effects; canonical and derived output writes are still required. A flag +alone does not prevent arbitrary code from making external changes. The contract +is retained now; the full rebuild workflow follows this cycle. + +## Current effective behavior and evidenced gap + +Read-only inspection of baseline `44ea91717a328ad7b5e049c6c0845b5e3bcc1718` +established the following before the human's cycle commitment: + +| Boundary | Current behavior | Gap / remaining evidence | +|---|---|---| +| `src/util.mjs#appendFile` | Appends and awaits `datasync()` | No explicit parent-directory sync for newly created files; failure propagation and torn records need tests | +| Command changesets / final offsets | Installed `write-file-atomic` syncs the temporary file and renames it | No parent-directory sync after publication | +| Base conversion / ID indexes | Ordinary `writeFileSync` | No explicit persistence barrier; classify recovery-required versus safely rebuildable artifacts | +| `src/server.mjs#checkCommand` / completion | Log/status appends awaited before acceptance; `done` awaited before query updates | Complete cross-file persistence dependency and error handling need verification | +| `src/recovery.mjs` | Reads statuses/log; some interrupted active commands become accepted for automatic replay | No administrative uncertainty decision or later-surviving-dataset eligibility guard | +| Recovery order | Log iteration queues accepted commands; status-map iteration selects committed changesets | Establish the authoritative execution order, including concurrent acceptance, before enforcing suffix recovery | +| Command inputs | Normal queue includes request context; replay uses `request: null`; default worker does not forward request to `runCommand` | The FIXME is not proof of differing default-handler inputs; test normal/replay equivalence, including configured workers | +| Existing test suites | Process-crash, corruption, retries, offsets, handler behavior and timeouts | Process termination does not establish loss of kernel/device caches or a tested administrator recovery procedure | + +The earlier conversation's fresh 59-test run is baseline context, not new +power-loss evidence. No runtime changes or new fault tests are claimed here. + +## Material assumptions and boundaries + +- First persistence envelope: a named Linux/local-filesystem configuration with + storage honoring flush requests. Record actual filesystem/mount/virtual-disk + settings in evidence; do not claim arbitrary hardware or remote filesystems. +- The intended ownership model is one writer. Verify it or enforce an appropriate + exclusion mechanism; administrator recovery cannot race ordinary queue writes. +- Recovery works on a copy and preserves the original command log and artifacts. + Revalidate a preview's inputs before applying it so a stale plan cannot bypass + the ordering guard. Promotion is a separate explicit administrator decision. +- A consistent backup plus surviving committed data supplies a recovery route + when rerunning is unsafe or disallowed. Report verified coverage, loss, and + uncertainty; do not promise to reconstruct irretrievably missing information. +- A failed sync can leave commit status uncertain. Prevent false success and + further mutations until recovery establishes a safe state; do not invent a + definitive failed outcome merely because persisting `done` returned an error. +- Preserve disk/query/REST compatibility by default. Any required durable-format + change must explicitly cover migration and legacy-store behavior. + +## Commitment and plan continuity + +The maintainer confirmed the preceding investigation and proposed cycle with +"Lets make a cycle to do all this work" on 2026-09-19. This is the execution +commitment for the agreed goal, not acceptance of its future implementation. + +Revise the immediate SRC-001 / REQ-001 / DES-001 sequence: crash/retry/handler +characterization and optional integrity exist, while file publication and +administrative recovery undermine broader confidence. Address these before +general randomized/soak work. Preserve the remaining roadmap and the recorded +full-rebuild follow-up. Reopen only material assumptions falsified by evidence. diff --git a/.spiral/understandings/UND-20260919-TTZ7C-19.ttl b/.spiral/understandings/UND-20260919-TTZ7C-19.ttl new file mode 100644 index 0000000..0620928 --- /dev/null +++ b/.spiral/understandings/UND-20260919-TTZ7C-19.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:UND-20260919-TTZ7C-19 + a sd:Understanding ; + dcterms:identifier "UND-20260919-TTZ7C-19" ; + sd:repositoryPath ".spiral/understandings/UND-20260919-TTZ7C-19.md" ; + sd:status sd:Active ; + sd:provenanceConfidence sd:Evidenced ; + sd:interprets [ + a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-18 ; + sd:gitCommit "806da371341225ee90cd9da1e0f71f9f3b5564e6" + ], [ + a sd:ArtifactReference ; + sd:artifact project:SRC-001 ; + sd:gitCommit "44ea91717a328ad7b5e049c6c0845b5e3bcc1718" + ] . diff --git a/AGENTS.md b/AGENTS.md index e8351d7..c52123e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,12 +41,28 @@ SimplyStore has a governing durability/extensibility direction in: - `.spiral/requests/REQ-001.md` - `.spiral/designs/DES-001.md` +The maintainer refined that direction on 2026-09-19 in: + +- `.spiral/sources/SRC-20260919-TTZ7C-18.md` +- `.spiral/understandings/UND-20260919-TTZ7C-19.md` +- `.spiral/requests/REQ-20260919-TTZ7C-20.md` + +Read this refinement alongside the original direction. It requires an +administrative recovery path, complete logged command inputs, and ordered rerun +eligibility; it records separate new-store rebuild semantics without making +full rebuild tooling part of the current power-loss cycle. + Before proposing a next durability/production-readiness cycle, re-read those durable references, identify the current position in the ordered plan, reconcile the latest evidence with that plan, and state whether the proposal continues, revises, or deliberately deviates from it. Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifacts and companion Turtle resources only when they are causally useful for an actual cycle, decision, implementation, evidence, acceptance, or lesson. ## Working Defaults +- Always use braces for control-flow bodies, including single-statement `if`, `else`, and loop bodies. Start block contents on the line after `{`, and put the closing `}` on its own line. This includes empty blocks and callbacks; `else`, `catch`, and `finally` start on the next line after `}`. + +- Prefer `if`/`else` to `?:` unless a short conditional expression clearly improves readability. +- Aim for code lines no wider than about 80 characters; allow exceptions when wrapping would reduce readability. +- Prefer named intermediate results over dense chains of operations. Use the existing JAQT library where it makes data selection and projection clearer; keep straightforward operations simple. - Treat `master` as the authoritative branch unless the human states otherwise. - For ordinary repository-changing Spiral cycle work, create one dedicated branch from the authoritative branch, normally `spiral/CYC-###-short-goal`. - For new Spiral artifacts after CYC-018, use distributed-safe IDs allocated by `node .spiral-core/bin/spiral.mjs allocate ` instead of scanning for the next legacy number. Existing `SRC-001` / `CYC-017` style IDs remain valid historical artifacts and must not be renamed. diff --git a/DURABILITY.md b/DURABILITY.md new file mode 100644 index 0000000..cd8366e --- /dev/null +++ b/DURABILITY.md @@ -0,0 +1,47 @@ +# Durability contract + +SimplyStore retains its existing base, changeset, command/status and optional +integrity formats. There is no data migration. The command log is authoritative: +acceptance serialization keeps execution in log order, including concurrent +requests and duplicate IDs. A command supplies all inputs beyond its preceding +dataset; handlers receive `undefined` as the third argument and metadata as the +fourth. Audit application commands that previously used request context. + +Before returning accepted, SimplyStore completely writes and syncs the command +and acceptance records and their parent directories. Before publishing done, it +finishes the changeset, index hooks and required artifacts, syncs file contents +and published directory entries, and persists terminal status. Short writes are +retried; write, sync, rename and close failures propagate. Uncertain persistence +halts mutation and leaves ownership evidence for inspection. + +The tested envelope is one cooperating writer on Linux/ext4, `data=ordered`, +`barrier=1`, with storage honoring flushes. A QEMU guest over a volatile NBD device +model exercises full guest/cache loss and recovery. This is bounded filesystem +and block-model evidence, not physical hardware certification. See the +[reproducible exercise](docs/power-loss-filesystem.md) and +[cycle evidence](.spiral/evidence/EVD-20260919-TTZ7C-27.md). + +Startup validates existing canonical files before executing user modules or +changing command status. Complete consistent stores reopen normally. Missing +logs, malformed records, missing/corrupt committed data, or pending/uncertain +commands stop startup for administrator assessment. Upgrading cannot prove that +older writers preserved the whole history. Explicit integrity configuration is +needed to detect a lost manifest; existence-based detection alone cannot do so. + +The [administrator guide](docs/recovery.md) provides read-only inspection, +source-bound preview, explicitly approved suffix recovery on a copy, interrupted +recovery handling, consistent backup/restore and deliberate promotion. No later +ever-accepted command may have a dataset when an earlier command is rerun, +including corrupt or unsafe later datasets. Missing files never establish that +external effects did not happen. Recovery cannot invent missing inputs or promise +exactly-once external effects. Changed-code rebuild with cooperative effect +suppression remains a separate follow-up contract. + +Ownership locks exclude cooperating server/admin writers across configured +paths. They are not protection against arbitrary filesystem writers or older +binaries. Stale locks require explicit offline assessment, never PID-age stealing. +Custom hooks must await their work; declare additional required artifacts with +`requiredFiles`. Default indexes are reconstructible; custom dependencies need +their own conformance checks. Unsupported directory-sync operations fail rather +than silently weakening the guarantee. Broader filesystems, random/soak testing, +and deployment-specific power-loss testing remain future validation work. diff --git a/README.md b/README.md index f822972..1210d77 100644 --- a/README.md +++ b/README.md @@ -38,18 +38,20 @@ Import the server in your main file like this: import simplystore from '@muze-nl/simplystore' ``` -Then configure and start the server, like this: +Initialize the store first using the [conversion procedure](docs/recovery.md#configure-and-initialize); opening an existing store requires its base and logs. Then configure and start the server: ```javascript simplystore.run({ - datafile: process.cwd().'data.json' + datafile: './store/data.jsontag', + commandLog: './store/command-log.jsontag', + commandStatus: './store/command-status.jsontag' }) -```` +``` simplystore is an [express application](https://expressjs.com/), with all the usual options. Other options are: - port: The port number to use, defaults to 3000 -- dataspace: an object or array with all the data that SimplyStore will serve. Optional, replaces the datafile. +- commandsFile: the module implementing commands; every invocation input must be in the logged command. HTTP request context is not passed to handlers. If you start your server: @@ -61,6 +63,10 @@ You should be able to go http://localhost:3000/query/ and see something like thi ![image](https://github.com/SimplyEdit/SimplyStore/assets/1006453/3bec6b97-ffa1-4114-9ed4-51a68f73476e) +## Durability and recovery + +See the [durability contract](DURABILITY.md) and [administrator recovery guide](docs/recovery.md). Command-log order governs execution. Acceptance and completion await file and directory barriers. Uncertain or pending work found at startup requires administrator assessment; missing data does not prove that external effects did not happen. Existing store formats are retained without migration. + ## Custom Index Modules Use `indexFile` to configure a module whose default export provides the existing diff --git a/docs/power-loss-baseline.md b/docs/power-loss-baseline.md new file mode 100644 index 0000000..39a563e --- /dev/null +++ b/docs/power-loss-baseline.md @@ -0,0 +1,110 @@ +# Power-loss failure matrix and baseline + +The historical matrix below records the first evidence slice against runtime +`44ea91717a328ad7b5e049c6c0845b5e3bcc1718`, preserved by +[EVD-23](../.spiral/evidence/EVD-20260919-TTZ7C-23.md). That baseline had ten +violated invariant assertions. It is not a description of the hardened runtime. + +## Run and interpret + +```sh +npm run test:power-loss +npm run test:power-loss:strict +npm run test:recovery +``` + +The current power-loss suite checks corrected invariants with no TODO allowances; +both power-loss commands must pass. Tests need localhost ports and use disposable +storage. Administrative policy cases now live in `test/admin-recovery.mjs`. +PL15's old request-retention expectation was superseded by the maintainer's +command-only contract: the replacement proves the request is unavailable and +logged JSONTag values survive. PL16 now compares both live execution and restart +with the authoritative command log, fixing the old queue order instead of +preserving it. Historical observations remain evidence of the original defects. + +Current coverage includes full-write loops, file/directory barriers and errors, +missing canonical files, no automatic effect repetition, concurrent acceptance +and duplicates, and recovery/backup policy. See [the current durability +contract](../DURABILITY.md), [administrator guide](recovery.md), and [filesystem +exercise](power-loss-filesystem.md) for the implemented behavior and its bounds. + +## Historical persistence boundaries + +| Artifact | Baseline writer / barrier | Required role | +|---|---|---| +| Base dataset | Converter uses `writeFileSync` | Durable starting state before operating or backing up a store; untested conversion publication remains open | +| Command log | `appendFile` then `datasync` | Complete inputs before accepted; persist first-created filename | +| Command status | `appendFile` then `datasync` | Accepted/active/done ordering; persist first-created filename; distinguish sync error from guaranteed rollback | +| Changeset | `write-file-atomic`: file sync then rename | Complete bytes and durable filename before done | +| Final offset index | `write-file-atomic`: file sync then rename | Current startup reconstructs without it; custom consumers still need an explicit contract | +| ID index | `writeFileSync` | Current startup reconstructs without it; no universal custom-index conclusion | +| Integrity manifest | Append then `datasync` | Ordered metadata for integrity-enabled stores; dedicated power-loss cases remain open | + +File sync and directory sync are distinct barriers. The namespace-loss model +uses this distinction from the [Linux fsync contract](https://man7.org/linux/man-pages/man2/fsync.2.html). +It does not assert that every tested filesystem would actually lose an entry. + +## Historical baseline matrix + +| Test | Fault or question | Observed baseline | Requirement disposition | +|---|---|---|---| +| PL01 | Instrumented normal commit, then process crash | Existing log contents sync; changeset temp file sync precedes rename; restart equals external expected state | Passing control, not physical power-loss proof | +| PL02 | Drop newly created status filename lacking a directory barrier after observed done | Startup serves empty base state despite external completion evidence | P2 violated: silent loss | +| PL03 | Drop newly created command-log filename after observed accepted | Server starts with pending status but no reconstructible command task | P1 violated: accepted inputs lost | +| PL04 | Drop renamed changeset filename lacking directory barrier after observed done | Startup refuses with missing changeset | P2 durability violated; explicit failure is preferable to silent state | +| PL05 | Remove default ID/offset sidecars from copied store | Startup reconstructs expected canonical state | Current default recovery does not require persisted sidecars | +| PL06 | Append a malformed status tail to a copied store | Startup refuses; original store preserved | Passing corruption classification; administrator path remains absent | +| PL07 | Changeset file fsync throws EIO | Command fails, no done or query update | Passing bounded error propagation | +| PL08 | Changeset rename throws EIO | Command fails, no done or query update | Passing bounded error propagation | +| PL09 | Changeset write throws ENOSPC | Command fails, no done or query update | Passing bounded error propagation | +| PL10 | Callback write succeeds with only half the changeset bytes | Command becomes done and live query looks correct; restart refuses truncated data | P3 violated: short write is not retried/validated | +| PL11 | Done-status datasync throws EIO | Server exits; unsynced done bytes remain visible in the file | Passing conservative termination; failure is not proof of rollback | +| PL12 | Pause before done sync, lose unsynced status tail and unpublished changeset name | Restart repeats an external side-effect witness, then commits one dataset transition | P7 violated: dataset idempotency is insufficient | +| PL13 | B accepted/missing; later accepted C has real dataset, with current status accepted or done | Recovery queues B in both cases | Two P6 violations: later-dataset guard absent | +| PL14 | JSONTag command with date, nested data, false/zero/null and Unicode | Logged/reconstructed body preserves the input | Passing body-completeness control | +| PL15 | Configured worker uses request query parameter | Normal execution uses it; log/reconstructed task omits it | P8 violated for this supported worker configuration | +| PL16 | A log sync completes but its acceptance is gated; B accepted while a blocker runs | External acceptance and original waiting queue B/A; command log A/B; restart executes A/B | P4/P6 violated: recovery order differs from waiting-queue order | +| PL17 | Active A and done C with missing C dataset | Startup appends accepted for A before refusing C; old records remain intact | P7 violated: uncertain active work reclassified without review | +| PL18 | Command-log datasync error | HTTP 500, no accepted response or query mutation | Passing acceptance barrier | +| PL19 | Accepted-status datasync error | HTTP 500, no accepted response or query mutation | Passing acceptance barrier | +| PL20 | Error reported after command-log close | HTTP 500, no accepted response or query mutation | Passing conservative close-error propagation | +| PL21 | Test custom finalizer explicitly syncs the parent directory | Trace observes the barrier; loss model preserves filenames; restart matches independent expectation | Positive control for directory tracing and loss-model discrimination | + +PL02/03/04/12 alter a stopped copy and leave the source store and observer files +untouched. PL06 is deliberate corruption, not cache loss. PL13 and PL17 are +synthetic policy/startup inputs, not claims that a power cut produced those +exact histories. PL15/16 use actual HTTP calls and the configured/default worker +paths respectively. Concurrent PL16 uses a controlled gate instead of a race. + +## Independent observations and model limits + +Each fixture has a store directory and a separate observer directory. Client +issued commands, acceptance responses, and polled completion responses go into +an external journal. Expected fixture values are derived from those inputs and +observations, not the store's command log/status. Actual data reconstruction +still uses OD-JSONTag parsing, so format decoding is not an independent parser. + +An explicitly loaded test-only preload traces promises FileHandle append/sync/ +close operations and callback open/write/fsync/rename/close used by the current +runtime and installed atomic-write dependency. Worker threads inherit it. +It records synchronized fixture-log bytes and observed namespace publication. +The loss model can drop a new filename only if no later directory sync appears +in the trace. PL12 restores the last completed status sync in a controlled +single-writer interval. It never removes acknowledged synced content arbitrarily. + +This is a bounded permitted-outcome model, not a general virtual filesystem: +it does not intercept every Node/native/custom I/O API or enumerate every +possible persistence ordering. The observer is assumed outside the failed +storage domain. It does not model sector corruption of earlier synced records, +hardware lies about flushes, real filesystem mount behavior, or external systems. +The external-effect witness proves repeated invocation only, not exactly-once +behavior of any real email/payment/network service. + +## Current evaluation + +Implementation and current test/VM results are recorded separately in +[EVD-27](../.spiral/evidence/EVD-20260919-TTZ7C-27.md). The bounded preload model +and a real ext4 guest over a volatile block-device model provide complementary +evidence. Neither is physical hardware certification or an exhaustive campaign. +Full changed-code rebuild tooling remains follow-up work under the recorded +side-effect-suppression contract. diff --git a/docs/power-loss-filesystem.md b/docs/power-loss-filesystem.md new file mode 100644 index 0000000..17ac1ef --- /dev/null +++ b/docs/power-loss-filesystem.md @@ -0,0 +1,47 @@ +# Disposable filesystem power-loss exercise + +`scripts/power-loss-vm/exercise.py` runs the actual server and storage code in a +Linux VM with an ext4 data volume (`data=ordered,barrier=1`). It creates only +regular image files beneath a new temporary workspace; it never mounts or writes +a host block device. A statically linked BusyBox, Node, Python, cpio, gzip, mkfs.ext4, +a QEMU x86 system binary/firmware, and a Linux kernel with built-in ext4 and virtio +block support are required. Installed packages are not modified by the script. + +```sh +python3 scripts/power-loss-vm/exercise.py \ + --qemu /usr/bin/qemu-system-x86_64 --kernel /path/to/readable/vmlinuz \ + --firmware-dir /usr/share/qemu --bios /usr/share/seabios/bios-256k.bin +``` + +`--library-path` can point at libraries unpacked with QEMU into a temporary +workspace. `--work` selects a new, nonexistent output directory; otherwise the +script allocates one under `/tmp`. No host root privileges or KVM access are +required: it uses TCG emulation with two CPUs and 1024 MiB RAM. No external guest +network is attached; client/server traffic stays on guest loopback. + +The host observes acknowledgment markers over the VM's serial output and kills +the entire VM, losing guest RAM and kernel caches. The test NBD device buffers +writes in memory and persists only completed FLUSH/FUA operations (it may persist +other bytes in the same page on FUA). At each cut the device also discards its +remaining dirty pages. Reboot attaches the surviving image and lets ext4 recover; +read-only engine inspection reports committed data and uncertain commands. + +The device has a separate negative control: flushed A survives while unflushed B +is lost. A filesystem negative control also verifies that an ordinary unsynced +write disappears. Other scenarios cut after accepted, during an active handler, +and after observed completion. The completion scenario also records guest ext4 +acceptance/completion latency for five commands. The output includes exact tool/ +kernel identification, mount/cache settings, source revision/diff digest, serial +logs, block write/flush counts, images and a results report. + +This tests a named Linux/filesystem/storage model, not physical power removal or +all hardware/cache implementations. It is a focused boundary exercise, not an +exhaustive randomized campaign. The device model honors flushes; devices that lie +about them are outside the guarantee. It cannot prove absence of historical loss +from a different writer/protocol. Local `npm run bench:durability` timings report +their filesystem separately; tmpfs timings are not disk performance measurements. + +The model uses QEMU's documented [block device cache settings](https://www.qemu.org/docs/master/system/qemu-manpage.html). +See also QEMU's [block I/O fault testing documentation](https://www.qemu.org/docs/master/devel/testing/blkdebug.html) +for the distinction between guest writes and flush events. The test backend is +purpose-built for this bounded exercise, not a production NBD server. diff --git a/docs/recovery.md b/docs/recovery.md new file mode 100644 index 0000000..2cc9d94 --- /dev/null +++ b/docs/recovery.md @@ -0,0 +1,195 @@ +# Administrator recovery + +SimplyStore preserves the existing OD-JSONTag data, command-log, status-log and +optional integrity formats. The command log defines command order. Commands must +contain every invocation input beyond the dataset; HTTP request context is no +longer supplied to command workers. The fourth handler argument still provides +metadata/index access; the former third request argument is `undefined`. + +Use this procedure with the hardened binary. Older binaries understand the file +formats but do not implement the new ownership, ordering, or recovery policy. +Do not run competing versions/writers on one store. + +## Configure and initialize + +Tools accept a declarative JSON file, not executable server configuration: + +```json +{ + "datafile": "/srv/simplystore/data.jsontag", + "commandLog": "/srv/simplystore/command-log.jsontag", + "commandStatus": "/srv/simplystore/command-status.jsontag", + "commandsFile": "/srv/application/commands.mjs", + "indexFile": "/srv/application/index.mjs", + "integrityFile": "/srv/simplystore/data.integrity.jsontag" +} +``` + +Omit `integrityFile` if integrity is unused. Set it explicitly when integrity is +required: existence-based detection cannot detect a lost manifest. Relative paths +resolve against the command's working directory, as server paths do. `requiredFiles` +can list additional required files; these are checked at startup and synced before +done. Keep custom artifact paths inside the declared directories or list them +explicitly. Custom hooks must finish all required work before resolving. + +For a new store, create an empty destination directory and run: + +```sh +node scripts/convert.mjs input.jsontag /srv/new-store/data.jsontag --integrity +``` + +The converter creates empty command/status logs alongside the base, finishes +indexes and optional integrity, and releases ownership after durable completion. +It refuses to overwrite an existing base/log/manifest. Opening an existing store +never creates missing logs. A bare historical base is not proof of an empty +history: preserve it and its surrounding evidence before deciding on initialization. +There is no migration or format-conversion step for an existing complete store. + +## Inspect before executing anything + +Stop the writer using the deployment's normal service procedure. Then: + +```sh +node scripts/recover.mjs inspect --store store.json --quiescent --out plan.json +``` + +Use `--quiescent` only for a stopped store or immutable copy. Without it, inspection +is diagnostic and the plan is not actionable. Plans must be new files outside the +source directories. Inspection does not import handlers, custom workers, or index +modules, and does not modify the source. + +Review `blocks`, the commands in log order, statuses and attempts, existing/missing +files, and the proposed `rerun` list. A complete command log fixes the order even +when an old queue/status sequence disagrees. Actually committed out-of-order +changesets require diagnosis; reordering positional data is not a valid repair. + +The tool cannot prove history completeness, original deployment identity, hidden +application inputs, or external effects from the surviving files alone. Use retained +backups, deployment records, application knowledge, and external-system evidence. +A current code hash binds the selected code against changes after preview; it +cannot prove which code ran historically. Missing business inputs are not invented +or injected by the tool. If these matters cannot be established, use a verified +backup or retain the diagnosis; do not claim a safe rerun. + +## Deal with ownership after a crash + +Each mutable directory has a `.simplystore-lock` during operation. Normal shutdown +drains accepted work before releasing ownership. A crash or storage fault leaves +locks. Neither PID age nor an apparently dead process authorizes automatic stealing. +A hung shutdown can be terminated, but then requires this offline procedure. + +After confirming that no writer remains, inspect and explicitly release locks: + +```sh +node scripts/recover.mjs unlock --store store.json --confirmed-stopped \ + --operator 'administrator name' --reason 'Writer stopped; reviewed retained evidence' \ + --out unlock-audit.json +``` + +This preserves lock diagnostics in a new external audit file before removal. It +does not authorize reruns or repair data. Normal startup still rejects unresolved +accepted/active work, missing committed data, and other inconsistent evidence. + +An interrupted recovery has an external audit bundle and candidate ownership, +including a parent lock while the candidate is being copied. Review that bundle +before releasing its locks. A candidate that looks complete but lacks its recovery +completion report must be finished with `finish` below, not merely unlocked. + +## Apply an approved trailing recovery + +For log order A/B/C, B cannot be rerun if C has a dataset—even if C is accepted, +failed, unsafe, or its dataset is corrupt or zero bytes. Status insertion order +cannot make C appear earlier. If the trailing commands have no datasets, the +administrator can approve their execution in log order from a verified prefix. +Absence of a dataset does not prove absence of external effects. + +```sh +node scripts/recover.mjs apply --plan plan.json --to /srv/recovered-store \ + --audit-dir /srv/recovery-audit --approve-rerun B,C,D \ + --operator 'administrator name' \ + --reason 'Verified complete history and inputs, original command meaning, and assessed external effects' +``` + +The approval must exactly match the complete ordered `rerun` list. Source, target, +and audit locations must not overlap; destination directories must be new. The +source must remain quiescent. The tool acquires ownership and rechecks the source +inventory, missing filenames, selected code, and rerun predicate before execution. +A newly appearing later dataset invalidates the plan. + +Original store artifacts are copied into the audit bundle. Unrelated files such as +application secrets are not swept into backups; retain deployment evidence separately. Candidate files preserve their +relative layout and permissions; mutable files are copied, never hard-linked. +Commands retain original IDs and logged inputs. Recovery appends existing status +records, preserving the original bytes as a prefix. Authorization/attempt links +live in the external audit, not a new store record format. Required persistence +errors stop work; a possibly committed done is never overwritten with failed. +A retained expected changeset digest is checked before a new digest or done is +written. Without that witness, structural correctness is not independent proof +of byte-identical historical output. + +On an interrupted or failed attempt, preserve the candidate and audit. Inspect the +candidate with its configuration from the audit authorization. Completed commands +are reconstructed, not selected again. An active attempt needs fresh effect +assessment and a new plan; existing partial data needs separate diagnosis and is +not automatically deleted to make a rerun eligible. After explicit offline lock +release, apply an eligible plan to another new destination. Do not restart recovery +against an older source while ignoring later candidate evidence. + +If all approved commands committed but only completion reporting was interrupted: + +```sh +node scripts/recover.mjs finish --audit-dir /srv/recovery-audit --confirmed-stopped \ + --operator 'administrator name' --reason 'Reviewed completed attempts and original authorization' +``` + +This validates the retained authorization, source, candidate, and approved command +coverage, writes the completion report, and releases ownership without execution. +A missing audit bundle prevents certification of that recovery attempt. + +## Consistent backup and restore + +Create backups with a stopped writer. The tool acquires ownership, copies canonical +data/log/integrity files, default index artifacts, and explicitly required files, validates reconstruction, +and writes an external completion manifest last: + +```sh +node scripts/recover.mjs backup --store store.json --quiescent --to /srv/backup-001 +node scripts/recover.mjs restore --backup /srv/backup-001 \ + --to /srv/restored-store --audit-dir /srv/restore-audit +``` + +A completed backup can be moved or transferred as one directory; restore validates +and rebases its external inventory while preserving canonical bytes. Incomplete +or changed backups are refused. Restore executes no handlers. Optional +`--store damaged-store.json --quiescent` compares backup coverage against a stopped +source under ownership. Review `sameBase`, `missingFromBackup` and the warnings; comparison includes +command bytes, status, and available completed data. Without a +source, loss relative to current history is unknown. A known older backup can be +an explicit rollback with disclosed loss; it is not recovery of all later work. +This first tool restores complete backups; it does not merge arbitrary surviving +changesets, repair malformed logs, or override unknown command inputs. + +## Validate and promote + +```sh +node scripts/recover.mjs verify --report /srv/recovery-audit/complete.json \ + --out activation-preview.json +``` + +Verification also rejects changes to the selected recovery code files. Review +the exact candidate configuration and command coverage. Undeclared dependencies +and deployment configuration still require independent review. Stop the old +service, preserve its directory and configuration, and explicitly switch the +service to the candidate configuration. The tool does not change a deployment. +The server validates the selected store before listening. Switching multiple +configured directories is not an atomic filesystem operation. After accepting +new commands, switching back can lose those commands and needs a fresh assessment. +Normal loading does not require the external administrative reports. + +## Rebuild is separate + +Full changed-code rebuild remains follow-up work. Its retained contract produces a +new dataset and supplies `mode: rebuild`, `sideEffectsAllowed: false` to commands +and hooks. This is a cooperative execution control, not hidden business input or +a sandbox. Ordinary recovery does not suppress effects that might still need to +happen, and has no `--force` escape from the later-dataset rule. diff --git a/eslint.config.js b/eslint.config.js index 2690113..fd8fb61 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -1,7 +1,26 @@ -import js from "@eslint/js"; -import globals from "globals"; -import { defineConfig } from "eslint/config"; +import js from '@eslint/js' +import globals from 'globals' +import { defineConfig } from 'eslint/config' export default defineConfig([ - { files: ["**/*.{js,mjs,cjs}"], plugins: { js }, extends: ["js/recommended"], languageOptions: { globals: globals.browser } }, -]); + { + rules: { + curly: ['error', 'all'], + 'brace-style': ['error', 'stroustrup', { allowSingleLine: false }], + 'max-len': ['warn', { + code: 80, + tabWidth: 4, + ignoreUrls: true, + ignoreStrings: true, + ignoreTemplateLiterals: true, + ignoreRegExpLiterals: true + }] + } + }, + { + files: ['**/*.{js,mjs,cjs}'], + plugins: { js }, + extends: ['js/recommended'], + languageOptions: { globals: globals.browser } + } +]) diff --git a/example/README.md b/example/README.md index a258fb8..8a23b67 100644 --- a/example/README.md +++ b/example/README.md @@ -9,15 +9,19 @@ npm install Then convert the dataset `swdb.jsontag` to internal SimplyStore format: ```shell -node ../scripts/convert.mjs swdb.jsontag data.jsontag +npm run build ``` +This creates `data.jsontag`, its indexes, and empty command and status logs +for the first startup. Build requires a new store and refuses to overwrite +existing data or logs. Run it once before starting the example. + Then start the server: ```shell npm start ``` -Now go to `https://localhost:3000/query/` and you can enter queries, e.g.: +Now go to `http://localhost:3000/query/` and you can enter queries, e.g.: ```javascript from(data.people) diff --git a/example/package.json b/example/package.json index 48f013f..34c150e 100644 --- a/example/package.json +++ b/example/package.json @@ -2,6 +2,7 @@ "name": "Example SimplyStore", "main": "server.mjs", "scripts": { + "build": "node ../scripts/convert.mjs swdb.jsontag data.jsontag", "start": "node server.mjs" }, "dependencies": { diff --git a/package.json b/package.json index 9dfad8a..663654d 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,11 @@ "test": "node --test test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs", "test:acid": "node --test test/acid-baseline.mjs", "test:durability": "node --test test/durability-crash.mjs", - "test:handlers": "node --test test/after-change-handlers.mjs" + "test:handlers": "node --test test/after-change-handlers.mjs", + "test:power-loss": "node --test --test-reporter=tap test/power-loss-baseline.mjs", + "test:power-loss:strict": "SIMPLYSTORE_BASELINE_STRICT=1 node --test --test-reporter=tap test/power-loss-baseline.mjs", + "test:recovery": "node --test test/storage.mjs test/admin-recovery.mjs", + "bench:durability": "node --test test/durability-latency.mjs" }, "author": "auke@muze.nl", "license": "MIT", @@ -35,6 +39,9 @@ }, "files": [ "README.md", + "DURABILITY.md", + "docs/recovery.md", + "docs/power-loss-filesystem.md", "LICENSE", "src/", "www/", diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 966bb52..cf0f0d9 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -1,37 +1,65 @@ +import { Buffer } from 'node:buffer' +import process from 'node:process' import JSONTag from '@muze-nl/jsontag' import serialize, { stringify } from '@muze-nl/od-jsontag/src/serialize.mjs' import Parser from '@muze-nl/od-jsontag' import { finalizeIndex } from '../src/index.mjs' import fs from 'node:fs' import path from 'node:path' +import { publishFile, syncFile } from '../src/storage.mjs' +import { acquireOwnership } from '../src/store-ownership.mjs' +import { + appendIntegrityRecord, + getDefaultIntegrityFile +} from '../src/integrity.mjs' -const __dirname = import.meta.dirname; +const __dirname = import.meta.dirname -if (process.argv.length<=3) { - console.log('usage: node ./convert.mjs {inputfile} {outputfile} {indexlib?}') +const integrity = process.argv.includes('--integrity') +const args = process.argv.slice(2).filter(value => value !== '--integrity') +if (args.length < 2) { + console.log( + 'usage: node ./convert.mjs {inputfile} {outputfile} {indexlib?} {schema?} [--integrity]' + ) process.exit() } // parse command line -let inputFile = process.argv[2] -let outputFile = process.argv[3] -let indexFile = process.argv[4] -if (indexFile && indexFile[0]!='/') { - indexFile = process.cwd()+'/'+indexFile -} else if (!indexFile) { - indexFile = __dirname+'/../src/index.mjs' +let inputFile = args[0] +let outputFile = args[1] +let indexFile = args[2] +if (indexFile && indexFile[0] != '/') { + indexFile = process.cwd() + '/' + indexFile } -let schemaFile = process.argv[5] -if (schemaFile && schemaFile[0]!='/') { - schemaFile = process.cwd()+'/'+schemaFile +else if (!indexFile) { + indexFile = __dirname + '/../src/index.mjs' +} +let schemaFile = args[3] +if (schemaFile && schemaFile[0] != '/') { + schemaFile = process.cwd() + '/' + schemaFile } async function main() { + const directory = path.resolve(path.dirname(outputFile)) + const logs = ['command-log.jsontag', 'command-status.jsontag'].map(name => + path.join(directory, name) + ) + const ownership = await acquireOwnership([directory]) + if ( + [outputFile, ...logs, getDefaultIntegrityFile(outputFile)].some(file => + fs.existsSync(file) + ) + ) { + await ownership.release() + throw new Error( + 'Conversion requires new output files; existing store is not overwritten' + ) + } // now create indexes console.log('Using index library:', indexFile) - + const index = await import(indexFile).then(mod => { - return mod.default + return mod.default }) let schema = {} @@ -53,7 +81,8 @@ async function main() { // indexes need the position data which is only available after // parsing the od-jsontag data - const parser = new Parser('',false) // allow mutations + // Keep the default URL base and allow index mutations. + const parser = new Parser(undefined, false) const odData = parser.parse(strData) let meta = { @@ -68,15 +97,35 @@ async function main() { meta.index.id.set(JSONTag.getAttribute(ob, 'id'), ob) } - index.create(odData, meta) + await index.create(odData, meta) console.log('Indexes created') strData = stringify(serialize(odData)) - fs.writeFileSync(outputFile, strData) - // Custom indexes may change record sizes or add records after initial parsing. + await publishFile(outputFile, strData) + // Custom indexes may change record sizes or add records after initial + // parsing. await finalizeIndex(index, strData, meta) - console.log('Converted data written to ',outputFile) + if (integrity) { + await appendIntegrityRecord( + getDefaultIntegrityFile(outputFile), + outputFile, + Buffer.from(strData) + ) + } + for (const file of logs) { + await publishFile(file, '') + } + for (const entry of fs.readdirSync(directory, { withFileTypes: true })) { + if (entry.isFile()) { + await syncFile(path.join(directory, entry.name)) + } + } + await ownership.release() + console.log('Converted data written to ', outputFile) } -main() +main().catch(error => { + console.error('Conversion failed:', error) + process.exitCode = 1 +}) diff --git a/scripts/power-loss-vm/exercise.py b/scripts/power-loss-vm/exercise.py new file mode 100644 index 0000000..429bb8c --- /dev/null +++ b/scripts/power-loss-vm/exercise.py @@ -0,0 +1,132 @@ +import os,socket,struct,threading,subprocess,selectors,time,json,pathlib,shutil,argparse,re,tempfile,hashlib +arguments=argparse.ArgumentParser(description='Disposable ext4 VM power-loss exercise; never uses a host block device') +arguments.add_argument('--qemu',required=True) +arguments.add_argument('--kernel',required=True) +arguments.add_argument('--firmware-dir',required=True) +arguments.add_argument('--bios',required=True) +arguments.add_argument('--library-path',default='') +arguments.add_argument('--work') +args=arguments.parse_args() +ROOT=pathlib.Path(args.work) if args.work else pathlib.Path(tempfile.mkdtemp(prefix='simplystore-vm-exercise-')) +if args.work: ROOT.mkdir(parents=True,exist_ok=False) +QEMU=pathlib.Path(args.qemu).resolve() +repo=pathlib.Path(__file__).resolve().parents[2] +env={**os.environ,**({'LD_LIBRARY_PATH':args.library_path} if args.library_path else {})} +guest=ROOT/'guest' +for directory in ['bin','proc','sys','dev','store','tmp','lib64','app']:(guest/directory).mkdir(parents=True,exist_ok=True) +shutil.copy(shutil.which('busybox'),guest/'bin/busybox') +for command in ['sh','mount','mkdir','sleep','cat','poweroff','ip']:(guest/'bin'/command).symlink_to('busybox') +node=shutil.which('node');shutil.copy(node,guest/'bin/node') +for library in re.findall(r'(/[^\s]+)',subprocess.check_output(['ldd',node],text=True)): + dest=guest/library.lstrip('/');dest.parent.mkdir(parents=True,exist_ok=True);shutil.copy(library,dest) +for folder in ['src','node_modules','www']:shutil.copytree(repo/folder,guest/'app'/folder) +shutil.copy(pathlib.Path(__file__).with_name('guest.mjs'),guest/'app/guest.mjs') +(guest/'app/package.json').write_text('{"type":"module"}') +(guest/'init').write_text("#!/bin/sh\nexport PATH=/bin\nmount -t proc proc /proc\nmount -t sysfs sysfs /sys\nmount -t devtmpfs devtmpfs /dev\nip link set lo up\nmount -t ext4 -o data=ordered,barrier=1 /dev/vda /store || exec sh\ncd /app\nnode /app/guest.mjs\nwhile true; do sleep 100; done\n") +os.chmod(guest/'init',0o755) +with open(ROOT/'initrd.gz','wb') as output: + files=subprocess.Popen(['find','.','-print0'],cwd=guest,stdout=subprocess.PIPE) + cpio=subprocess.Popen(['cpio','--null','-o','--format=newc'],cwd=guest,stdin=files.stdout,stdout=subprocess.PIPE,stderr=subprocess.DEVNULL) + files.stdout.close() + subprocess.run(['gzip','-1'],stdin=cpio.stdout,stdout=output,check=True) + cpio.stdout.close();assert cpio.wait()==0 and files.wait()==0 +metadata={'qemu':subprocess.check_output([str(QEMU),'--version'],env=env,text=True),'node':subprocess.check_output([node,'--version'],text=True),'kernelSha256':hashlib.sha256(pathlib.Path(args.kernel).read_bytes()).hexdigest(),'repositoryHead':subprocess.check_output(['git','rev-parse','HEAD'],cwd=repo,text=True).strip(),'worktreeDiffSha256':hashlib.sha256(subprocess.check_output(['git','diff','HEAD'],cwd=repo)).hexdigest(),'filesystem':'ext4 data=ordered,barrier=1','device':'virtio-blk, QEMU cache=writeback, volatile NBD backend honoring FLUSH/FUA','acceleration':'TCG, 2 CPUs, 1024 MiB','physicalPowerCut':False} +(ROOT/'environment.json').write_text(json.dumps(metadata,indent=2)) +print('Artifacts: '+str(ROOT),flush=True) +class Disk: + def __init__(self,image): + self.image=image;self.data=bytearray(image.read_bytes());self.dirty=set();self.flushes=0;self.writes=0;self.lost=0 + self.sockpath=str(ROOT/'block.sock') + try:os.unlink(self.sockpath) + except FileNotFoundError:pass + self.server=socket.socket(socket.AF_UNIX);self.server.bind(self.sockpath);self.server.listen(1) + self.thread=threading.Thread(target=self.serve,daemon=True);self.thread.start() + def serve(self): + try: + self.conn,_=self.server.accept();c=self.conn + c.sendall(struct.pack('>QQQI',0x4e42444d41474943,0x0000420281861253,len(self.data),1|4|8)+bytes(124)) + def receive(n): + data=b'' + while len(data)IHHQQI',receive(28)) + assert magic==0x25609513 + response=b'' + if kind==0:response=bytes(self.data[offset:offset+length]) + elif kind==1: + self.data[offset:offset+length]=receive(length);pages=set(range(offset//4096,(offset+length+4095)//4096));self.dirty.update(pages);self.writes+=1 + if flags&1:persist(pages) + elif kind==2:break + elif kind==3:self.flushes+=1;persist(set(self.dirty)) + else:raise RuntimeError('Unsupported NBD command '+str(kind)) + c.sendall(struct.pack('>IIQ',0x67446698,0,handle)+response) + except (EOFError,ConnectionResetError,BrokenPipeError,OSError):pass + def stop(self): + self.server.close() + if hasattr(self,'conn'): + try:self.conn.shutdown(socket.SHUT_RDWR) + except OSError:pass + self.conn.close() + self.thread.join(2);self.lost=len(self.dirty) + +def boot(image,scenario,recover=False): + disk=Disk(image) + command=[str(QEMU),'-L',args.firmware_dir,'-vga','none','-machine','pc,accel=tcg','-cpu','max','-m','1024','-smp','2','-display','none','-monitor','none','-serial','stdio','-no-reboot','-net','none','-bios',args.bios,'-kernel',args.kernel,'-initrd',str(ROOT/'initrd.gz'),'-append',f'console=ttyS0 quiet panic=-1 scenario={scenario}','-drive',f'file=nbd:unix:{disk.sockpath},if=virtio,format=raw,cache=writeback'] + p=subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,env=env) + selector=selectors.DefaultSelector();selector.register(p.stdout,selectors.EVENT_READ) + lines=b'';deadline=time.time()+90;found=None + marker=b'VERIFY ' if recover else {'done':b'CUT_DONE','accepted':b'CUT_ACCEPTED','active':b'CUT_ACTIVE','unsynced':b'CUT_UNSYNCED'}[scenario] + try: + while time.time()IHHQQI',0x25609513,0,kind,1,offset,len(data))+data) + assert recv_exact(16)==struct.pack('>IIQ',0x67446698,0,1) +request(1,0,b'A'*4096);request(3);request(1,4096,b'B'*4096) +client.close();disk.stop() +assert control.read_bytes()[:4096]==b'A'*4096 and control.read_bytes()[4096:8192]==bytes(4096) +assert disk.lost==1 +print(json.dumps({'deviceControl':'flushed A survives; unflushed B lost','discardedDirtyPages':disk.lost}),flush=True) +results=[] +for scenario in ['done','accepted','active','unsynced']: + image=ROOT/f'{scenario}.img' + with open(image,'wb') as f:f.truncate(128*1024*1024) + subprocess.run(['mkfs.ext4','-q','-F','-O','^metadata_csum_seed',str(image)],check=True) + cut=boot(image,scenario) + result=json.loads(boot(image,scenario,True).removeprefix('VERIFY ')) + assert not result['errors'],result + if scenario=='done':assert result['ready'] and result['persons']==[{'name':name} for name in ['A','L0','L1','L2','L3']],result + elif scenario=='unsynced':assert result['ready'] and not result['unsyncedPresent'],result + else:assert len(result['commands'])==1 and result['commands'][0]['id']=='A' and result['commands'][0]['status'] in ['accepted','active'],result + results.append({'scenario':scenario,'cut':cut,'recovered':result}) +(ROOT/'results.json').write_text(json.dumps(results,indent=2)) diff --git a/scripts/power-loss-vm/guest.mjs b/scripts/power-loss-vm/guest.mjs new file mode 100644 index 0000000..c4668d4 --- /dev/null +++ b/scripts/power-loss-vm/guest.mjs @@ -0,0 +1,147 @@ +import process from 'node:process' +import fs from 'node:fs/promises' +import JSONTag from '@muze-nl/jsontag' +import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' +import server from './src/server.mjs' +import { publishFile } from './src/storage.mjs' +import { inspectStore } from './src/store-inspection.mjs' +const mode = + (await fs.readFile('/proc/cmdline', 'utf8')).match(/scenario=(\w+)/)?.[1] || + 'done' +const config = { + datafile: '/store/data.jsontag', + commandLog: '/store/command-log.jsontag', + commandStatus: '/store/command-status.jsontag', + commandsFile: '/app/commands.mjs', + indexFile: '/app/index.mjs' +} +console.log( + 'ENVELOPE ' + + JSON.stringify({ + kernel: await fs.readFile('/proc/version', 'utf8'), + mounts: await fs.readFile('/proc/mounts', 'utf8'), + writeCache: await fs.readFile( + '/sys/block/vda/queue/write_cache', + 'utf8' + ) + }) +) +const exists = await fs.access(config.datafile).then( + () => true, + () => false +) +if (exists) { + const report = await inspectStore(config) + console.log( + 'VERIFY ' + + JSON.stringify({ + ready: report.ready, + errors: report.errors, + commands: report.commands.map(c => ({ + id: c.id, + status: c.status, + present: c.present + })), + persons: report.data?.persons, + unsyncedPresent: await fs.access('/store/unsynced').then( + () => true, + () => false + ) + }) + ) + process.exit(0) +} +await publishFile(config.datafile, serialize(JSONTag.parse('{"persons":[]}'))) +await publishFile(config.commandLog, '') +await publishFile(config.commandStatus, '') +if (mode === 'unsynced') { + await fs.writeFile('/store/unsynced', 'not guaranteed durable') + console.log('CUT_UNSYNCED') + await new Promise(() => { + + }) +} +let commandBody = 'data.persons.push(command.value)' +if (mode === 'active') { + commandBody = "console.log('CUT_ACTIVE');while(true){}" +} +else if (mode === 'accepted') { + commandBody = 'while(true){}' +} +const commandModule = `export default {addPerson(data,command){${commandBody}}}` +await fs.writeFile('/app/commands.mjs', commandModule) +await fs.writeFile( + '/app/index.mjs', + 'export default {update(){},create(){},load(){return {}}}' +) +await server.run({ + ...config, + port: 3000, + maxWorkers: 1, + commandTimeout: 0, + wwwroot: '/app/www' +}) +await new Promise(r => setTimeout(r, 100)) +const before = performance.now() +const response = await fetch('http://127.0.0.1:3000/command', { + method: 'POST', + headers: { 'content-type': 'application/jsontag' }, + body: JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) +}) +console.log( + 'ACCEPTED ' + + JSON.stringify({ + http: response.status, + ms: performance.now() - before + }) +) +if (mode === 'accepted') { + console.log('CUT_ACCEPTED') +} +if (mode === 'done') { + for (;;) { + const status = await fetch('http://127.0.0.1:3000/command/A').then(r => + r.json() + ) + if (status.status === 'done') { + break + } + await new Promise(r => setTimeout(r, 10)) + } + const samples = [{ id: 'A', doneMs: performance.now() - before }] + for (let i = 0; i < 4; i++) { + const id = 'L' + i, + start = performance.now() + const accepted = await fetch('http://127.0.0.1:3000/command', { + method: 'POST', + headers: { 'content-type': 'application/jsontag' }, + body: JSONTag.stringify({ + id, + name: 'addPerson', + value: { name: id } + }) + }) + if (accepted.status !== 202) { + throw new Error('Acceptance failed') + } + const acceptMs = performance.now() - start + for (;;) { + const status = await fetch( + 'http://127.0.0.1:3000/command/' + id + ).then(r => r.json()) + if (status.status === 'done') { + break + } + await new Promise(r => setTimeout(r, 10)) + } + samples.push({ id, acceptMs, doneMs: performance.now() - start }) + } + console.log('CUT_DONE ' + JSON.stringify({ samples })) +} +await new Promise(() => { + +}) diff --git a/scripts/recover.mjs b/scripts/recover.mjs new file mode 100644 index 0000000..26fdb12 --- /dev/null +++ b/scripts/recover.mjs @@ -0,0 +1,133 @@ +import process from 'node:process' +import fs from 'node:fs/promises' +import path from 'node:path' +import { + planRecovery, + writePlan, + applyRecovery, + backupStore, + restoreBackup, + releaseOfflineLocks, + finishRecovery, + verifyCandidate +} from '../src/admin-recovery.mjs' +import { inspectStore } from '../src/store-inspection.mjs' + +const [action, ...args] = process.argv.slice(2) +const options = {} +for (let i = 0; i < args.length; i++) { + if (!args[i].startsWith('--')) { + throw new Error(`Expected option, got ${args[i]}`) + } + const key = args[i].slice(2) + const next = args[i + 1] + if (next && !next.startsWith('--')) { + options[key] = args[++i] + } + else { + options[key] = true + } +} +const read = async file => JSON.parse(await fs.readFile(file, 'utf8')) +const required = key => { + if (typeof options[key] !== 'string') { + throw new Error(`--${key} is required`) + } + return options[key] +} +async function main() { + switch (action) { + case 'inspect': { + const plan = await planRecovery(await read(required('store')), { + quiescent: options.quiescent === true + }) + await writePlan(required('out'), plan) + return plan + } + case 'apply': { + const plan = await read(required('plan')) + const to = required('to') + const auditDir = required('audit-dir') + let approveRerun = [] + if (typeof options['approve-rerun'] === 'string') { + approveRerun = options['approve-rerun'] + .split(',') + .filter(Boolean) + } + return applyRecovery(plan, { + to, + auditDir, + approveRerun, + operator: required('operator'), + reason: required('reason') + }) + } + case 'finish': + return finishRecovery(required('audit-dir'), { + operator: required('operator'), + reason: required('reason'), + confirmedStopped: options['confirmed-stopped'] === true + }) + case 'backup': + return backupStore(await read(required('store')), { + to: required('to'), + quiescent: options.quiescent === true + }) + case 'restore': { + const backup = required('backup') + const to = required('to') + const auditDir = required('audit-dir') + let source + if (options.store) { + source = await read(options.store) + } + return restoreBackup(backup, { + to, + auditDir, + source, + sourceQuiescent: options.quiescent === true + }) + } + case 'unlock': { + const store = await read(required('store')) + const release = await releaseOfflineLocks(store, { + operator: required('operator'), + reason: required('reason'), + confirmedStopped: options['confirmed-stopped'] === true + }) + const audit = path.resolve(required('out')) + // writePlan enforces new output outside every source data + // directory. + await writePlan(audit, { + ...release, + config: (await inspectStore(store)).config + }) + await release.finish() + return { audit, released: release.removed.map(item => item.lock) } + } + case 'verify': { + const report = await read(required('report')) + const current = await verifyCandidate(report) + const output = path.resolve(required('out')) + await writePlan(output, { + config: report.config, + kind: 'simplystore-activation-preview', + store: { ...report.config, ...report.code }, + committed: current.committed, + instruction: + 'Stop the old service, preserve its configuration, and explicitly select this store. This command does not deploy or switch a service.' + }) + return { verified: true, activationPreview: output } + } + default: + throw new Error( + 'Usage: recover.mjs inspect|apply|backup|restore|unlock|finish|verify --store config.json ...; see docs/recovery.md' + ) + } +} +main() + .then(result => console.log(JSON.stringify(result, null, 2))) + .catch(error => { + console.error(error.message) + process.exitCode = 1 + }) diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs new file mode 100644 index 0000000..ba77742 --- /dev/null +++ b/src/admin-recovery.mjs @@ -0,0 +1,862 @@ +import { Buffer } from 'node:buffer' +import fs from 'node:fs/promises' +import path from 'node:path' +import { fileURLToPath } from 'node:url' +import JSONTag from '@muze-nl/jsontag' +import { from, _, anyOf, not } from '@muze-nl/jaqt' +import Parser from '@muze-nl/od-jsontag/src/parse.mjs' +import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' +import { + inspectStore, + inventory, + hash, + mutableDirectories, + storePaths +} from './store-inspection.mjs' +import { acquireOwnership } from './store-ownership.mjs' +import { + publishFile, + appendRecord, + durableMkdir, + syncDirectory, + syncFile +} from './storage.mjs' +import { executeWorker } from './execute-worker.mjs' +import { nextActiveCommandStatus } from './recovery.mjs' +import { loadIntegrityManifest, verifyIntegrity } from './integrity.mjs' + +const defaultWorker = fileURLToPath( + new URL('./command-worker.mjs', import.meta.url) +) +const defaultIndex = fileURLToPath(new URL('./index.mjs', import.meta.url)) +const defaultCommands = fileURLToPath( + new URL('./commands.mjs', import.meta.url) +) +function deployment(options) { + return { + commandsFile: path.resolve(options.commandsFile || defaultCommands), + indexFile: path.resolve(options.indexFile || defaultIndex), + commandWorker: path.resolve(options.commandWorker || defaultWorker) + } +} +async function codeHashes(code) { + const entries = Object.entries(code) + const hashes = await Promise.all( + entries.map(async ([key, file]) => { + const bytes = await fs.readFile(file) + return [key, hash(bytes)] + }) + ) + return Object.fromEntries(hashes) +} +async function canonicalNew(file) { + const absolute = path.resolve(file) + try { + await fs.lstat(absolute) + throw new Error(`Destination already exists: ${absolute}`) + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } + return path.join( + await fs.realpath(path.dirname(absolute)), + path.basename(absolute) + ) +} +function outside(target, roots) { + for (const root of roots) { + if ( + target === root || + target.startsWith(root + path.sep) || + root.startsWith(target + path.sep) + ) { + throw new Error( + `Overlapping source/destination: ${target} and ${root}` + ) + } + } +} +async function assertFresh(plan) { + const files = await inventory(plan.config) + if (hash(JSON.stringify(files)) !== plan.fingerprint) { + throw new Error('Stale recovery plan: source inventory changed') + } + if ( + JSON.stringify(await codeHashes(plan.code)) !== + JSON.stringify(plan.codeHashes) + ) { + throw new Error('Stale recovery plan: selected code changed') + } +} +export async function planRecovery(options, { quiescent = false } = {}) { + const report = await inspectStore(options) + const code = deployment(options) + const commands = from(report.commands) + const candidates = commands.where( + anyOf( + { status: anyOf('accepted', 'active') }, + { status: 'done', present: false } + ) + ) + const uncertain = commands.where( + anyOf( + { condition: 'corrupt' }, + { present: Boolean, status: not('done') } + ) + ) + const blockedCandidates = candidates.where({ + laterDatasets: datasets => datasets.length > 0 + }) + + const blocks = [...report.errors] + for (const command of uncertain) { + blocks.push( + `${command.id}: existing uncertain dataset requires diagnosis` + ) + } + for (const command of blockedCandidates) { + const laterIds = command.laterDatasets.join(',') + blocks.push(`${command.id}: later accepted datasets exist: ${laterIds}`) + } + + const rerun = [...candidates.select(_.id)] + const summaries = commands.select(record => { + const summary = { ...record } + delete summary.command + return summary + }) + const selectedCodeHashes = await codeHashes(code) + return { + kind: 'simplystore-recovery-plan', + config: report.config, + code, + codeHashes: selectedCodeHashes, + files: report.files, + fingerprint: report.fingerprint, + actionable: quiescent && blocks.length === 0, + blocks, + rerun, + committed: report.committed, + commands: [...summaries], + warnings: report.warnings, + attestation: + 'Approval asserts source quiescence, complete authoritative history, complete logged command inputs, original command meaning, and assessed external effects.' + } +} +export async function writePlan(file, plan) { + const target = await canonicalNew(file) + const roots = await Promise.all( + mutableDirectories(plan.config).map(dir => fs.realpath(dir)) + ) + outside(target, roots) + await publishFile(target, JSON.stringify(plan, null, 2) + '\n') +} +async function copyStore(config, files, target) { + const directories = [ + ...new Set(Object.keys(files).map(file => path.dirname(file))) + ].sort() + let common = directories[0] + while ( + !directories.every( + dir => + dir === common || + dir.startsWith(common + path.sep) || + common === path.parse(common).root + ) + ) { + common = path.dirname(common) + } + const mapFile = file => path.join(target, path.relative(common, file)) + for (const directory of directories) { + await durableMkdir(path.join(target, path.relative(common, directory))) + } + for (const [file, digest] of Object.entries(files)) { + if (digest === null) { + continue + } + const bytes = await fs.readFile(file) + if (hash(bytes) !== digest) { + throw new Error(`Source changed while copying: ${file}`) + } + await publishFile(mapFile(file), bytes, { + mode: (await fs.stat(file)).mode & 0o777 + }) + } + const result = { + ...config, + datafile: mapFile(config.datafile), + commandLog: mapFile(config.commandLog), + commandStatus: mapFile(config.commandStatus), + integrityFile: mapFile(config.integrityFile), + requiredFiles: config.requiredFiles.map(mapFile) + } + if (config.schemaFile) { + result.schemaFile = mapFile(config.schemaFile) + } + return result +} +async function prepareWorkspace(target, audit, roots) { + target = await canonicalNew(target) + audit = await canonicalNew(audit) + outside(target, roots) + outside(audit, roots) + outside(target, [audit]) + await durableMkdir(target) + await durableMkdir(audit) + return { target, audit } +} + +export async function applyRecovery( + plan, + { to, auditDir, approveRerun = [], operator, reason } = {} +) { + if (plan.kind !== 'simplystore-recovery-plan' || !plan.actionable) { + throw new Error( + 'Plan is not actionable; inspect a quiescent source and resolve blocking evidence' + ) + } + if (!operator || !reason) { + throw new Error('Operator and assessment reason are required') + } + if (JSON.stringify(approveRerun) !== JSON.stringify(plan.rerun)) { + throw new Error( + 'Explicit approval must match the complete ordered rerun list' + ) + } + const sourceOwner = await acquireOwnership(mutableDirectories(plan.config)) + let candidateOwner + try { + await assertFresh(plan) + // Do not trust editable plan fields to bypass the source predicate. + const fresh = await planRecovery( + { ...plan.config, ...plan.code }, + { quiescent: true } + ) + if ( + !fresh.actionable || + JSON.stringify(fresh.rerun) !== JSON.stringify(plan.rerun) + ) { + throw new Error( + 'Source no longer satisfies ordered recovery predicate' + ) + } + const workspace = await prepareWorkspace( + to, + auditDir, + sourceOwner.directories + ) + const { target, audit } = workspace + const rootOwner = await acquireOwnership([target], { + purpose: 'recovery', + auditDir: audit + }) + // A root lock protects partially copied candidates before layout is + // complete. + const original = await copyStore( + plan.config, + plan.files, + path.join(audit, 'original') + ) + const config = await copyStore(plan.config, plan.files, target) + candidateOwner = await acquireOwnership( + mutableDirectories(config).filter(dir => dir !== target), + { ancestorToken: rootOwner.token } + ) + const planHash = hash(JSON.stringify(plan)) + await publishFile( + path.join(audit, 'plan.json'), + JSON.stringify(plan, null, 2) + ) + await publishFile( + path.join(audit, 'original.json'), + JSON.stringify(original, null, 2) + ) + const journal = path.join(audit, 'attempts.jsonl') + await appendRecord( + journal, + JSON.stringify({ + event: 'authorized', + planHash, + operator, + reason, + rerun: plan.rerun, + config, + attestation: plan.attestation + }) + ) + const inspection = await inspectStore(config) + if (inspection.errors.length) { + throw new Error(inspection.errors.join('; ')) + } + const parser = new Parser() + let data + for (const bytes of inspection.buffers) { + data = parser.parse(bytes) + } + // Serialize committed prefix, rebuilding parser indexes without custom + // hooks. + let meta = { + index: { id: new Map() }, + data: path.dirname(config.datafile), + parts: inspection.committed.length + } + const buffers = [serialize(data, { meta })] + if (config.schemaFile) { + meta.schema = JSONTag.parse( + await fs.readFile(config.schemaFile, 'utf8') + ) + } + let expectedManifest = null + if (plan.files[plan.config.integrityFile] != null) { + expectedManifest = await loadIntegrityManifest( + plan.config.integrityFile + ) + } + for (const id of plan.rerun) { + await assertFresh(plan) + const command = fresh.commands.find(c => c.id === id) + const active = nextActiveCommandStatus(id, command.history.at(-1)) + await appendRecord( + journal, + JSON.stringify({ + event: 'attempt', + id, + attempt: active.attempt, + planHash + }) + ) + await appendRecord(config.commandStatus, JSONTag.stringify(active)) + const result = await executeWorker( + plan.code.commandWorker, + { + id, + command: command.line, + meta, + data: buffers, + commandsFile: plan.code.commandsFile, + indexFile: plan.code.indexFile, + datafile: config.datafile, + // Verify a retained digest before allowing replacement + // manifest/done records. + integrityFile: null, + integrityRequired: false + }, + 30000 + ) + if ( + !result || + result.storageFailure || + result.status === 'failed' || + result.status === 'unsafe' || + result.code >= 300 + ) { + await appendRecord( + journal, + JSON.stringify({ + event: 'stopped', + id, + result: result && { + status: result.status, + message: result.message + } + }) + ) + throw new Error( + `Recovery attempt ${id} uncertain or failed; inspect before approving another attempt` + ) + } + for (const file of config.requiredFiles) { + await syncFile(file) + } + if (expectedManifest && command.status === 'done') { + verifyIntegrity( + expectedManifest, + plan.config.integrityFile, + command.file, + result.data, + { required: true } + ) + } + if (config.integrity || expectedManifest) { + const { appendIntegrityRecord } = await import( + './integrity.mjs' + ) + const targetFile = path.join( + path.dirname(config.datafile), + path.basename(command.file) + ) + await appendIntegrityRecord( + config.integrityFile, + targetFile, + result.data + ) + } + await appendRecord( + config.commandStatus, + JSONTag.stringify({ command: id, code: 200, status: 'done' }) + ) + buffers.push(result.data) + Object.assign(meta, result.meta) + await appendRecord( + journal, + JSON.stringify({ event: 'done', id, attempt: active.attempt }) + ) + } + const complete = await inspectStore(config) + if (!complete.ready) { + const problems = from(complete.commands).where({ problem: Boolean }) + const diagnostic = { + errors: complete.errors, + commands: [...problems] + } + const details = JSON.stringify(diagnostic) + throw new Error(`Recovered candidate is not ready: ${details}`) + } + const report = { + kind: 'simplystore-recovery-complete', + planHash, + config, + code: plan.code, + codeHashes: plan.codeHashes, + files: complete.files, + fingerprint: complete.fingerprint, + committed: complete.committed, + operator, + reason, + warnings: complete.warnings + } + await publishFile( + path.join(audit, 'complete.json'), + JSON.stringify(report, null, 2) + ) + await candidateOwner.release() + candidateOwner = null + await rootOwner.release() + return report + } + finally { + // On failure retain candidate locks and its audit. The unchanged source + // can be released. + await sourceOwner.release() + } +} + +export async function backupStore(options, { to, quiescent = false } = {}) { + if (!quiescent) { + throw new Error('Backup requires a stopped/quiescent source') + } + const config = storePaths(options), + owner = await acquireOwnership(mutableDirectories(config)) + try { + const report = await inspectStore(config) + if (!report.ready) { + throw new Error('Backup source is not a complete validated store') + } + const target = await canonicalNew(to) + outside(target, owner.directories) + await durableMkdir(target) + const backupOwner = await acquireOwnership([target]) + const copied = await copyStore( + config, + report.files, + path.join(target, 'store') + ) + const check = await inspectStore(copied) + if (!check.ready) { + throw new Error('Backup reconstruction failed') + } + if ( + hash(JSON.stringify(await inventory(config))) !== report.fingerprint + ) { + throw new Error('Backup source changed') + } + const manifest = { + kind: 'simplystore-backup-complete', + root: target, + config: copied, + files: check.files, + fingerprint: check.fingerprint, + committed: check.committed, + sourceFingerprint: report.fingerprint, + warnings: report.warnings + } + await publishFile( + path.join(target, 'complete.json'), + JSON.stringify(manifest, null, 2) + ) + await backupOwner.release() + return manifest + } + finally { + await owner.release() + } +} + +function backupCoverage(retained, restored) { + const originalBase = retained.files[retained.config.datafile] + const backupBase = restored.files[restored.config.datafile] + const sameBase = originalBase === backupBase + const backupCommands = from(restored.commands) + + function isCovered(command) { + if (!sameBase) { + return false + } + const matching = backupCommands.where({ + id: command.id, + line: command.line, + status: command.status + }) + if (command.status !== 'done' || !command.present) { + return matching.length > 0 + } + return matching.some(backupCommand => { + const originalDigest = retained.files[command.file] + const backupDigest = restored.files[backupCommand.file] + return originalDigest === backupDigest + }) + } + + const missing = from(retained.commands) + .where({ accepted: true }) + .where(not(isCovered)) + .select(_.id) + return { sameBase, missingFromBackup: [...missing] } +} + +export async function restoreBackup( + backupDirectory, + { to, auditDir, source, sourceQuiescent = false } = {} +) { + const backup = await fs.realpath(backupDirectory) + const manifest = JSON.parse( + await fs.readFile(path.join(backup, 'complete.json'), 'utf8') + ) + if (manifest.kind !== 'simplystore-backup-complete') { + throw new Error('Backup is incomplete or unrecognized') + } + const oldRoot = manifest.root + if ( + typeof oldRoot !== 'string' || + !path.isAbsolute(oldRoot) || + manifest.fingerprint !== hash(JSON.stringify(manifest.files)) + ) { + throw new Error('Invalid backup inventory') + } + const rebase = file => { + if ( + typeof file !== 'string' || + file !== path.resolve(file) || + !file.startsWith(oldRoot + path.sep) + ) { + throw new Error('Backup manifest points outside backup') + } + return path.join(backup, path.relative(oldRoot, file)) + } + manifest.files = Object.fromEntries( + Object.entries(manifest.files).map(([file, digest]) => [ + rebase(file), + digest + ]) + ) + manifest.fingerprint = hash(JSON.stringify(manifest.files)) + for (const key of [ + 'datafile', + 'commandLog', + 'commandStatus', + 'integrityFile', + 'schemaFile' + ]) { + if (manifest.config[key]) { + manifest.config[key] = rebase(manifest.config[key]) + } + } + manifest.config.requiredFiles = manifest.config.requiredFiles.map(rebase) + if ( + mutableDirectories(manifest.config).some( + dir => dir !== backup && !dir.startsWith(backup + path.sep) + ) + ) { + throw new Error('Backup configuration points outside backup') + } + if (source && !sourceQuiescent) { + throw new Error('Source comparison requires a stopped/quiescent source') + } + const directories = mutableDirectories(manifest.config) + if (source) { + const sourceConfig = storePaths(source) + directories.push(...mutableDirectories(sourceConfig)) + } + const owner = await acquireOwnership(directories) + try { + const report = await inspectStore(manifest.config) + if (!report.ready || report.fingerprint !== manifest.fingerprint) { + throw new Error('Backup contents differ from completed manifest') + } + const { target, audit } = await prepareWorkspace(to, auditDir, [ + backup, + ...owner.directories + ]) + const rootOwner = await acquireOwnership([target], { + purpose: 'restore', + auditDir: audit + }) + const config = await copyStore(manifest.config, manifest.files, target) + const candidate = await acquireOwnership( + mutableDirectories(config).filter(dir => dir !== target), + { ancestorToken: rootOwner.token } + ) + const check = await inspectStore(config) + if (!check.ready) { + throw new Error('Restored candidate failed validation') + } + let missingFromBackup = null, + sameBase = null + if (source) { + const retained = await inspectStore(source) + const coverage = backupCoverage(retained, check) + sameBase = coverage.sameBase + missingFromBackup = coverage.missingFromBackup + } + let coverage = 'Loss relative to current source is unknown' + if (source) { + coverage = + 'Compared with supplied retained history; ' + + 'completeness still requires external evidence' + } + const complete = { + kind: 'simplystore-restore-complete', + config, + files: check.files, + fingerprint: check.fingerprint, + committed: check.committed, + missingFromBackup, + sameBase, + warnings: check.warnings, + coverage + } + await publishFile( + path.join(audit, 'complete.json'), + JSON.stringify(complete, null, 2) + ) + await candidate.release() + await rootOwner.release() + return complete + } + finally { + await owner.release() + } +} + +// Explicit offline operation: no PID-age or liveness heuristic can authorize +// it. +export async function releaseOfflineLocks( + options, + { operator, reason, confirmedStopped = false } = {} +) { + if (!confirmedStopped || !operator || !reason) { + throw new Error( + 'Offline release requires confirmed stopped writer, operator, and reason' + ) + } + const config = storePaths(options), + report = await inspectStore(config) + const removed = [] + const lockDirectories = new Set( + await Promise.all( + mutableDirectories(config).map(dir => fs.realpath(dir)) + ) + ) + for (const directory of [...lockDirectories]) { + for ( + let parent = path.dirname(directory); + path.dirname(parent) !== parent; + parent = path.dirname(parent) + ) { + try { + await fs.access(path.join(parent, '.simplystore-lock')) + lockDirectories.add(parent) + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } + } + } + for (const directory of [...lockDirectories].sort().reverse()) { + const lock = path.join(directory, '.simplystore-lock') + let entries + try { + entries = await fs.readdir(lock) + } + catch (error) { + if (error.code === 'ENOENT') { + continue + } + throw error + } + if ( + entries.some( + entry => entry !== 'owner.json' && !entry.endsWith('.tmp') + ) + ) { + throw new Error(`Unknown lock contents: ${lock}`) + } + const saved = [] + for (const entry of entries) { + saved.push({ + name: entry, + bytes: (await fs.readFile(path.join(lock, entry))).toString( + 'base64' + ) + }) + } + const ownerRecord = saved.find(item => item.name === 'owner.json') + let owner = null + if (ownerRecord) { + const ownerBytes = Buffer.from(ownerRecord.bytes, 'base64') + owner = JSON.parse(ownerBytes.toString()) + } + if (owner?.purpose === 'recovery' && report.ready) { + let completed + try { + completed = JSON.parse( + await fs.readFile( + path.join(owner.auditDir, 'complete.json'), + 'utf8' + ) + ) + } + catch { + /* Missing report is not completion evidence. */ + } + if (completed?.fingerprint !== report.fingerprint) { + throw new Error( + 'Completed-looking recovery candidate requires finish with its retained audit before unlocking' + ) + } + } + removed.push({ lock, ownerFiles: saved }) + } + // Return preview only; the CLI persists its external audit before calling + // finish. + const commands = report.commands.map(command => { + return { + id: command.id, + status: command.status, + problem: command.problem + } + }) + const inspection = { ready: report.ready, errors: report.errors, commands } + return { + inspection, + operator, + reason, + removed, + async finish() { + for (const { lock, ownerFiles } of removed) { + for (const entry of ownerFiles) { + if ( + ( + await fs.readFile(path.join(lock, entry.name)) + ).toString('base64') !== entry.bytes + ) { + throw new Error('Lock changed after preview') + } + } + for (const entry of ownerFiles) { + await fs.unlink(path.join(lock, entry.name)) + } + await fs.rmdir(lock) + await syncDirectory(path.dirname(lock)) + } + } + } +} + +export async function finishRecovery( + auditDirectory, + { operator, reason, confirmedStopped = false } = {} +) { + if (!confirmedStopped || !operator || !reason) { + throw new Error( + 'Finishing requires confirmed stopped writer and administrator assessment' + ) + } + const audit = await fs.realpath(auditDirectory) + const plan = JSON.parse( + await fs.readFile(path.join(audit, 'plan.json'), 'utf8') + ) + const authorization = JSON.parse( + (await fs.readFile(path.join(audit, 'attempts.jsonl'), 'utf8')).split( + '\n' + )[0] + ) + if (authorization.planHash !== hash(JSON.stringify(plan))) { + throw new Error('Audit authorization does not match retained plan') + } + await assertFresh(plan) + const report = await inspectStore(authorization.config) + if ( + !report.ready || + plan.rerun.some(id => !report.committed.includes(id)) + ) { + throw new Error( + 'Recovery is incomplete; inspect and approve remaining attempts, never automatically rerun' + ) + } + const complete = { + kind: 'simplystore-recovery-complete', + planHash: authorization.planHash, + config: report.config, + code: plan.code, + codeHashes: plan.codeHashes, + files: report.files, + fingerprint: report.fingerprint, + committed: report.committed, + operator, + reason, + warnings: report.warnings + } + await publishFile( + path.join(audit, 'complete.json'), + JSON.stringify(complete, null, 2) + ) + const release = await releaseOfflineLocks(report.config, { + operator, + reason, + confirmedStopped + }) + await publishFile( + path.join(audit, 'finish-unlock.json'), + JSON.stringify(release) + ) + await release.finish() + return complete +} + +export async function verifyCandidate(report) { + if ( + ![ + 'simplystore-recovery-complete', + 'simplystore-restore-complete' + ].includes(report.kind) + ) { + throw new Error('Expected completed recovery/restore report') + } + const current = await inspectStore(report.config) + if (!current.ready || current.fingerprint !== report.fingerprint) { + throw new Error('Candidate changed or is incomplete') + } + if ( + report.code && + JSON.stringify(await codeHashes(report.code)) !== + JSON.stringify(report.codeHashes) + ) { + throw new Error('Selected code changed after recovery') + } + return current +} diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 79f7109..a602493 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -1,8 +1,12 @@ import JSONTag from '@muze-nl/jsontag' -import {getIndex, resultSet, isChanged} from '@muze-nl/od-jsontag/src/symbols.mjs' +import { + getIndex, + resultSet, + isChanged +} from '@muze-nl/od-jsontag/src/symbols.mjs' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' -import writeFileAtomic from 'write-file-atomic' +import { publishFile as writeFileAtomic, storageError } from './storage.mjs' import { faultPoint } from './faults.mjs' import { appendIntegrityRecord } from './integrity.mjs' import { finalizeIndex } from './index.mjs' @@ -14,37 +18,41 @@ let dataspace let datafile, basefile, extension, integrityFile let meta = {} let metaProxy = { - index: { - } + index: {} } const parser = new Parser() parser.immutable = false export const metaIdProxy = { - forEach: (callback) => { - parser.meta.index.id.forEach((ref,id) => { - callback({ - deref: () => { - return resultArr[ref] - } - },id) + forEach: callback => { + parser.meta.index.id.forEach((ref, id) => { + callback( + { + deref: () => { + return resultArr[ref] + } + }, + id + ) }) }, - set: (id,ref) => { + set: (id, ref) => { if (!parser.meta.index.id.has(id)) { if (ref[getIndex]) { parser.meta.index.id.set(id, ref[getIndex]) - } else { + } + else { throw new Error('cannot set index.id for non-proxy') } - } else { + } + else { let line = parser.meta.index.id.get(id) resultArr[line] = ref } }, - get: (id) => { + get: id => { let index = parser.meta.index.id.get(id) - if (index || index===0) { + if (index || index === 0) { return { deref: () => { return resultArr[index] @@ -52,7 +60,7 @@ export const metaIdProxy = { } } }, - has: (id) => { + has: id => { return parser.meta.index.id.has(id) } } @@ -68,7 +76,7 @@ export async function initialize(task) { if (task.meta) { parser.meta = task.meta } - for(let jsontag of task.data) { + for (let jsontag of task.data) { dataspace = parser.parse(jsontag) } resultArr = dataspace[resultSet] @@ -80,7 +88,8 @@ export async function initialize(task) { datafile = task.datafile integrityFile = task.integrityFile extension = datafile.split('.').pop() - basefile = datafile.substring(0, datafile.length - (extension.length + 1)) //+1 for . character + // Include the dot before the extension. + basefile = datafile.substring(0, datafile.length - (extension.length + 1)) commands = await import(task.commandsFile).then(mod => { return mod.default }) @@ -89,57 +98,73 @@ export async function initialize(task) { }) } -export default async function runCommand(commandStr, request) { +export default async function runCommand(commandStr) { let response = { jsontag: true } + let publishing = false try { let task = JSONTag.parse(commandStr, null, metaReadProxy) - if (!task.id) { throw new Error('missing command id')} - if (!task.name) { throw new Error('missing command name parameter')} + if (!task.id) { + throw new Error('missing command id') + } + if (!task.name) { + throw new Error('missing command name parameter') + } if (commands[task.name]) { let time = Date.now() - commands[task.name](dataspace, task, request, metaProxy) - //TODO: if command/task makes no changes, skip updating data.jsontag and writing it, skip response.data - + await commands[task.name](dataspace, task, undefined, metaProxy) + // TODO: if command/task makes no changes, skip updating + // data.jsontag and writing it; skip response.data. + const changes = meta.resultArray.filter(e => e[isChanged]) //FIXME: new entities should also report isChanged = true if (changes.length) { changes.uuid = task.id - index.update(dataspace, meta, changes) + await index.update(dataspace, meta, changes) } - const uint8sab = serialize(dataspace, {meta, changes: true}) // serialize only changes + // Serialize only changes. + const uint8sab = serialize(dataspace, { meta, changes: true }) response.data = uint8sab response.meta = { index: { id: meta.index.id } } - //TODO: write data every x commands or x minutes, in seperate thread? + // TODO: write data every x commands or x minutes, + // in a separate thread? let newfilename = basefile + '.' + task.id + '.' + extension + publishing = true await faultPoint('before-command-changeset-write') await writeFileAtomic(newfilename, uint8sab) - // Final bytes include new records and mutations made by the custom index hook. + // Final bytes include new records and mutations made by the custom + // index hook. await finalizeIndex(index, uint8sab, meta, task.id) if (integrityFile) { - await appendIntegrityRecord(integrityFile, newfilename, uint8sab) + await appendIntegrityRecord( + integrityFile, + newfilename, + uint8sab + ) } await faultPoint('after-command-changeset-write') meta.parts++ response.meta.parts = meta.parts let end = Date.now() - console.log('task time',end-time) - } else { + console.log('task time', end - time) + } + else { console.error('Command not found', task.name) throw { code: 404, - message: "Command "+task.name+" not found" + message: 'Command ' + task.name + ' not found' } } - } catch(err) { + } + catch (err) { console.error('task error', err) - throw err + throw publishing ? storageError(err) : err } return response } diff --git a/src/command-worker.mjs b/src/command-worker.mjs index 4b0deff..aa98509 100644 --- a/src/command-worker.mjs +++ b/src/command-worker.mjs @@ -1,9 +1,19 @@ import { parentPort } from 'node:worker_threads' -import runCommand, { initialize } from '../src/command-worker-module.mjs' +import runCommand, { initialize } from '../src/command-worker-module.mjs' parentPort.on('message', async data => { let result await initialize(data) - result = await runCommand(data.command) + try { + result = await runCommand(data.command) + } + catch (error) { + result = { + status: 'failed', + code: 500, + message: error.message, + storageFailure: Boolean(error.storageFailure) + } + } parentPort.postMessage(result) -}) \ No newline at end of file +}) diff --git a/src/execute-worker.mjs b/src/execute-worker.mjs new file mode 100644 index 0000000..75d0f9d --- /dev/null +++ b/src/execute-worker.mjs @@ -0,0 +1,55 @@ +import { Worker } from 'node:worker_threads' + +export function executeWorker(filename, task, timeout = 30000) { + return new Promise((resolve, reject) => { + const worker = new Worker(filename) + let settled = false, + timer + const finish = async (settle, result) => { + if (settled) { + return + } + settled = true + clearTimeout(timer) + try { + await worker.terminate() + } + catch (error) { + reject(error) + return + } + settle(result) + } + worker.on('message', result => { + void finish(resolve, result) + }) + worker.on('error', error => { + void finish(reject, error) + }) + worker.on('exit', code => { + if (!settled) { + void finish( + reject, + new Error( + `Command worker exited without a result (${code})` + ) + ) + } + }) + if (timeout) { + timer = setTimeout(() => { + void finish(resolve, { + status: 'unsafe', + code: 504, + message: `command worker timed out after ${timeout}ms` + }) + }, timeout) + } + try { + worker.postMessage(task) + } + catch (error) { + void finish(reject, error) + } + }) +} diff --git a/src/index.id.mjs b/src/index.id.mjs index bf20991..6383758 100644 --- a/src/index.id.mjs +++ b/src/index.id.mjs @@ -1,11 +1,13 @@ +import { publishFileSync } from './storage.mjs' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' import { getIndex } from '@muze-nl/od-jsontag/src/symbols.mjs' export default { create(data, meta) { - console.log('creating '+meta.data+'/index.id.json') - // jsontag parse automatically fills meta.index.id, so no need to create anything + console.log('creating ' + meta.data + '/index.id.json') + // jsontag parse automatically fills meta.index.id, so no need to create + // anything // just store meta.index.id in index.id.json const index = {} for (const key of meta.index.id.keys()) { @@ -18,7 +20,7 @@ export default { } index[key] = entity[getIndex] } - fs.writeFileSync(meta.data+'/index.id.json', JSON.stringify(index)) + publishFileSync(meta.data + '/index.id.json', JSON.stringify(index)) }, update(data, meta, changes) { if (!changes.length) { @@ -31,15 +33,19 @@ export default { index[id] = entry[getIndex] } } - fs.writeFileSync(meta.data+'/index.id.'+changes.uuid+'.json', JSON.stringify(index)) + publishFileSync( + meta.data + '/index.id.' + changes.uuid + '.json', + JSON.stringify(index) + ) }, - load(uuid=null) { + load(uuid = null) { let filename if (!uuid) { filename = 'index.id.json' - } else { - filename = 'index.id.'+filename+'.json' } - return JSON.parse(fs.readFileSync(meta.data+filename)) + else { + filename = 'index.id.' + filename + '.json' + } + return JSON.parse(fs.readFileSync(meta.data + filename)) } } diff --git a/src/index.offset.mjs b/src/index.offset.mjs index c2a9b2a..99b6c7d 100644 --- a/src/index.offset.mjs +++ b/src/index.offset.mjs @@ -1,30 +1,51 @@ +import { publishFileSync } from './storage.mjs' import fs from 'fs' import path from 'node:path' import { Buffer } from 'node:buffer' -import writeFileAtomic from 'write-file-atomic' +import { publishFile as writeFileAtomic } from './storage.mjs' import { getIndex, position } from '@muze-nl/od-jsontag/src/symbols.mjs' import { scanOdJsonTagRecords } from './recovery.mjs' export default { async writeSerialized(buffer, meta, uuid = null) { - const filename = path.join(meta.data, uuid === null ? 'index.offset.json' : `index.offset.${uuid}.json`) - const bytes = typeof buffer === 'string' ? Buffer.from(buffer) : Buffer.from(buffer.buffer, buffer.byteOffset, buffer.byteLength) + let name = 'index.offset.json' + if (uuid !== null) { + name = `index.offset.${uuid}.json` + } + const filename = path.join(meta.data, name) + let bytes + if (typeof buffer === 'string') { + bytes = Buffer.from(buffer) + } + else { + bytes = Buffer.from( + buffer.buffer, + buffer.byteOffset, + buffer.byteLength + ) + } const index = {} - scanOdJsonTagRecords(bytes, filename, 'offset source OD-JSONTag data', (record, start, end) => { - index[record] = [start, end] - }) + scanOdJsonTagRecords( + bytes, + filename, + 'offset source OD-JSONTag data', + (record, start, end) => { + index[record] = [start, end] + } + ) await writeFileAtomic(filename, JSON.stringify(index)) }, create(data, meta) { - console.log('creating '+meta.data+'/index.offset.json') - // jsontag parse automatically fills meta.index.offset, so no need to create anything + console.log('creating ' + meta.data + '/index.offset.json') + // jsontag parse automatically fills meta.index.offset, so no need to + // create anything const index = {} const max = meta.resultArray.length - for (let i=0;i= maxCrashAttempts) { - nextStatus = { - command: commandId, - code: 500, - status: unsafeCommandStatus, - message: `Command marked unsafe after ${attempt} crashed active attempt(s)`, - attempt - } - } else { - nextStatus = { - command: commandId, - code: 202, - status: pendingCommandStatus, - message: `Retrying command after ${attempt} crashed active attempt(s)`, - attempt - } - } - status.set(commandId, nextStatus) - await appendFile(commandStatusFile, JSONTag.stringify(nextStatus)) +export async function recoverActiveCommands(status) { + if ( + [...status.values()].some( + command => command?.status === activeCommandStatus + ) + ) { + throw new RecoveryIntegrityError( + 'Active command requires administrator assessment; automatic rerun is disabled' + ) } - return status } @@ -160,27 +176,44 @@ export function nextActiveCommandStatus(commandId, currentStatus) { export function getChangesetPath(dataFile, commandId) { const extension = dataFile.split('.').pop() - const basefile = dataFile.substring(0, dataFile.length - (extension.length + 1)) + const basefile = dataFile.substring( + 0, + dataFile.length - (extension.length + 1) + ) return `${basefile}.${commandId}.${extension}` } export function assertChangesetExists(dataFile, commandId) { const changesetPath = getChangesetPath(dataFile, commandId) if (!fs.existsSync(changesetPath)) { - throw new Error(`Missing changeset for committed command ${commandId}: ${changesetPath}`) + throw new Error( + `Missing changeset for committed command ${commandId}: ${changesetPath}` + ) } return changesetPath } -export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag data') { +export function assertOdJsonTagFraming( + buffer, + file, + recordKind = 'OD-JSONTag data' +) { scanOdJsonTagRecords(buffer, file, recordKind) } -export function scanOdJsonTagRecords(buffer, file, recordKind = 'OD-JSONTag data', onRecord) { +export function scanOdJsonTagRecords( + buffer, + file, + recordKind = 'OD-JSONTag data', + onRecord +) { let offset = 0 let record = 0 while (offset < buffer.length) { - while (offset < buffer.length && (buffer[offset] === 10 || buffer[offset] === 13)) { + while ( + offset < buffer.length && + (buffer[offset] === 10 || buffer[offset] === 13) + ) { offset++ } if (offset >= buffer.length) { @@ -189,43 +222,74 @@ export function scanOdJsonTagRecords(buffer, file, recordKind = 'OD-JSONTag data if (buffer[offset] === 43) { offset++ const skipStart = offset - while (offset < buffer.length && buffer[offset] >= 48 && buffer[offset] <= 57) { + while ( + offset < buffer.length && + buffer[offset] >= 48 && + buffer[offset] <= 57 + ) { offset++ } - if (offset === skipStart || (offset < buffer.length && buffer[offset] !== 10 && buffer[offset] !== 13)) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: malformed skip record`, { - file, - recordKind - }) + if ( + offset === skipStart || + (offset < buffer.length && + buffer[offset] !== 10 && + buffer[offset] !== 13) + ) { + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: malformed skip record`, + { + file, + recordKind + } + ) } - record += Number.parseInt(buffer.subarray(skipStart, offset).toString('utf8'), 10) + record += Number.parseInt( + buffer.subarray(skipStart, offset).toString('utf8'), + 10 + ) continue } if (buffer[offset] !== 40) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: expected record length`, { - file, - recordKind - }) + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: expected record length`, + { + file, + recordKind + } + ) } offset++ const lengthStart = offset - while (offset < buffer.length && buffer[offset] >= 48 && buffer[offset] <= 57) { + while ( + offset < buffer.length && + buffer[offset] >= 48 && + buffer[offset] <= 57 + ) { offset++ } if (offset === lengthStart || buffer[offset] !== 41) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: malformed record length`, { - file, - recordKind - }) + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: malformed record length`, + { + file, + recordKind + } + ) } - const payloadLength = Number.parseInt(buffer.subarray(lengthStart, offset).toString('utf8'), 10) + const payloadLength = Number.parseInt( + buffer.subarray(lengthStart, offset).toString('utf8'), + 10 + ) offset++ const payloadEnd = offset + payloadLength if (payloadEnd > buffer.length) { - throw new RecoveryIntegrityError(`Invalid ${recordKind}: truncated record payload`, { - file, - recordKind - }) + throw new RecoveryIntegrityError( + `Invalid ${recordKind}: truncated record payload`, + { + file, + recordKind + } + ) } onRecord?.(record++, offset, payloadEnd) offset = payloadEnd diff --git a/src/server.mjs b/src/server.mjs index 5ce4609..51d4929 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -1,14 +1,25 @@ +import { setImmediate } from 'node:timers' import express from 'express' import fs from 'fs' import JSONTag from '@muze-nl/jsontag' +import { from, _, anyOf, not } from '@muze-nl/jaqt' import WorkerPool from './workerPool.mjs' import { Worker } from 'worker_threads' import { fileURLToPath } from 'url' -import {appendFile} from './util.mjs' +import { appendFile } from './util.mjs' import path from 'path' import httpStatusCodes from './statusCodes.mjs' import process from 'node:process' -import { getCommittedCommandIds, loadCommandLog, loadCommandStatus, nextActiveCommandStatus, pendingCommandStatus, recoverActiveCommands, unsafeCommandStatus } from './recovery.mjs' +import { nextActiveCommandStatus } from './recovery.mjs' +import { serialWriter, syncFile } from './storage.mjs' +import { + inspectStore, + storePaths, + mutableDirectories, + validCommandId +} from './store-inspection.mjs' +import { acquireOwnership } from './store-ownership.mjs' +import { executeWorker } from './execute-worker.mjs' import { assertRuntimeEnvironmentConfiguration } from './runtime-environment.mjs' import { faultPoint } from './faults.mjs' import { getDefaultIntegrityFile } from './integrity.mjs' @@ -31,12 +42,19 @@ class WorkerTimeoutError extends Error { } function normalizeWorkerTimeout(name, value, defaultValue) { - if (value === 0 || value === false || value === null || value === Infinity) { + if ( + value === 0 || + value === false || + value === null || + value === Infinity + ) { return 0 } const timeout = value ?? defaultValue if (!Number.isFinite(timeout) || timeout < 0) { - throw new Error(`${name} must be a non-negative finite number, 0, false, null, or Infinity`) + throw new Error( + `${name} must be a non-negative finite number, 0, false, null, or Infinity` + ) } return timeout } @@ -45,61 +63,133 @@ async function main(options) { assertRuntimeEnvironmentConfiguration() if (!options) { options = {} - } - const port = options.port || 3000 - const datafile = options.datafile || './data.od-jsontag' - const schemaFile = options.schemaFile || null - const wwwroot = options.wwwroot || __dirname+'/www' - const maxWorkers = options.maxWorkers || 8 - const queryWorker = options.queryWorker || __dirname+'/src/query-worker.mjs' - const loadWorker = options.loadWorker || __dirname+'/src/load-worker.mjs' - const commandWorker = options.commandWorker || __dirname+'/src/command-worker.mjs' - const commandsFile = options.commandsFile || __dirname+'/src/commands.mjs' - const indexFile = options.indexFile || __dirname+'/src/index.mjs' - const commandLog = options.commandLog || './command-log.jsontag' + } + const port = options.port || 3000 + const datafile = options.datafile || './data.od-jsontag' + const schemaFile = options.schemaFile || null + const wwwroot = options.wwwroot || __dirname + '/www' + const maxWorkers = options.maxWorkers || 8 + const queryWorker = + options.queryWorker || __dirname + '/src/query-worker.mjs' + const loadWorker = options.loadWorker || __dirname + '/src/load-worker.mjs' + const commandWorker = + options.commandWorker || __dirname + '/src/command-worker.mjs' + const commandsFile = options.commandsFile || __dirname + '/src/commands.mjs' + const indexFile = options.indexFile || __dirname + '/src/index.mjs' + const commandLog = options.commandLog || './command-log.jsontag' const commandStatus = options.commandStatus || './command-status.jsontag' - const integrityFile = options.integrityFile || getDefaultIntegrityFile(datafile) - const integrityEnabled = Boolean(options.integrity || options.integrityFile || fs.existsSync(integrityFile)) - const maxCommandCrashAttempts = options.maxCommandCrashAttempts ?? 2 - const access = options.access || null - const timeout = options.timeout || 1000 - const slowTimeout = options.slowTimeout || 10000 - const commandTimeout = normalizeWorkerTimeout('commandTimeout', options.commandTimeout, defaultCommandTimeout) - const loadTimeout = normalizeWorkerTimeout('loadTimeout', options.loadTimeout, defaultLoadTimeout) + const integrityFile = + options.integrityFile || getDefaultIntegrityFile(datafile) + const integrityEnabled = Boolean( + options.integrity || + options.integrityFile || + fs.existsSync(integrityFile) + ) + const activeIntegrityFile = integrityEnabled ? integrityFile : null + const access = options.access || null + const timeout = options.timeout || 1000 + const slowTimeout = options.slowTimeout || 10000 + const commandTimeout = normalizeWorkerTimeout( + 'commandTimeout', + options.commandTimeout, + defaultCommandTimeout + ) + const loadTimeout = normalizeWorkerTimeout( + 'loadTimeout', + options.loadTimeout, + defaultLoadTimeout + ) server.get('/', serveHomepage) // allow access to raw body, used to parse a query send as post body - server.use(express.raw({ - type: () => true, // parse body on all requests - limit: '50MB' - })) - - let status = loadCommandStatus(commandStatus) - status = await recoverActiveCommands(status, commandStatus, { - maxCrashAttempts: maxCommandCrashAttempts - }) + server.use( + express.raw({ + type: () => true, // parse body on all requests + limit: '50MB' + }) + ) + const storeOptions = { + ...options, + datafile, + commandLog, + commandStatus, + integrity: integrityEnabled + } + if (integrityEnabled) { + storeOptions.integrityFile = integrityFile + } + const config = storePaths(storeOptions) + const ownership = await acquireOwnership(mutableDirectories(config)) + let inspection try { - let data = await loadData(getCommittedCommandIds(status)) // only completed command ids are used to generate filenames of committed changes + inspection = await inspectStore(config) + if (!inspection.ready) { + const commands = from(inspection.commands) + + const problems = commands + .where({ problem: Boolean }) + .select(_.problem) + + const pending = commands + .where({ status: anyOf('accepted', 'active') }) + .select(command => { + return `${command.id}: ${command.status}; administrator assessment required` + }) + + const uncommitted = commands + .where({ present: Boolean, status: not('done') }) + .select(command => { + return `${command.id}: uncommitted dataset` + }) + + const reasons = [ + ...inspection.errors, + ...problems, + ...pending, + ...uncommitted + ] + + const details = reasons.join('; ') + throw new Error(`Administrative recovery required: ${details}`) + } + for (const [file, digest] of Object.entries(inspection.files)) { + if (digest !== null) { + await syncFile(file) + } + } + const data = await loadData(inspection.committed) jsontagBuffers = [data.data] meta = data.meta - } catch(err) { - console.error('ERROR: SimplyStore cannot load '+datafile, err) - process.exit(1) + } + catch (error) { + await ownership.release() + throw error + } + const status = new Map( + inspection.commands.map(c => [c.id, c.history.at(-1)]) + ) + const commandQueue = [] + const acceptSerial = serialWriter() + let storageFailed = false, + closing = false + let runner = Promise.resolve() + function failStorage(error) { + if (storageFailed) { + return + } + storageFailed = true + console.error( + 'Storage outcome uncertain; stopping mutation. Administrator recovery required:', + error + ) + // Retain ownership evidence; a process exit must never imply a + // rollback. + setImmediate(() => process.exit(1)) } - let commandQueue = loadCommandLog(status, commandLog, { - meta, - data:jsontagBuffers, - commandsFile, - indexFile, - datafile, - integrityFile: integrityEnabled ? integrityFile : null, - integrityRequired: integrityEnabled - }) - - const queryWorkerInitTask = () => { + const queryWorkerInitTask = () => { return { name: 'init', req: { @@ -117,52 +207,75 @@ async function main(options) { return result } - let queryWorkerPool = new WorkerPool(maxWorkers, queryWorker, queryWorkerInitTask()) - let slowQueryWorkerPool = new WorkerPool(1, queryWorker, slowQueryWorkerInitTask()) - let commandWorkerInstance + let queryWorkerPool = new WorkerPool( + maxWorkers, + queryWorker, + queryWorkerInitTask() + ) + let slowQueryWorkerPool = new WorkerPool( + 1, + queryWorker, + slowQueryWorkerInitTask() + ) let commandRunnerActive = false - server.get('/query/', (req,next) => handleGetQuery(req,next)) - server.get('/query/*remainder', (req,next) => handleGetQuery(req,next)) - server.get('/slowquery/', (req,next) => handleSlowGetQuery(req,next)) - server.get('/slowquery/*remainder', (req,next) => handleSlowGetQuery(req,next)) - server.post('/query/', (req,next) => handlePostQuery(req,next)) - server.post('/query/*remainder', (req,next) => handlePostQuery(req,next)) - server.post('/slowquery/', (req,next) => handleSlowPostQuery(req,next)) - server.post('/slowquery/*remainder', (req,next) => handleSlowPostQuery(req,next)) - server.post('/command', (req,next) => handlePostCommand(req,next)) - server.get('/command/:id', (req,next) => handleGetCommand(req,next)) + server.get('/query/', (req, next) => handleGetQuery(req, next)) + server.get('/query/*remainder', (req, next) => handleGetQuery(req, next)) + server.get('/slowquery/', (req, next) => handleSlowGetQuery(req, next)) + server.get('/slowquery/*remainder', (req, next) => + handleSlowGetQuery(req, next) + ) + server.post('/query/', (req, next) => handlePostQuery(req, next)) + server.post('/query/*remainder', (req, next) => handlePostQuery(req, next)) + server.post('/slowquery/', (req, next) => handleSlowPostQuery(req, next)) + server.post('/slowquery/*remainder', (req, next) => + handleSlowPostQuery(req, next) + ) + server.post('/command', (req, next) => handlePostCommand(req, next)) + server.get('/command/:id', (req, next) => handleGetCommand(req, next)) server.use(express.static(wwwroot)) - try { - await fetch(`http://localhost:${port}`, { - signal: AbortSignal.timeout(2000) - }) - console.error(`Port ${port} is already occupied, aborting.`) - process.exit() - } catch { + const listener = server.listen(port, () => { + console.log('SimplyStore listening on port ' + port) + }) + listener.on('error', error => { + failStorage(error) + }) + async function shutdown() { + if (closing) { + return + } + closing = true + listener.close() try { - await drainCommandQueue() - } catch { - // console.log(err) // ignore errors here, already logged to console + await acceptSerial(async () => { + + }) + await runner + queryWorkerPool.close() + slowQueryWorkerPool.close() + if (!storageFailed) { + await ownership.release() + } + process.exit(storageFailed ? 1 : 0) + } + catch (error) { + failStorage(error) } - server.listen(port, () => { - console.log('SimplyStore listening on port '+port) - let used = Math.round(process.memoryUsage().rss / 1024 / 1024); - console.log(`(${used} MB)`); - }) } + process.once('SIGTERM', shutdown) + process.once('SIGINT', shutdown) /* ------ */ function serveHomepage(req, res) { - res.setHeader('content-type', 'text/html'); - res.send(fs.readFileSync(wwwroot+'/home.html')) + res.setHeader('content-type', 'text/html') + res.send(fs.readFileSync(wwwroot + '/home.html')) } function loadData(commands) { - return new Promise((resolve,reject) => { + return new Promise((resolve, reject) => { let worker = new Worker(loadWorker) let settled = false let timeoutId @@ -179,7 +292,10 @@ async function main(options) { } if (loadTimeout) { timeoutId = setTimeout(() => { - const error = new WorkerTimeoutError('load worker', loadTimeout) + const error = new WorkerTimeoutError( + 'load worker', + loadTimeout + ) void finish(reject, error) }, loadTimeout) } @@ -191,68 +307,80 @@ async function main(options) { }) try { worker.postMessage({ - dataFile:datafile, + dataFile: datafile, indexFile, schemaFile, commands, - integrityFile: integrityEnabled ? integrityFile : null, + integrityFile: activeIntegrityFile, integrityRequired: integrityEnabled }) - } catch (error) { + } + catch (error) { void finish(reject, error) } }) } - async function handleGetQuery(req, res, pool=null) { + async function handleGetQuery(req, res, pool = null) { let start = Date.now() if (!pool) { pool = queryWorkerPool } - if ( !accept(req,res, - ['application/jsontag','application/json','text/html','text/javascript','image/*'], - function(req, res, accept) { - let result = true - switch(accept) { - case 'text/html': - case 'image/*': - case 'text/javascript': - handleWebRequest(req,res,{root:wwwroot}); - result = false - break + if ( + !accept( + req, + res, + [ + 'application/jsontag', + 'application/json', + 'text/html', + 'text/javascript', + 'image/*' + ], + function (req, res, accept) { + let result = true + switch (accept) { + case 'text/html': + case 'image/*': + case 'text/javascript': + handleWebRequest(req, res, { root: wwwroot }) + result = false + break + } + return result } - return result - } - )) { + ) + ) { // done return } let path = req.params.remainder?.join('/') || '/' - console.log('query',path) + console.log('query', path) let request = { method: req.method, url: req.originalUrl, query: req.query, path: path } - if (accept(req,res,['application/jsontag'])) { + if (accept(req, res, ['application/jsontag'])) { request.jsontag = true } try { let result = await pool.run('query', request, { timeout }) sendResponse(result, res) - } catch(error) { + } + catch (error) { sendError(error, res) } let end = Date.now() - console.log(path, (end-start), process.memoryUsage()) + console.log(path, end - start, process.memoryUsage()) } async function handleSlowGetQuery(req, res) { return handleGetQuery(req, res, slowQueryWorkerPool) } - async function handlePostQuery(req,res,pool=null, slowTimeout=null) { + async function handlePostQuery(req, res, pool = null, slowTimeout = null) { if (!pool) { pool = queryWorkerPool } @@ -260,10 +388,15 @@ async function main(options) { slowTimeout = timeout } let start = Date.now() - if ( !accept(req,res, - ['application/jsontag','application/json']) - ) { - sendError({code:406, message:'Not Acceptable',accept:['application/json','application/jsontag']},res) + if (!accept(req, res, ['application/jsontag', 'application/json'])) { + sendError( + { + code: 406, + message: 'Not Acceptable', + accept: ['application/json', 'application/jsontag'] + }, + res + ) return } let path = req.params.remainder?.join('/') || '/' @@ -274,18 +407,19 @@ async function main(options) { path: path, body: req.body.toString() } - if (accept(req,res,['application/jsontag'])) { + if (accept(req, res, ['application/jsontag'])) { request.jsontag = true } try { - let result = await pool.run('query', request, {slowTimeout}) + let result = await pool.run('query', request, { slowTimeout }) sendResponse(result, res) - } catch(error) { + } + catch (error) { sendError(error, res) } let end = Date.now() - console.log(path, (end-start), process.memoryUsage()) -// queryWorkerPool.memoryUsage() + console.log(path, end - start, process.memoryUsage()) + // queryWorkerPool.memoryUsage() } async function handleSlowPostQuery(req, res) { @@ -293,205 +427,194 @@ async function main(options) { } async function handlePostCommand(req, res) { - let commandId = await checkCommand(req, res) - if (!commandId) { - return + if (storageFailed || closing) { + return sendResponse( + { + code: 503, + body: JSON.stringify({ message: 'Store is unavailable' }) + }, + res + ) } try { - let commandStr = req.body.toString() - let request = { - method: req.method, - url: req.originalUrl, - query: req.query - } - - commandQueue.push({ - id:commandId, - command:commandStr, - request, - meta, - data:jsontagBuffers, - commandsFile, - indexFile, - datafile, - integrityFile: integrityEnabled ? integrityFile : null, - integrityRequired: integrityEnabled + await acceptSerial(async () => { + if (storageFailed) { + throw new Error('Store is unavailable') + } + if (closing) { + return sendResponse( + { + code: 503, + body: JSON.stringify({ + message: 'Store is closing' + }) + }, + res + ) + } + const accepted = await checkCommand(req, res) + if (!accepted) { + return + } + if (storageFailed) { + throw new Error('Store failed during acceptance') + } + commandQueue.push({ id: accepted.id, command: accepted.line }) + sendResponse( + { + code: 202, + body: JSON.stringify(status.get(accepted.id)) + }, + res + ) }) - await drainCommandQueue() - } catch(err) { - let s = {code:err.code||500, status:'failed', message:err.message, details:err.details} - status.set(commandId, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:commandId}, s))) - console.error('ERROR: SimplyStore cannot run command ', commandId, err) + void drainCommandQueue() + } + catch (error) { + if (!res.headersSent) { + sendResponse( + { + code: 500, + body: JSON.stringify({ + message: 'Storage outcome uncertain' + }) + }, + res + ) + } + failStorage(error) } } function handleGetCommand(req, res) { if (status.has(req.params.id)) { let result = status.get(req.params.id) - sendResponse({ - jsontag: false, - body: JSON.stringify(result) - },res) - } else { - sendResponse({ - code: 404, - jsontag: false, - body: JSON.stringify({code: 404, message: "Command not found", details: req.params.id}) - }, res) + sendResponse( + { + jsontag: false, + body: JSON.stringify(result) + }, + res + ) + } + else { + sendResponse( + { + code: 404, + jsontag: false, + body: JSON.stringify({ + code: 404, + message: 'Command not found', + details: req.params.id + }) + }, + res + ) } } - async function drainCommandQueue() { - if (commandRunnerActive) { - return + function drainCommandQueue() { + if (commandRunnerActive || storageFailed) { + return runner } commandRunnerActive = true - try { - let result - do { - result = await runNextCommand() - } while(result) - } finally { - commandRunnerActive = false - } - } - - async function runNextCommand() { - return new Promise((mainResolve, mainReject) => { - if (commandWorkerInstance) { - mainResolve(false) - return - } - let command = commandQueue.shift() - while (command && status.get(command.id)?.status !== pendingCommandStatus) { - console.log('skipping non-pending command', command.id) - command = commandQueue.shift() - } - if (command) { - console.log('starting command',command.id) - let start = (resolve, reject) => { - const worker = new Worker(commandWorker) - commandWorkerInstance = worker - let settled = false - let timeoutId - const finish = async (settle, value) => { - if (settled) { - return - } - settled = true - if (timeoutId) { - clearTimeout(timeoutId) + runner = (async () => { + try { + while (commandQueue.length && !storageFailed) { + const command = commandQueue.shift() + console.log('starting command', command.id) + const active = nextActiveCommandStatus( + command.id, + status.get(command.id) + ) + await appendFile(commandStatus, JSONTag.stringify(active)) + if (storageFailed) { + throw new Error('Store failed before execution') + } + status.set(command.id, active) + await faultPoint( + 'after-active-status-before-command-worker' + ) + const result = await executeWorker( + commandWorker, + { + ...command, + meta, + data: jsontagBuffers, + commandsFile, + indexFile, + datafile, + integrityFile: activeIntegrityFile, + integrityRequired: integrityEnabled + }, + commandTimeout + ) + if (storageFailed) { + throw new Error('Store failed during execution') + } + if (result?.storageFailure) { + throw new Error( + result.message || 'Worker persistence failure' + ) + } + if ( + !result || + result.status === 'failed' || + result.status === 'unsafe' || + result.code >= 300 + ) { + let terminalStatus = 'failed' + if (result?.status === 'unsafe') { + terminalStatus = 'unsafe' } - await worker.terminate() - if (commandWorkerInstance === worker) { - commandWorkerInstance = null + const terminal = { + command: command.id, + status: terminalStatus, + code: result?.code || 500, + message: result?.message || 'Command failed', + attempt: active.attempt } - settle(value) + await appendFile( + commandStatus, + JSONTag.stringify(terminal) + ) + status.set(command.id, terminal) + continue } - if (commandTimeout) { - timeoutId = setTimeout(() => { - const error = new WorkerTimeoutError('command worker', commandTimeout) - void finish(resolve, { - code: error.code, - status: unsafeCommandStatus, - message: error.message, - details: { - timeout: error.timeout, - workerKind: error.workerKind - } - }) - }, commandTimeout) + for (const file of config.requiredFiles) { + await syncFile(file) } - worker.on('message', result => { - void finish(resolve, result) - }) - worker.on('error', error => { - void finish(reject, error) - }) - try { - worker.postMessage(command) - } catch (error) { - void finish(reject, error) + await faultPoint('before-command-done-status') + const done = { + command: command.id, + code: 200, + status: 'done' } - } - const activeStatus = nextActiveCommandStatus(command.id, status.get(command.id)) - status.set(command.id, activeStatus) - void (async () => { - await appendFile(commandStatus, JSONTag.stringify(activeStatus)) - await faultPoint('after-active-status-before-command-worker') - start( - // resolve() - async (data) => { - let s - if (data?.status === unsafeCommandStatus) { - s = { - code: data.code || 504, - status: unsafeCommandStatus, - message: data.message, - details: data.details, - attempt: activeStatus.attempt - } - status.set(command.id, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - mainResolve(s) - } else if (!data || (data.code>=300 && data.code<=499)) { - console.error('ERROR: SimplyStore cannot run command ', command.id, data) - if (!data?.code) { - s = {code: 500, status: "failed"} - } else { - s = {code: data.code, status: "failed", message: data.message, details: data.details} - } - status.set(command.id, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - mainReject(s) - } else { - await faultPoint('before-command-done-status') - s = {code: 200, status: "done"} - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - await faultPoint('after-command-done-status-before-query-update') - status.set(command.id, s) - if (data.data) { // data has changed, commands may do other things instead of changing data - jsontagBuffers.push(data.data) // push changeset to jsontagBuffers so that new query workers get all changes from scratch - Object.assign(meta, data.meta) - const updateTask = { - name: 'update', - req: { - body: jsontagBuffers[jsontagBuffers.length-1], // only add the last change, update tasks for earlier changes have already been sent - meta - } - } - queryWorkerPool.update(updateTask) - slowQueryWorkerPool.update(updateTask) - } - mainResolve(s) - } - }, - //reject() - async (error) => { - let s = {status: "failed", code: error.code, message: error.message, details: error.details} - status.set(command.id, s) - await appendFile(commandStatus, JSONTag.stringify(Object.assign({command:command.id}, s))) - console.log('command error', command.id, error) - mainReject(s) - } + await appendFile(commandStatus, JSONTag.stringify(done)) + await faultPoint( + 'after-command-done-status-before-query-update' ) - })().catch(mainReject) - return - } else { - console.log('no pending commands') - // this code can never be triggered from the post(/command/) route, since it always adds a command to the queue - // so you can only get here from commandWorkerInstance.on() route - // which means that the commandWorkerInstance has finished running the previous command - if (commandWorkerInstance) { - commandWorkerInstance.terminate().then(() => { - mainResolve(false) - }, mainReject) - return + status.set(command.id, done) + if (result.data) { + jsontagBuffers.push(result.data) + Object.assign(meta, result.meta) + const task = { + name: 'update', + req: { body: result.data, meta } + } + queryWorkerPool.update(task) + slowQueryWorkerPool.update(task) + } } - mainResolve(false) } - }) + catch (error) { + failStorage(error) + } + finally { + commandRunnerActive = false + } + })() + return runner } async function checkCommand(req, res) { @@ -505,44 +628,50 @@ async function main(options) { code: 202, status: 'accepted' } - } catch(err) { + } + catch (err) { error = { code: 400, - message: "Bad request", + message: 'Bad request', details: err.message } - console.error('Error: ',err) - sendResponse({code: 400, body: JSON.stringify(error)}, res) + console.error('Error: ', err) + sendResponse({ code: 400, body: JSON.stringify(error) }, res) return false } - if (!command || !command.id) { + if (!command || !validCommandId(command.id)) { error = { code: 422, - message: "Command has no id", + message: 'Command has no id', details: command } - sendResponse({code: 422, body: JSON.stringify(error)}, res) + sendResponse({ code: 422, body: JSON.stringify(error) }, res) return false - } else if (status.has(command.id)) { - const currentStatus = Object.assign({command: command.id}, status.get(command.id)) - sendResponse({body: JSON.stringify(currentStatus)}, res) + } + else if (status.has(command.id)) { + const currentStatus = Object.assign( + { command: command.id }, + status.get(command.id) + ) + sendResponse({ body: JSON.stringify(currentStatus) }, res) return false - } else if (!command.name) { + } + else if (!command.name) { error = { code: 422, - message: "Command has no name", + message: 'Command has no name', details: command } - sendResponse({code:422, body: JSON.stringify(error)}, res) - return false + sendResponse({ code: 422, body: JSON.stringify(error) }, res) + return false } - await appendFile(commandLog, JSONTag.stringify(command)) //FIXME: this loses request data + const line = JSONTag.stringify(command) + await appendFile(commandLog, line) await faultPoint('after-command-log-before-accepted-status') await appendFile(commandStatus, JSONTag.stringify(commandOK)) - status.set(command.id, commandOK) + status.set(command.id, commandOK) await faultPoint('after-command-accepted-status-before-response') - sendResponse({code: 202, body: JSON.stringify(commandOK)}, res) - return command.id + return { id: command.id, line } } } @@ -551,21 +680,23 @@ function sendResponse(response, res) { res.status(response.code) } if (response.jsontag) { - res.setHeader('content-type','application/jsontag') - } else { - res.setHeader('content-type','application/json') + res.setHeader('content-type', 'application/jsontag') + } + else { + res.setHeader('content-type', 'application/json') } - res.send(response.body)+"\n" + res.send(response.body) + '\n' } function sendError(error, res) { console.error(error) if (error.code && httpStatusCodes[error.code]) { res.status(error.code) - } else { + } + else { res.status(500) } - res.setHeader('content-type','application/json') + res.setHeader('content-type', 'application/json') res.send(JSON.stringify(error)) } @@ -576,7 +707,7 @@ function accept(req, res, mimetypes, handler) { let accept = req.accepts(mimetypes) if (!accept) { res.status(406) - res.send("

406 Unacceptable

\n") + res.send('

406 Unacceptable

\n') return false } if (typeof handler === 'function') { @@ -585,23 +716,23 @@ function accept(req, res, mimetypes, handler) { return true } -function handleWebRequest(req,res,options) -{ - let path = req.path; +function handleWebRequest(req, res, options) { + let path = req.path path = path.replace(/[^a-z0-9_.\-/]*/gi, '') // whitelist acceptable file paths path = path.replace(/\.+/g, '.') // blacklist '..' if (!path) { path = '/' } - if (path.substring(path.length-1)==='/') { + if (path.substring(path.length - 1) === '/') { path += 'index.html' } const fileOptions = { root: options.root } - if (fs.existsSync(fileOptions.root+path)) { + if (fs.existsSync(fileOptions.root + path)) { res.sendFile(path, fileOptions) - } else { + } + else { res.sendFile('/index.html', fileOptions) } } diff --git a/src/storage.mjs b/src/storage.mjs new file mode 100644 index 0000000..7a81070 --- /dev/null +++ b/src/storage.mjs @@ -0,0 +1,251 @@ +import fs from 'node:fs' +import fsp from 'node:fs/promises' +import path from 'node:path' +import { randomUUID } from 'node:crypto' +import { Buffer } from 'node:buffer' + +export function storageError(error) { + if (!(error instanceof Error)) { + error = new Error(String(error)) + } + error.storageFailure = true + return error +} + +// A rejected operation poisons its writer: later operations must not pass it. +export function serialWriter() { + let tail = Promise.resolve() + return operation => { + const result = tail.then(operation) + tail = result + void result.catch(() => { + + }) + return result + } +} + +async function withHandle(file, flags, operation, mode) { + let handle, failure, result + try { + handle = await fsp.open(file, flags, mode) + result = await operation(handle) + } + catch (error) { + failure = error + } + if (handle) { + try { + await handle.close() + } + catch (error) { + failure ??= error + } + } + if (failure) { + throw storageError(failure) + } + return result +} + +export async function writeAll(handle, data) { + const bytes = Buffer.isBuffer(data) ? data : Buffer.from(data) + let offset = 0 + while (offset < bytes.length) { + const { bytesWritten } = await handle.write( + bytes, + offset, + bytes.length - offset, + null + ) + if ( + !Number.isInteger(bytesWritten) || + bytesWritten <= 0 || + bytesWritten > bytes.length - offset + ) { + throw storageError(new Error('Invalid or zero-progress write')) + } + offset += bytesWritten + } +} + +export async function syncDirectory(directory) { + await withHandle(directory, 'r', handle => handle.sync()) +} + +export async function syncAncestors(directory) { + const device = (await fsp.stat(directory)).dev + for (let current = directory; ; current = path.dirname(current)) { + await syncDirectory(current) + const parent = path.dirname(current) + if (parent === current || (await fsp.stat(parent)).dev !== device) { + break + } + } +} + +export async function syncFile(file) { + await withHandle(file, 'r', handle => handle.sync()) + await syncDirectory(path.dirname(file)) +} + +export async function durableMkdir(directory) { + try { + await fsp.mkdir(directory) + } + catch (error) { + if (error.code === 'ENOENT') { + await durableMkdir(path.dirname(directory)) + await fsp.mkdir(directory) + } + else if (error.code !== 'EEXIST') { + throw storageError(error) + } + } + await syncDirectory(directory) + await syncDirectory(path.dirname(directory)) +} + +export async function publishFile(file, data, { mode } = {}) { + const temporary = `${file}.${randomUUID()}.tmp` + let renamed = false + try { + try { + const stat = await fsp.lstat(file) + if (stat.isSymbolicLink()) { + throw new Error('Refusing to publish over a symlink') + } + mode ??= stat.mode & 0o777 + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } + await withHandle( + temporary, + 'wx', + async handle => { + if (mode !== undefined) { + await handle.chmod(mode) + } + await writeAll(handle, data) + await handle.sync() + }, + mode + ) + await fsp.rename(temporary, file) + renamed = true + await syncDirectory(path.dirname(file)) + } + catch (error) { + if (!renamed) { + await fsp.unlink(temporary).catch(() => { + + }) + } + throw storageError(error) + } +} + +const appenders = new Map() +export function appendRecord(file, record) { + const key = path.resolve(file) + if (!appenders.has(key)) { + appenders.set(key, serialWriter()) + } + return appenders.get(key)(async () => { + await withHandle(file, 'a', async handle => { + await writeAll(handle, record + '\n') + await handle.datasync() + }) + await syncDirectory(path.dirname(file)) + }) +} + +// Existing synchronous index hooks keep their contract while gaining barriers. +export function publishFileSync(file, data) { + const temporary = `${file}.${randomUUID()}.tmp` + let fd, + failure, + renamed = false + try { + const bytes = Buffer.from(data) + let mode + try { + const stat = fs.lstatSync(file) + if (stat.isSymbolicLink()) { + throw new Error('Refusing to publish over a symlink') + } + mode = stat.mode & 0o777 + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } + fd = fs.openSync(temporary, 'wx', mode) + if (mode !== undefined) { + fs.fchmodSync(fd, mode) + } + let offset = 0 + while (offset < bytes.length) { + const count = fs.writeSync(fd, bytes, offset, bytes.length - offset) + if ( + !Number.isInteger(count) || + count <= 0 || + count > bytes.length - offset + ) { + throw new Error('Invalid or zero-progress write') + } + offset += count + } + fs.fsyncSync(fd) + } + catch (error) { + failure = error + } + if (fd !== undefined) { + try { + fs.closeSync(fd) + } + catch (error) { + failure ??= error + } + } + try { + if (failure) { + throw failure + } + fs.renameSync(temporary, file) + renamed = true + fd = fs.openSync(path.dirname(file), 'r') + failure = undefined + try { + fs.fsyncSync(fd) + } + catch (error) { + failure = error + } + try { + fs.closeSync(fd) + } + catch (error) { + failure ??= error + } + if (failure) { + throw failure + } + } + catch (error) { + if (!renamed) { + try { + fs.unlinkSync(temporary) + } + catch { + /* Preserve the primary failure. */ + } + } + throw storageError(error) + } +} diff --git a/src/store-inspection.mjs b/src/store-inspection.mjs new file mode 100644 index 0000000..5168bd8 --- /dev/null +++ b/src/store-inspection.mjs @@ -0,0 +1,404 @@ +import fs from 'node:fs/promises' +import path from 'node:path' +import JSONTag from '@muze-nl/jsontag' +import { from, _, anyOf, not } from '@muze-nl/jaqt' +import Parser from '@muze-nl/od-jsontag/src/parse.mjs' +import { createHash } from 'node:crypto' +import { assertOdJsonTagFraming, getChangesetPath } from './recovery.mjs' +import { + getDefaultIntegrityFile, + loadIntegrityManifest, + verifyIntegrity +} from './integrity.mjs' + +export const hash = bytes => createHash('sha256').update(bytes).digest('hex') +export const validCommandId = id => + typeof id === 'string' && id.length > 0 && !/[/\\\0]/.test(id) +export function storePaths(options = {}) { + const datafile = path.resolve(options.datafile || './data.od-jsontag') + let integrity + if (options.integrity === undefined) { + integrity = Boolean(options.integrityFile) + } + else { + integrity = Boolean(options.integrity) + } + const config = { + datafile, + commandLog: path.resolve(options.commandLog || './command-log.jsontag'), + commandStatus: path.resolve( + options.commandStatus || './command-status.jsontag' + ), + integrityFile: path.resolve( + options.integrityFile || getDefaultIntegrityFile(datafile) + ), + integrity, + requiredFiles: (options.requiredFiles || []).map(file => + path.resolve(file) + ) + } + if (options.schemaFile) { + config.schemaFile = path.resolve(options.schemaFile) + } + return config +} +export function mutableDirectories(config) { + return [ + ...new Set( + [ + config.datafile, + config.commandLog, + config.commandStatus, + config.integrityFile, + ...config.requiredFiles + ].map(file => path.dirname(file)) + ) + ].sort() +} +export async function inventory(config) { + const files = {} + const artifacts = [ + config.datafile, + config.commandLog, + config.commandStatus, + config.integrityFile, + ...config.requiredFiles + ] + if (config.schemaFile) { + artifacts.push(config.schemaFile) + } + const expected = new Set(artifacts) + const extension = path.extname(config.datafile), + stem = path.basename(config.datafile, extension) + '.' + for (const directory of mutableDirectories(config)) { + const entries = await fs.readdir(directory, { withFileTypes: true }) + for (const entry of entries.sort((a, b) => + a.name.localeCompare(b.name) + )) { + if (entry.name === '.simplystore-lock') { + continue + } + const file = path.join(directory, entry.name) + const dataArtifact = + directory === path.dirname(config.datafile) && + entry.name.startsWith(stem) && + (entry.name.endsWith(extension) || entry.name.endsWith('.tmp')) + const indexArtifact = + directory === path.dirname(config.datafile) && + entry.name.startsWith('index.') + const temporary = [...expected].some( + name => file.startsWith(name + '.') && file.endsWith('.tmp') + ) + if ( + !expected.has(file) && + !dataArtifact && + !indexArtifact && + !temporary + ) { + continue + } + if (entry.isSymbolicLink()) { + throw new Error(`Symlink in store artifact: ${file}`) + } + if (entry.isFile()) { + files[file] = hash(await fs.readFile(file)) + } + } + } + const required = [config.datafile, config.commandLog, config.commandStatus] + if (config.integrity) { + required.push(config.integrityFile) + } + required.push(...config.requiredFiles) + if (config.schemaFile) { + required.push(config.schemaFile) + } + for (const file of required) { + if (!(file in files)) { + try { + files[file] = hash(await fs.readFile(file)) + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + files[file] = null + } + } + } + return Object.fromEntries( + Object.entries(files).sort(([a], [b]) => a.localeCompare(b)) + ) +} + +async function records(file, errors, kind) { + let text + try { + text = await fs.readFile(file, 'utf8') + } + catch (error) { + errors.push(`${kind}: ${error.message}`) + return [] + } + if (text && !text.endsWith('\n')) { + errors.push(`${kind}: incomplete final record in ${file}`) + } + const result = [] + for (const [index, line] of text.split('\n').entries()) { + if (!line) { + continue + } + try { + const value = JSONTag.parse(line) + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new Error('record is not an object') + } + result.push({ value, line, lineNumber: index + 1 }) + } + catch (error) { + errors.push(`${kind} ${file}:${index + 1}: ${error.message}`) + } + } + return result +} + +// Reads canonical files only; deliberately never imports user handlers/index +// modules. +export async function inspectStore(options = {}) { + const config = storePaths(options) + const errors = [], + warnings = [] + const configuredPaths = [ + config.datafile, + config.commandLog, + config.commandStatus, + config.integrityFile + ] + const canonicalPaths = await Promise.all( + configuredPaths.map(async file => { + const directory = await fs.realpath(path.dirname(file)) + return path.join(directory, path.basename(file)) + }) + ) + if (new Set(canonicalPaths).size !== canonicalPaths.length) { + errors.push('Configured canonical artifact paths overlap') + } + const before = await inventory(config) + const log = await records(config.commandLog, errors, 'command log') + const statuses = await records( + config.commandStatus, + errors, + 'command status' + ) + const commands = [], + byId = new Map() + for (const record of log) { + const { id, name } = record.value + if (!validCommandId(id) || typeof name !== 'string' || !name) { + errors.push(`Invalid command at log line ${record.lineNumber}`) + continue + } + if (byId.has(id)) { + if ( + JSONTag.stringify(byId.get(id).command) !== + JSONTag.stringify(record.value) + ) { + errors.push(`Conflicting command ID ${id}`) + } + continue + } + const command = { + id, + position: commands.length, + command: record.value, + line: record.line, + history: [], + accepted: false, + status: null, + file: getChangesetPath(config.datafile, id) + } + byId.set(id, command) + commands.push(command) + } + const doneOrder = new Map() + for (const { value, lineNumber } of statuses) { + const command = byId.get(value.command) + if (!command) { + errors.push( + `Status line ${lineNumber} has no logged command: ${value.command}` + ) + continue + } + if ( + !['accepted', 'active', 'done', 'failed', 'unsafe'].includes( + value.status + ) + ) { + errors.push(`Unknown status for ${command.id}: ${value.status}`) + continue + } + if (value.status === 'accepted') { + command.accepted = true + } + if (value.status !== 'accepted' && !command.accepted) { + errors.push(`No acceptance evidence for ${command.id}`) + } + if (value.status === 'done') { + doneOrder.delete(command.id) + doneOrder.set(command.id, command.position) + } + command.history.push(value) + command.status = value.status + } + let lastDonePosition = -1 + for (const [id, position] of doneOrder) { + if (byId.get(id).status !== 'done') { + continue + } + if (position < lastDonePosition) { + errors.push( + `Committed execution differs from command-log order at ${id}` + ) + } + lastDonePosition = position + } + let manifest + try { + // Preserve optional existence-based compatibility; required integrity + // is explicit. + const enabled = config.integrity || before[config.integrityFile] != null + if (enabled) { + if (before[config.integrityFile] == null) { + throw new Error( + `Missing integrity manifest ${config.integrityFile}` + ) + } + await records(config.integrityFile, errors, 'integrity manifest') + manifest = await loadIntegrityManifest(config.integrityFile) + } + } + catch (error) { + errors.push(error.message) + } + const parser = new Parser() + const buffers = [], + committed = [] + let data, + prefixValid = errors.length === 0 + async function validateData(file) { + const bytes = await fs.readFile(file) + assertOdJsonTagFraming(bytes, file) + if (manifest) { + verifyIntegrity(manifest, config.integrityFile, file, bytes, { + required: true + }) + } + return bytes + } + try { + const bytes = await validateData(config.datafile) + if (!bytes.length) { + throw new Error('Empty base dataset') + } + data = parser.parse(bytes) + buffers.push(bytes) + } + catch (error) { + errors.push(error.message) + prefixValid = false + } + for (const command of commands) { + command.present = before[command.file] != null + command.condition = command.present ? 'present' : 'missing' + if (!command.accepted) { + errors.push(`Orphan command without acceptance: ${command.id}`) + } + if (command.present) { + try { + await validateData(command.file) + } + catch (error) { + command.condition = 'corrupt' + command.problem = error.message + } + } + if (command.status === 'done') { + if (!command.present || command.condition === 'corrupt') { + command.problem ||= `Missing changeset for committed command ${command.id}: ${command.file}` + prefixValid = false + } + else if (prefixValid) { + try { + const bytes = await validateData(command.file) + data = parser.parse(bytes) + buffers.push(bytes) + committed.push(command.id) + } + catch (error) { + command.problem = error.message + prefixValid = false + } + } + } + else if ( + command.status === 'accepted' || + command.status === 'active' + ) { + prefixValid = false + } + const laterDatasets = from(commands) + .where({ + position: position => position > command.position, + accepted: true, + file: file => before[file] != null + }) + .select(_.id) + command.laterDatasets = [...laterDatasets] + } + const extension = path.extname(config.datafile), + stem = path.basename(config.datafile, extension) + '.' + const known = new Set(commands.map(command => command.file)) + for (const file of Object.keys(before)) { + if ( + path.dirname(file) === path.dirname(config.datafile) && + path.basename(file).startsWith(stem) && + file.endsWith(extension) && + file !== config.datafile && + file !== config.integrityFile && + !known.has(file) + ) { + errors.push(`Unexplained changeset: ${file}`) + } + } + for (const file of config.requiredFiles) { + if (before[file] == null) { + errors.push(`Missing required artifact: ${file}`) + } + } + const after = await inventory(config) + if (JSON.stringify(before) !== JSON.stringify(after)) { + errors.push('Store changed during inspection') + } + const commandQuery = from(commands) + const pending = commandQuery.where({ status: anyOf('accepted', 'active') }) + const damaged = commandQuery.where( + anyOf({ problem: Boolean }, { present: Boolean, status: not('done') }) + ) + const ready = + errors.length === 0 && pending.length === 0 && damaged.length === 0 + warnings.push( + 'History completeness and original code/hidden inputs require independent administrator evidence; current files alone cannot prove them.' + ) + return { + config, + files: before, + fingerprint: hash(JSON.stringify(before)), + errors, + warnings, + commands, + committed, + ready, + data, + buffers + } +} diff --git a/src/store-ownership.mjs b/src/store-ownership.mjs new file mode 100644 index 0000000..0ee6c5e --- /dev/null +++ b/src/store-ownership.mjs @@ -0,0 +1,112 @@ +import fs from 'node:fs/promises' +import path from 'node:path' +import os from 'node:os' +import process from 'node:process' +import { randomUUID } from 'node:crypto' +import { publishFile, syncDirectory, syncAncestors } from './storage.mjs' + +export async function acquireOwnership( + directories, + { ancestorToken, purpose, auditDir } = {} +) { + const canonical = [ + ...new Set(await Promise.all(directories.map(dir => fs.realpath(dir)))) + ].sort() + for (const directory of canonical) { + for ( + let parent = path.dirname(directory); + ; + parent = path.dirname(parent) + ) { + if (!canonical.includes(parent)) { + const lock = path.join(parent, '.simplystore-lock') + try { + await fs.access(lock) + let owner + try { + owner = JSON.parse( + await fs.readFile( + path.join(lock, 'owner.json'), + 'utf8' + ) + ) + } + catch { + /* An incomplete lock remains exclusive. */ + } + if (!ancestorToken || owner?.token !== ancestorToken) { + throw new Error( + `Store is locked by unfinished parent operation: ${lock}` + ) + } + } + catch (error) { + if (error.code !== 'ENOENT') { + throw error + } + } + } + if (path.dirname(parent) === parent) { + break + } + } + } + const token = randomUUID(), + held = [] + try { + for (const directory of canonical) { + const lock = path.join(directory, '.simplystore-lock') + try { + await fs.mkdir(lock) + } + catch (error) { + if (error.code === 'EEXIST') { + throw new Error( + `Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.` + ) + } + throw error + } + held.push(lock) + await publishFile( + path.join(lock, 'owner.json'), + JSON.stringify({ + token, + purpose, + auditDir, + pid: process.pid, + host: os.hostname(), + started: new Date().toISOString() + }) + ) + await syncAncestors(directory) + } + } + catch (error) { + // No store writes have begun, so only our own acquired locks can be + // released. + for (const lock of held.reverse()) { + await fs.rm(lock, { recursive: true }) + await syncDirectory(path.dirname(lock)) + } + throw error + } + return { + token, + directories: canonical, + async release() { + for (const lock of [...held].reverse()) { + const owner = JSON.parse( + await fs.readFile(path.join(lock, 'owner.json'), 'utf8') + ) + if (owner.token !== token) { + throw new Error(`Ownership changed: ${lock}`) + } + await fs.unlink(path.join(lock, 'owner.json')) + await fs.rmdir(lock) + await syncDirectory(path.dirname(lock)) + } + held.length = 0 + } + } +} diff --git a/src/util.mjs b/src/util.mjs index 311266c..1f3965d 100644 --- a/src/util.mjs +++ b/src/util.mjs @@ -1,29 +1,31 @@ import JSONTag from '@muze-nl/jsontag' -import fs from 'node:fs/promises' +import { appendRecord } from './storage.mjs' export function deepFreeze(obj) { - Object.freeze(obj) - Object.keys(obj).forEach(prop => { - if (typeof obj[prop] === 'object' && !Object.isFrozen(obj[prop])) { - deepFreeze(obj[prop]) - } - }) - return obj + Object.freeze(obj) + Object.keys(obj).forEach(prop => { + if (typeof obj[prop] === 'object' && !Object.isFrozen(obj[prop])) { + deepFreeze(obj[prop]) + } + }) + return obj } -export function isString(s) -{ +export function isString(s) { return typeof s === 'string' || s instanceof String } export function joinArgs(args) { - return args = args.map(arg => { - if (isString(arg)) { - return arg - } else { - return JSONTag.stringify(arg) - } - }).join(' ') + return (args = args + .map(arg => { + if (isString(arg)) { + return arg + } + else { + return JSONTag.stringify(arg) + } + }) + .join(' ')) } /** @@ -32,15 +34,4 @@ export function joinArgs(args) { * @param {string} data The line to write * @return {void} */ -export async function appendFile(filename, data) { - let handle; - try { - handle = await fs.open(filename, 'a+') - await handle.appendFile(data+"\n") - await handle.datasync() - return true - } finally { - await handle.close() - } -} - +export const appendFile = appendRecord diff --git a/test/admin-recovery.mjs b/test/admin-recovery.mjs new file mode 100644 index 0000000..fa9ec53 --- /dev/null +++ b/test/admin-recovery.mjs @@ -0,0 +1,710 @@ +import { Buffer } from 'node:buffer' +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import path from 'node:path' +import os from 'node:os' +import JSONTag from '@muze-nl/jsontag' +import { makeServerFixture } from './durability-helpers.mjs' +import { + planRecovery, + applyRecovery, + backupStore, + restoreBackup, + verifyCandidate +} from '../src/admin-recovery.mjs' +import { inspectStore } from '../src/store-inspection.mjs' +import { acquireOwnership } from '../src/store-ownership.mjs' + +async function setup(t, entries) { + const store = await makeServerFixture(t) + const outputs = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-admin-output-') + ) + t.after(() => fs.rm(outputs, { recursive: true, force: true })) + await fs.writeFile( + store.commandLog, + entries + .map(([id]) => + JSONTag.stringify({ + id, + name: 'addPerson', + value: { name: id } + }) + ) + .join('\n') + '\n' + ) + const statuses = [] + for (const [id, status] of entries) { + statuses.push({ command: id, status: 'accepted', code: 202 }) + if (status !== 'accepted') { + statuses.push({ command: id, status, code: 200 }) + } + } + const statusLines = statuses.map(status => JSONTag.stringify(status)) + await fs.writeFile(store.commandStatus, statusLines.join('\n') + '\n') + return { store, outputs } +} +async function runRecovery(store, outputs, plan) { + return applyRecovery(plan, { + to: path.join(outputs, 'recovered'), + auditDir: path.join(outputs, 'audit'), + approveRerun: plan.rerun, + operator: 'test admin', + reason: 'Verified complete log and command inputs; side effects assessed' + }) +} +test('administrator recovers trailing missing/waiting commands in log order on a copy', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'done'], + ['B', 'accepted'] + ]) + const before = await fs.readFile(store.commandStatus) + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(plan.rerun, ['A', 'B']) + assert.deepEqual( + structuredClone(plan), + plan, + 'Recovery plans remain plain cloneable data' + ) + const result = await runRecovery(store, outputs, plan) + const actual = await inspectStore(result.config) + assert.equal(actual.ready, true) + assert.deepEqual( + structuredClone(actual.commands), + actual.commands, + 'Inspection does not expose JAQT proxies' + ) + assert.deepEqual( + actual.data.persons.map(p => p.name), + ['A', 'B'] + ) + assert.deepEqual(await fs.readFile(store.commandStatus), before) + assert.ok( + (await fs.readFile(result.config.commandStatus)) + .subarray(0, before.length) + .equals(before) + ) + assert.deepEqual( + (await fs.readFile(result.config.commandLog)).toString(), + (await fs.readFile(store.commandLog)).toString() + ) +}) +for (const state of ['accepted', 'done', 'unsafe']) { + test(`later ${state} dataset blocks rerun, including zero-byte no-op`, async t => { + const { store } = await setup(t, [ + ['A', 'done'], + ['B', state] + ]) + await fs.writeFile(path.join(store.dir, 'data.B.jsontag'), '') + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, false) + assert.ok(plan.blocks.some(s => s.includes('later accepted datasets'))) + }) +} +test('new later file makes an approved plan stale before executing a handler', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'done'], + ['B', 'accepted'] + ]) + const plan = await planRecovery(store, { quiescent: true }) + await fs.writeFile(path.join(store.dir, 'data.B.jsontag'), '') + await assert.rejects( + runRecovery(store, outputs, plan), + /Stale recovery plan/ + ) + await assert.rejects(fs.access(path.join(outputs, 'recovered')), /ENOENT/) +}) +test('changed code and missing explicit approval reject apply', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const plan = await planRecovery(store, { quiescent: true }) + await assert.rejects( + applyRecovery(plan, { + to: path.join(outputs, 'x'), + auditDir: path.join(outputs, 'y'), + operator: 'admin', + reason: 'test' + }), + /approval/ + ) + await fs.appendFile(store.commandsFile, '\n// changed\n') + await assert.rejects( + runRecovery(store, outputs, plan), + /Stale recovery plan/ + ) +}) +test('cooperative ownership excludes competing writer and path aliases', async t => { + const { store, outputs } = await setup(t, []) + const owner = await acquireOwnership([store.dir]) + try { + await fs.symlink(store.dir, path.join(outputs, 'alias')) + await assert.rejects( + acquireOwnership([path.join(outputs, 'alias')]), + /locked/ + ) + } + finally { + await owner.release() + } +}) +test('backup and restore validate saved bytes without executing commands', async t => { + const { store, outputs } = await setup(t, [['A', 'done']]) + const result = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + const backup = path.join(outputs, 'backup') + await backupStore(result.config, { to: backup, quiescent: true }) + const restored = await restoreBackup(backup, { + to: path.join(outputs, 'restored'), + auditDir: path.join(outputs, 'restore-audit'), + source: store, + sourceQuiescent: true + }) + assert.deepEqual( + (await inspectStore(restored.config)).data.persons.map(p => p.name), + ['A'] + ) + assert.deepEqual(restored.missingFromBackup, []) + const saved = JSON.parse( + await fs.readFile(path.join(backup, 'complete.json'), 'utf8') + ) + await fs.appendFile(saved.config.commandLog, ' ') + await assert.rejects( + restoreBackup(backup, { + to: path.join(outputs, 'bad'), + auditDir: path.join(outputs, 'bad-audit') + }), + /Backup contents/ + ) +}) + +test('read-only inspection never imports handlers and rejects conflicting done order', async t => { + const { store } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + await fs.writeFile( + store.commandsFile, + "throw new Error('must not be imported')" + ) + const before = await fs.readFile(store.commandStatus) + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(await fs.readFile(store.commandStatus), before) + await fs.appendFile( + store.commandStatus, + [ + { command: 'B', status: 'done' }, + { command: 'A', status: 'done' } + ] + .map(JSON.stringify) + .join('\n') + '\n' + ) + assert.ok( + (await planRecovery(store, { quiescent: true })).blocks.some(s => + s.includes('differs from command-log order') + ) + ) +}) +test('historical status order never makes a later log dataset eligible for bypass', async t => { + const { store } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'], + ['C', 'accepted'] + ]) + await fs.writeFile( + store.commandStatus, + [ + { command: 'C', status: 'accepted' }, + { command: 'B', status: 'accepted' }, + { command: 'A', status: 'accepted' } + ] + .map(JSON.stringify) + .join('\n') + '\n' + ) + await fs.writeFile(path.join(store.dir, 'data.C.jsontag'), '') + const plan = await planRecovery(store, { quiescent: true }) + assert.ok( + plan.blocks.some(s => s.includes('B: later accepted datasets exist: C')) + ) +}) +test('interrupted recovery preserves completed attempts and blocks an unfinished candidate', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + const witness = path.join(outputs, 'effects') + await fs.writeFile( + store.commandsFile, + `import fs from 'node:fs';export default {addPerson(data,c){fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n');if(c.id==='B')throw new Error('external effect already happened');data.persons.push(c.value)}}` + ) + const plan = await planRecovery(store, { quiescent: true }) + await assert.rejects( + runRecovery(store, outputs, plan), + /uncertain or failed/ + ) + const authorization = JSON.parse( + ( + await fs.readFile( + path.join(outputs, 'audit', 'attempts.jsonl'), + 'utf8' + ) + ).split('\n')[0] + ) + const partial = await inspectStore(authorization.config) + assert.equal(partial.ready, false) + assert.deepEqual(partial.committed, ['A']) + assert.equal(partial.commands.find(c => c.id === 'B').status, 'active') + assert.equal(await fs.readFile(witness, 'utf8'), 'A\nB\n') + await assert.rejects( + acquireOwnership([path.dirname(authorization.config.datafile)]), + /locked/ + ) + await assert.rejects( + fs.access(path.join(outputs, 'audit', 'complete.json')), + /ENOENT/ + ) + const preview = await planRecovery( + { + ...authorization.config, + commandsFile: store.commandsFile, + indexFile: store.indexFile + }, + { quiescent: true } + ) + assert.deepEqual( + preview.rerun, + ['B'], + 'completed A is reconstructed, never selected for re-execution' + ) +}) +test('retained digest mismatch stops recovery before replacing integrity or appending done', async t => { + const { store, outputs } = await setup(t, [['A', 'done']]) + const { appendIntegrityRecord } = await import('../src/integrity.mjs') + const integrity = path.join(store.dir, 'data.integrity.jsontag') + await appendIntegrityRecord( + integrity, + store.datafile, + await fs.readFile(store.datafile) + ) + await appendIntegrityRecord( + integrity, + path.join(store.dir, 'data.A.jsontag'), + Buffer.from('different historical bytes') + ) + store.integrityFile = integrity + const plan = await planRecovery(store, { quiescent: true }) + await assert.rejects( + runRecovery(store, outputs, plan), + /Integrity mismatch/ + ) + const authorization = JSON.parse( + ( + await fs.readFile( + path.join(outputs, 'audit', 'attempts.jsonl'), + 'utf8' + ) + ).split('\n')[0] + ) + assert.deepEqual( + await fs.readFile(authorization.config.integrityFile), + await fs.readFile(integrity) + ) + const statuses = ( + await fs.readFile(authorization.config.commandStatus, 'utf8') + ) + .trim() + .split('\n') + .map(JSONTag.parse) + assert.equal(statuses.at(-1).status, 'active') +}) +test('malformed tails and overlapping paths cannot be accepted as valid stores', async t => { + const { store } = await setup(t, [['A', 'accepted']]) + await fs.appendFile(store.commandLog, '{"id":"unfinished"') + assert.equal( + (await planRecovery(store, { quiescent: true })).actionable, + false + ) + assert.ok( + ( + await inspectStore({ ...store, commandStatus: store.commandLog }) + ).errors.some(e => e.includes('paths overlap')) + ) +}) + +test('approved recovery resumes from completed candidate prefix without repeating external effects', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + const witness = path.join(outputs, 'idempotent-effects') + await fs.writeFile( + store.commandsFile, + `import fs from 'node:fs';export default {addPerson(data,c){ + const previous=fs.existsSync(${JSON.stringify(witness)})?fs.readFileSync(${JSON.stringify(witness)},'utf8'):''; + if(!previous.split('\\n').includes(c.id)){ + fs.appendFileSync(${JSON.stringify(witness)},c.id+'\\n'); + if(c.id==='B')throw new Error('connection lost after effect'); + } + data.persons.push(c.value) + }}` + ) + await assert.rejects( + runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ), + /uncertain or failed/ + ) + const authorization = JSON.parse( + ( + await fs.readFile( + path.join(outputs, 'audit', 'attempts.jsonl'), + 'utf8' + ) + ).split('\n')[0] + ) + const partial = { + ...authorization.config, + commandsFile: store.commandsFile, + indexFile: store.indexFile + } + const plan = await planRecovery(partial, { quiescent: true }) + assert.deepEqual(plan.rerun, ['B']) + const { releaseOfflineLocks } = await import('../src/admin-recovery.mjs') + const release = await releaseOfflineLocks(partial, { + operator: 'test admin', + reason: 'Inspected retained recovery audit; idempotent B can retry', + confirmedStopped: true + }) + await fs.writeFile( + path.join(outputs, 'unlock-evidence.json'), + JSON.stringify(release) + ) + await release.finish() + const retry = path.join(outputs, 'retry') + await fs.mkdir(retry) + const result = await runRecovery(partial, retry, plan) + assert.deepEqual( + (await inspectStore(result.config)).data.persons.map(p => p.name), + ['A', 'B'] + ) + assert.equal(await fs.readFile(witness, 'utf8'), 'A\nB\n') +}) +test('finishing an interrupted completion report reconstructs data without executing again', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const result = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + const audit = path.join(outputs, 'audit') + // Fixture models the boundary after done, before completion report/lock + // release. + await fs.unlink(path.join(audit, 'complete.json')) + await acquireOwnership([path.dirname(result.config.datafile)], { + purpose: 'recovery', + auditDir: audit + }) + const { finishRecovery, releaseOfflineLocks } = await import( + '../src/admin-recovery.mjs' + ) + await assert.rejects( + releaseOfflineLocks(result.config, { + operator: 'admin', + reason: 'test', + confirmedStopped: true + }), + /requires finish/ + ) + const before = await fs.readFile(result.config.commandStatus) + const finished = await finishRecovery(audit, { + operator: 'admin', + reason: 'All authorized commands already committed', + confirmedStopped: true + }) + assert.deepEqual(await fs.readFile(result.config.commandStatus), before) + assert.deepEqual(finished.committed, ['A']) + const owner = await acquireOwnership([path.dirname(result.config.datafile)]) + await owner.release() +}) +test('multi-directory integrity paths and file permissions survive backup and restore', async t => { + const { store, outputs } = await setup(t, []) + const logdir = path.join(store.dir, 'logs') + await fs.mkdir(logdir) + store.commandLog = path.join(logdir, 'commands.jsontag') + store.commandStatus = path.join(logdir, 'status.jsontag') + await fs.writeFile(store.commandLog, '') + await fs.writeFile(store.commandStatus, '') + store.integrityFile = path.join(logdir, 'integrity.jsontag') + const { appendIntegrityRecord } = await import('../src/integrity.mjs') + await appendIntegrityRecord( + store.integrityFile, + store.datafile, + await fs.readFile(store.datafile) + ) + await fs.chmod(store.datafile, 0o600) + const backup = path.join(outputs, 'backup') + await backupStore(store, { to: backup, quiescent: true }) + const restored = await restoreBackup(backup, { + to: path.join(outputs, 'restored'), + auditDir: path.join(outputs, 'audit') + }) + assert.equal((await inspectStore(restored.config)).ready, true) + assert.equal((await fs.stat(restored.config.datafile)).mode & 0o777, 0o600) + assert.deepEqual( + await fs.readFile(restored.config.integrityFile), + await fs.readFile(store.integrityFile) + ) +}) + +test('backup excludes unrelated secrets while retaining declared custom artifacts', async t => { + const { store, outputs } = await setup(t, []) + await fs.writeFile( + path.join(store.dir, '.env'), + 'private application secret' + ) + const required = path.join(store.dir, 'derived-custom.txt') + await fs.writeFile(required, 'required output') + const report = await backupStore( + { ...store, requiredFiles: [required] }, + { to: path.join(outputs, 'backup'), quiescent: true } + ) + assert.equal( + Object.keys(report.files).some(file => path.basename(file) === '.env'), + false + ) + assert.equal( + await fs.readFile(report.config.requiredFiles[0], 'utf8'), + 'required output' + ) +}) + +test('a backup remains restorable after transfer to a different directory', async t => { + const { store, outputs } = await setup(t, []) + const old = path.join(outputs, 'old-backup') + await backupStore(store, { to: old, quiescent: true }) + const moved = path.join(outputs, 'transferred-backup') + await fs.rename(old, moved) + const result = await restoreBackup(moved, { + to: path.join(outputs, 'restored'), + auditDir: path.join(outputs, 'restore-audit') + }) + assert.equal((await inspectStore(result.config)).ready, true) +}) + +test('promotion verification rejects code changed after completed recovery', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const result = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + assert.equal((await verifyCandidate(result)).ready, true) + await fs.appendFile(store.commandsFile, '\n// altered after recovery\n') + await assert.rejects(verifyCandidate(result), /Selected code changed/) +}) + +test('verified A prefix permits only missing B, waiting C and missing D in log order', async t => { + const { store, outputs } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'], + ['C', 'accepted'], + ['D', 'accepted'] + ]) + const complete = await runRecovery( + store, + outputs, + await planRecovery(store, { quiescent: true }) + ) + const source = { + ...complete.config, + commandsFile: store.commandsFile, + indexFile: store.indexFile + } + for (const id of ['B', 'C', 'D']) { + await fs.unlink( + path.join(path.dirname(source.datafile), `data.${id}.jsontag`) + ) + } + await fs.appendFile( + source.commandStatus, + JSONTag.stringify({ command: 'C', status: 'accepted', code: 202 }) + + '\n' + ) + const prefix = await fs.readFile( + path.join(path.dirname(source.datafile), 'data.A.jsontag') + ) + const plan = await planRecovery(source, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(plan.committed, ['A']) + assert.deepEqual(plan.rerun, ['B', 'C', 'D']) + const next = path.join(outputs, 'next') + await fs.mkdir(next) + const result = await runRecovery(source, next, plan) + assert.deepEqual( + (await inspectStore(result.config)).data.persons.map(p => p.name), + ['A', 'B', 'C', 'D'] + ) + assert.deepEqual( + await fs.readFile( + path.join(path.dirname(result.config.datafile), 'data.A.jsontag') + ), + prefix + ) +}) + +test('restore reports accepted commands absent from an older backup', async t => { + const { store, outputs } = await setup(t, []) + const backup = path.join(outputs, 'old-backup') + await backupStore(store, { to: backup, quiescent: true }) + await fs.appendFile( + store.commandLog, + JSONTag.stringify({ + id: 'later', + name: 'addPerson', + value: { name: 'later' } + }) + '\n' + ) + await fs.appendFile( + store.commandStatus, + JSONTag.stringify({ command: 'later', status: 'accepted', code: 202 }) + + '\n' + ) + const result = await restoreBackup(backup, { + to: path.join(outputs, 'rollback'), + auditDir: path.join(outputs, 'audit'), + source: store, + sourceQuiescent: true + }) + assert.equal(result.sameBase, true) + assert.deepEqual(result.missingFromBackup, ['later']) + assert.deepEqual(result.committed, []) +}) + +test('identical historical duplicate retains first log position; conflicting duplicate blocks', async t => { + const { store } = await setup(t, [ + ['A', 'accepted'], + ['B', 'accepted'] + ]) + await fs.appendFile( + store.commandLog, + JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) + '\n' + ) + const plan = await planRecovery(store, { quiescent: true }) + assert.equal(plan.actionable, true) + assert.deepEqual(plan.rerun, ['A', 'B']) + await fs.appendFile( + store.commandLog, + JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'different' } + }) + '\n' + ) + assert.ok( + (await planRecovery(store, { quiescent: true })).blocks.some(s => + s.includes('Conflicting command ID A') + ) + ) +}) + +test('backup coverage distinguishes changed history and data from missing source data', async t => { + const { store, outputs } = await setup(t, [['A', 'accepted']]) + const plan = await planRecovery(store, { quiescent: true }) + const recovered = await runRecovery(store, outputs, plan) + const source = recovered.config + const backup = path.join(outputs, 'coverage-backup') + await backupStore(source, { to: backup, quiescent: true }) + + const changeset = path.join(path.dirname(source.datafile), 'data.A.jsontag') + const original = new Map() + for (const file of [ + source.datafile, + source.commandLog, + source.commandStatus, + changeset + ]) { + original.set(file, await fs.readFile(file)) + } + const differentCommand = JSONTag.stringify({ + id: 'A', + name: 'addPerson', + value: { name: 'changed' } + }) + const failedStatus = JSONTag.stringify({ + command: 'A', + status: 'failed', + code: 500 + }) + const cases = [ + { name: 'same', missing: [], sameBase: true }, + { + name: 'command', + file: source.commandLog, + bytes: differentCommand + '\n', + missing: ['A'], + sameBase: true + }, + { + name: 'status', + file: source.commandStatus, + bytes: original.get(source.commandStatus) + failedStatus + '\n', + missing: ['A'], + sameBase: true + }, + { + name: 'data', + file: changeset, + bytes: original.get(changeset) + ' ', + missing: ['A'], + sameBase: true + }, + { + name: 'base', + file: source.datafile, + bytes: original.get(source.datafile) + ' ', + missing: ['A'], + sameBase: false + }, + { name: 'missing', file: changeset, missing: [], sameBase: true } + ] + for (const scenario of cases) { + for (const [file, bytes] of original) { + await fs.writeFile(file, bytes) + } + if (scenario.file) { + if (scenario.bytes === undefined) { + await fs.unlink(scenario.file) + } + else { + await fs.writeFile(scenario.file, scenario.bytes) + } + } + const result = await restoreBackup(backup, { + to: path.join(outputs, scenario.name), + auditDir: path.join(outputs, scenario.name + '-audit'), + source, + sourceQuiescent: true + }) + assert.deepEqual( + result.missingFromBackup, + scenario.missing, + scenario.name + ) + assert.equal(result.sameBase, scenario.sameBase, scenario.name) + assert.deepEqual( + structuredClone(result), + result, + 'Restore reports remain plain cloneable data' + ) + } +}) diff --git a/test/after-change-handlers.mjs b/test/after-change-handlers.mjs index eaab756..089cdac 100644 --- a/test/after-change-handlers.mjs +++ b/test/after-change-handlers.mjs @@ -1,3 +1,4 @@ +import { waitForExit } from './durability-helpers.mjs' import assert from 'node:assert/strict' import fs from 'node:fs/promises' import path from 'node:path' @@ -23,7 +24,9 @@ async function waitForCommandStatus(port, commandId, expectedStatus) { } await new Promise(resolve => setTimeout(resolve, 25)) } - assert.fail(`Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}`) + assert.fail( + `Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}` + ) } async function postAndWait(port, command, expectedStatus = 'done') { @@ -57,20 +60,29 @@ test('handler lifecycle: index update failure prevents command commit', async t ` }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { - id: 'index-fails', - name: 'addPerson', - value: {name: 'Should Not Commit'} - }, 'failed') + const status = await postAndWait( + port, + { + id: 'index-fails', + name: 'addPerson', + value: { name: 'Should Not Commit' } + }, + 'failed' + ) assert.equal(status.status, 'failed') assert.match(status.message, /index update failed/) assert.deepEqual(await queryNames(port), ['Initial']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) - await assert.rejects(fs.access(path.join(fixture.dir, 'data.index-fails.jsontag')), /ENOENT/) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) + await assert.rejects( + fs.access(path.join(fixture.dir, 'data.index-fails.jsontag')), + /ENOENT/ + ) }) test('handler lifecycle: derived index file can remain after failed command', async t => { @@ -90,21 +102,39 @@ export default { ` }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { - id: 'index-writes-then-fails', - name: 'addPerson', - value: {name: 'Should Not Commit'} - }, 'failed') + const status = await postAndWait( + port, + { + id: 'index-writes-then-fails', + name: 'addPerson', + value: { name: 'Should Not Commit' } + }, + 'failed' + ) assert.equal(status.status, 'failed') assert.match(status.message, /derived index failed after write/) assert.deepEqual(await queryNames(port), ['Initial']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) - await assert.doesNotReject(fs.access(path.join(fixture.dir, 'derived-before-failure.index-writes-then-fails.txt'))) - await assert.rejects(fs.access(path.join(fixture.dir, 'data.index-writes-then-fails.jsontag')), /ENOENT/) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) + await assert.doesNotReject( + fs.access( + path.join( + fixture.dir, + 'derived-before-failure.index-writes-then-fails.txt' + ) + ) + ) + await assert.rejects( + fs.access( + path.join(fixture.dir, 'data.index-writes-then-fails.jsontag') + ), + /ENOENT/ + ) }) test('handler lifecycle: final offset write failure prevents command commit', async t => { @@ -115,36 +145,47 @@ test('handler lifecycle: final offset write failure prevents command commit', as const commandId = 'offset-write-fails' await fs.mkdir(path.join(fixture.dir, `index.offset.${commandId}.json`)) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { + const response = await postCommand(port, { id: commandId, name: 'addPerson', - value: {name: 'Should Not Commit'} - }, 'failed') + value: { name: 'Should Not Commit' } + }) + assert.equal(response.status, 202) + assert.equal((await waitForExit(server.child)).code, 1) + assert.match(server.getOutput(), /Storage outcome uncertain/) - assert.match(status.message, /EISDIR|EEXIST|ENOTEMPTY/) - assert.deepEqual(await queryNames(port), ['Initial']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) + assert.match(server.getOutput(), /EISDIR|EEXIST|ENOTEMPTY/) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) }) test('handler lifecycle: custom finalizer rejection prevents command commit', async t => { const fixture = await makeServerFixture(t, { initialData: '{"persons":[{"name":"Initial"}]}', commandsSource: baseCommands, - indexSource: 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}' + indexSource: + 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}' }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) - const status = await postAndWait(port, { - id: 'custom-finalizer-fails', name: 'addPerson', value: {name: 'Should Not Commit'} - }, 'failed') - assert.match(status.message, /custom finalization failed/) - assert.deepEqual(await queryNames(port), ['Initial']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) + const response = await postCommand(port, { + id: 'custom-finalizer-fails', + name: 'addPerson', + value: { name: 'Should Not Commit' } + }) + assert.equal(response.status, 202) + assert.equal((await waitForExit(server.child)).code, 1) + assert.match(server.getOutput(), /Storage outcome uncertain/) + assert.match(server.getOutput(), /custom finalization failed/) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial' + ]) }) test('handler lifecycle: current index update can mutate canonical state before commit', async t => { @@ -161,15 +202,23 @@ test('handler lifecycle: current index update can mutate canonical state before ` }) const port = await getOpenPort() - const server = startServer(t, fixture, {port}) + const server = startServer(t, fixture, { port }) await waitForServer(server.child, server.getOutput, port) await postAndWait(port, { id: 'index-mutates', name: 'addPerson', - value: {name: 'Command Mutation'} + value: { name: 'Command Mutation' } }) - assert.deepEqual(await queryNames(port), ['Initial', 'Command Mutation', 'Indexer Mutation']) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial', 'Command Mutation', 'Indexer Mutation']) + assert.deepEqual(await queryNames(port), [ + 'Initial', + 'Command Mutation', + 'Indexer Mutation' + ]) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), [ + 'Initial', + 'Command Mutation', + 'Indexer Mutation' + ]) }) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs index f228aca..1b25ff9 100644 --- a/test/conversion-offsets.mjs +++ b/test/conversion-offsets.mjs @@ -10,53 +10,95 @@ import { promisify } from 'node:util' import Parser from '@muze-nl/od-jsontag' const run = promisify(execFile) -const converter = fileURLToPath(new URL('../scripts/convert.mjs', import.meta.url)) +const converter = fileURLToPath( + new URL('../scripts/convert.mjs', import.meta.url) +) const baseIndex = new URL('../src/index.mjs', import.meta.url).href async function fixture(t) { const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-convert-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) + t.after(() => fs.rm(dir, { recursive: true, force: true })) const input = path.join(dir, 'input.jsontag') const output = path.join(dir, 'data.jsontag') - await fs.writeFile(input, '{"items":[{"name":"first"},{"name":"a long original name"}]}') - return {dir, input, output} + await fs.writeFile( + input, + '{"items":[{"name":"first"},{"name":"a long original name"}]}' + ) + return { dir, input, output } } -async function convert({input, output}, indexFile) { - await run(process.execPath, [converter, input, output, ...indexFile ? [indexFile] : []], {timeout: 10000}) +async function convert({ input, output }, indexFile) { + const args = [converter, input, output] + if (indexFile) { + args.push(indexFile) + } + await run(process.execPath, args, { timeout: 10000 }) } -async function verifyOffsets({dir, output}) { +async function verifyOffsets({ dir, output }) { const bytes = await fs.readFile(output) - const offsets = JSON.parse(await fs.readFile(path.join(dir, 'index.offset.json'), 'utf8')) + const offsets = JSON.parse( + await fs.readFile(path.join(dir, 'index.offset.json'), 'utf8') + ) const payloads = [] let cursor = 0 - // Derive byte ranges directly from disk framing, independently of Parser.meta. + // Derive byte ranges directly from disk framing, independently of + // Parser.meta. while (cursor < bytes.length) { - const match = /^\((\d+)\)/.exec(bytes.subarray(cursor, cursor + 32).toString('ascii')) + const match = /^\((\d+)\)/.exec( + bytes.subarray(cursor, cursor + 32).toString('ascii') + ) assert.ok(match, `record ${payloads.length} has a length prefix`) const start = cursor + match[0].length const end = start + Number(match[1]) assert.ok(end <= bytes.length, 'record ends within the file') - assert.deepEqual(offsets[payloads.length], [start, end], `record ${payloads.length} byte range`) - payloads.push(bytes.subarray(...offsets[payloads.length]).toString('utf8')) + assert.deepEqual( + offsets[payloads.length], + [start, end], + `record ${payloads.length} byte range` + ) + payloads.push( + bytes.subarray(...offsets[payloads.length]).toString('utf8') + ) cursor = end if (cursor < bytes.length) { assert.equal(bytes[cursor], 10, 'records separated by a newline') cursor++ } } - assert.equal(Object.keys(offsets).length, payloads.length, 'no missing or extra offsets') - return {payloads, data: new Parser('').parse(bytes.toString('utf8'))} + assert.equal( + Object.keys(offsets).length, + payloads.length, + 'no missing or extra offsets' + ) + return { payloads, data: new Parser().parse(bytes.toString('utf8')) } } +test('conversion supports the example dataset and its URL values', async t => { + const files = await fixture(t) + await fs.copyFile( + new URL('../example/swdb.jsontag', import.meta.url), + files.input + ) + await convert(files) + const { data } = await verifyOffsets(files) + assert.equal(String(data.people[0].name), 'Luke Skywalker') + assert.equal( + String(data.people[0].homeworld.url), + 'http://swapi.co/api/planets/1/' + ) + assert.equal(String(data.planets[0].url), 'http://swapi.co/api/planets/2/') +}) + test('default conversion offsets match every final record', async t => { const files = await fixture(t) await convert(files) - const {payloads, data} = await verifyOffsets(files) + const { payloads, data } = await verifyOffsets(files) assert.equal(payloads.length, 3) assert.equal(data.items[1].name, 'a long original name') - const ids = JSON.parse(await fs.readFile(path.join(files.dir, 'index.id.json'), 'utf8')) + const ids = JSON.parse( + await fs.readFile(path.join(files.dir, 'index.id.json'), 'utf8') + ) assert.match(payloads[ids['/first']], /id="\/first"/) assert.match(payloads[ids['/second']], /id="\/second"/) }) @@ -64,7 +106,9 @@ test('default conversion offsets match every final record', async t => { test('conversion finalizes offsets after custom indexing changes record sizes and adds records', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} + await fs.writeFile( + indexFile, + `import index from ${JSON.stringify(baseIndex)} export default { create(data, meta) { index.create(data, meta) @@ -74,13 +118,20 @@ export default { data.items.push({name: 'new derived record'}) } } -`) +` + ) await convert(files, indexFile) - const {payloads, data} = await verifyOffsets(files) + const { payloads, data } = await verifyOffsets(files) assert.equal(payloads.length, 4) - assert.equal(data.items[0].name, 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a much longer derived name') + assert.equal( + data.items[0].name, + 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a much longer derived name' + ) assert.equal(data.items[0].derived[0].name, 'x') - assert.equal(Object.getOwnPropertyDescriptor(data.items[0], 'derived').enumerable, false) + assert.equal( + Object.getOwnPropertyDescriptor(data.items[0], 'derived').enumerable, + false + ) assert.equal(data.items[2].name, 'new derived record') assert.match(payloads[3], /new derived record/) }) @@ -88,10 +139,16 @@ export default { test('conversion replaces stale offsets even when the custom hook writes no index', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, 'export default Object.freeze({ create(data) { data.items[0].name = "changed" } })\n') - await fs.writeFile(path.join(files.dir, 'index.offset.json'), '{"0":[0,1],"99":[2,3]}') + await fs.writeFile( + indexFile, + 'export default Object.freeze({ create(data) { data.items[0].name = "changed" } })\n' + ) + await fs.writeFile( + path.join(files.dir, 'index.offset.json'), + '{"0":[0,1],"99":[2,3]}' + ) await convert(files, indexFile) - const {payloads, data} = await verifyOffsets(files) + const { payloads, data } = await verifyOffsets(files) assert.equal(payloads.length, 3) assert.equal(data.items[0].name, 'changed') }) @@ -99,7 +156,9 @@ test('conversion replaces stale offsets even when the custom hook writes no inde test('conversion awaits custom finalization with the original receiver and final bytes', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} + await fs.writeFile( + indexFile, + `import index from ${JSON.stringify(baseIndex)} import fs from 'node:fs/promises' import {setTimeout} from 'node:timers/promises' export default { @@ -112,16 +171,68 @@ export default { await fs.writeFile(meta.data + '/finalized.json', JSON.stringify({marker: this.marker, uuid})) } } -`) +` + ) await convert(files, indexFile) await verifyOffsets(files) - assert.deepEqual(JSON.parse(await fs.readFile(path.join(files.dir, 'finalized.json'), 'utf8')), {marker: 'custom receiver', uuid: null}) + assert.deepEqual( + JSON.parse( + await fs.readFile(path.join(files.dir, 'finalized.json'), 'utf8') + ), + { marker: 'custom receiver', uuid: null } + ) }) test('conversion propagates rejection from custom finalization', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, 'export default {create() {}, async finalize() { throw new Error("custom finalization failed") }}\n') - await assert.rejects(convert(files, indexFile), /custom finalization failed/) - await assert.rejects(fs.access(path.join(files.dir, 'index.offset.json')), /ENOENT/) + await fs.writeFile( + indexFile, + 'export default {create() {}, async finalize() { throw new Error("custom finalization failed") }}\n' + ) + await assert.rejects( + convert(files, indexFile), + /custom finalization failed/ + ) + await assert.rejects( + fs.access(path.join(files.dir, 'index.offset.json')), + /ENOENT/ + ) +}) + +test('conversion initializes existing-format logs and optional base integrity without overwriting a store', async t => { + const files = await fixture(t) + await run( + process.execPath, + [converter, files.input, files.output, '--integrity'], + { timeout: 10000 } + ) + const { loadIntegrityManifest, verifyIntegrity } = await import( + '../src/integrity.mjs' + ) + const integrity = path.join(files.dir, 'data.integrity.jsontag') + assert.equal( + verifyIntegrity( + await loadIntegrityManifest(integrity), + integrity, + files.output, + await fs.readFile(files.output), + { required: true } + ), + true + ) + assert.equal( + await fs.readFile(path.join(files.dir, 'command-log.jsontag'), 'utf8'), + '' + ) + assert.equal( + await fs.readFile( + path.join(files.dir, 'command-status.jsontag'), + 'utf8' + ), + '' + ) + const before = await fs.readFile(files.output) + await assert.rejects(convert(files), /existing store is not overwritten/) + assert.deepEqual(await fs.readFile(files.output), before) }) diff --git a/test/durability-crash.mjs b/test/durability-crash.mjs index 3d4be48..ef1152c 100644 --- a/test/durability-crash.mjs +++ b/test/durability-crash.mjs @@ -3,10 +3,18 @@ import fs from 'node:fs/promises' import path from 'node:path' import test from 'node:test' import { faultPoint } from '../src/faults.mjs' -import { appendIntegrityRecord, getDefaultIntegrityFile, loadIntegrityManifest } from '../src/integrity.mjs' +import { + appendIntegrityRecord, + getDefaultIntegrityFile, + loadIntegrityManifest +} from '../src/integrity.mjs' import { getChangesetPath } from '../src/recovery.mjs' -import { assertRuntimeEnvironmentConfiguration, getRuntimeEnvironment } from '../src/runtime-environment.mjs' import { + assertRuntimeEnvironmentConfiguration, + getRuntimeEnvironment +} from '../src/runtime-environment.mjs' +import { + unlockStoppedFixture, getCommandStatus, getOpenPort, makeServerFixture, @@ -25,8 +33,12 @@ async function postCommandExpectingCrash(port, command) { try { const response = await postCommand(port, command) assert.equal(response.status, 202) - } catch (error) { - assert.match(error.message, /fetch failed|terminated|socket|other side closed|aborted/i) + } + catch (error) { + assert.match( + error.message, + /fetch failed|terminated|socket|other side closed|aborted/i + ) } } @@ -38,8 +50,14 @@ async function postCommandStatus(port, command, expectedHttpStatus = 200) { async function assertServerStateMatchesOracle(port, fixture, expectedNames) { const persons = await queryPersons(port) - assert.deepEqual(persons.map(person => person.name), expectedNames) - assert.deepEqual(await reconstructCommittedPersonNames(fixture), expectedNames) + assert.deepEqual( + persons.map(person => person.name), + expectedNames + ) + assert.deepEqual( + await reconstructCommittedPersonNames(fixture), + expectedNames + ) } async function waitForCommandStatus(port, commandId, expectedStatus) { @@ -52,141 +70,81 @@ async function waitForCommandStatus(port, commandId, expectedStatus) { } await new Promise(resolve => setTimeout(resolve, 25)) } - assert.fail(`Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}`) + assert.fail( + `Timed out waiting for ${commandId} to become ${expectedStatus}; latest status: ${JSON.stringify(lastStatus)}` + ) } test('runtime environment defaults to production and ignores production fault points', () => { assert.equal(getRuntimeEnvironment({}), 'production') assert.doesNotThrow(() => assertRuntimeEnvironmentConfiguration({})) - assert.doesNotThrow(() => assertRuntimeEnvironmentConfiguration({ - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - })) - assert.doesNotThrow(() => assertRuntimeEnvironmentConfiguration({ - SIMPLYSTORE_ENV: 'test', - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - })) + assert.doesNotThrow(() => + assertRuntimeEnvironmentConfiguration({ + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }) + ) + assert.doesNotThrow(() => + assertRuntimeEnvironmentConfiguration({ + SIMPLYSTORE_ENV: 'test', + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }) + ) assert.throws( - () => assertRuntimeEnvironmentConfiguration({SIMPLYSTORE_ENV: 'prod'}), + () => + assertRuntimeEnvironmentConfiguration({ SIMPLYSTORE_ENV: 'prod' }), /Invalid SIMPLYSTORE_ENV/ ) }) test('fault points are inert outside test environment', async () => { - assert.equal(await faultPoint('before-command-done-status', { - SIMPLYSTORE_ENV: 'production', - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - }), false) - assert.equal(await faultPoint('before-command-done-status', { - SIMPLYSTORE_ENV: 'development', - SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' - }), false) -}) - -test('crash after command log but before accepted status is not committed', async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const command = { - id: 'crash-after-log', - name: 'addPerson', - value: {name: 'After Log'} - } - - const first = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: 'after-command-log-before-accepted-status' - }) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, {port}) - await waitForServer(second.child, second.getOutput, port) - - await assertServerStateMatchesOracle(port, fixture, []) - assert.deepEqual(await readCommandStatusRecords(fixture), []) - assert.equal((await readCommandLogRecords(fixture)).length, 1) -}) - -test('duplicate command log records do not replay an accepted command twice', async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const command = { - id: 'duplicate-log-command', - name: 'addPerson', - value: {name: 'Once'} - } - - const first = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: 'after-command-log-before-accepted-status' - }) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: 'after-command-accepted-status-before-response' - }) - await waitForServer(second.child, second.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(second.child)).signal, 'SIGKILL') - - const third = startServer(t, fixture, {port}) - await waitForServer(third.child, third.getOutput, port) - - await assertServerStateMatchesOracle(port, fixture, ['Once']) - assert.equal((await readCommandLogRecords(fixture)).length, 2) - assert.equal((await readCommandStatusRecords(fixture)).filter(record => record.status === 'done').length, 1) + assert.equal( + await faultPoint('before-command-done-status', { + SIMPLYSTORE_ENV: 'production', + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }), + false + ) + assert.equal( + await faultPoint('before-command-done-status', { + SIMPLYSTORE_ENV: 'development', + SIMPLYSTORE_FAULT_POINT: 'before-command-done-status' + }), + false + ) }) -test('accepted command crash boundaries replay to one committed state', async t => { - const replayFaultPoints = [ - 'after-command-accepted-status-before-response', - 'after-active-status-before-command-worker', - 'before-command-changeset-write', - 'after-command-changeset-write', - 'before-command-done-status' - ] - - for (const faultPointName of replayFaultPoints) { - await t.test(faultPointName, async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const command = { - id: faultPointName, - name: 'addPerson', - value: {name: faultPointName} - } - - const first = startServer(t, fixture, { - port, - runtimeEnvironment: 'test', - faultPoint: faultPointName - }) - await waitForServer(first.child, first.getOutput, port) - await postCommandExpectingCrash(port, command) - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, {port}) - await waitForServer(second.child, second.getOutput, port) - - const retryStatus = await postCommandStatus(port, { - id: command.id, - name: 'addPerson', - value: {name: `${faultPointName} duplicate`} - }) - assert.equal(retryStatus.command, command.id) - assert.equal(retryStatus.status, 'done') - - await assertServerStateMatchesOracle(port, fixture, [faultPointName]) - assert.equal((await getCommandStatus(port, command.id)).status, 'done') +for (const faultPointName of [ + 'after-command-log-before-accepted-status', + 'after-command-accepted-status-before-response', + 'after-active-status-before-command-worker', + 'before-command-changeset-write', + 'after-command-changeset-write', + 'before-command-done-status' +]) { + test(`crash at ${faultPointName} preserves evidence and never automatically reruns`, async t => { + const fixture = await makeServerFixture(t) + const port = await getOpenPort() + const first = startServer(t, fixture, { + port, + runtimeEnvironment: 'test', + faultPoint: faultPointName }) - } -}) + await waitForServer(first.child, first.getOutput, port) + await postCommandExpectingCrash(port, { + id: 'A', + name: 'addPerson', + value: { name: 'A' } + }) + assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') + const before = await fs.readFile(fixture.commandStatus) + await unlockStoppedFixture(t, fixture) + const second = startServer(t, fixture, { port }) + assert.equal((await waitForExit(second.child)).code, 1) + assert.match(second.getOutput(), /Administrative recovery required/) + assert.deepEqual(await fs.readFile(fixture.commandStatus), before) + assert.equal((await readCommandLogRecords(fixture))[0].id, 'A') + }) +} test('crash after done status but before query update recovers committed state without replay', async t => { const fixture = await makeServerFixture(t) @@ -194,7 +152,7 @@ test('crash after done status but before query update recovers committed state w const command = { id: 'done-before-query-update', name: 'addPerson', - value: {name: 'Committed'} + value: { name: 'Committed' } } const first = startServer(t, fixture, { @@ -206,74 +164,55 @@ test('crash after done status but before query update recovers committed state w await postCommandExpectingCrash(port, command) assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') + await unlockStoppedFixture(t, fixture) const statusAfterCrash = await readCommandStatusRecords(fixture) assert.equal(statusAfterCrash.at(-1).status, 'done') - const second = startServer(t, fixture, {port}) + const second = startServer(t, fixture, { port }) await waitForServer(second.child, second.getOutput, port) const retryStatus = await postCommandStatus(port, { id: command.id, name: 'addPerson', - value: {name: 'Duplicate'} + value: { name: 'Duplicate' } }) assert.equal(retryStatus.command, command.id) assert.equal(retryStatus.status, 'done') await assertServerStateMatchesOracle(port, fixture, ['Committed']) - assert.equal((await readCommandStatusRecords(fixture)).filter(record => record.status === 'active').length, 1) + assert.equal( + (await readCommandStatusRecords(fixture)).filter( + record => record.status === 'active' + ).length, + 1 + ) assert.equal((await getCommandStatus(port, command.id)).status, 'done') }) -test('repeated active command crashes are marked unsafe instead of replaying forever', async t => { - const fixture = await makeServerFixture(t) - const port = await getOpenPort() - const commandId = 'poison-command' - const crashOptions = { +test('crash ownership cannot be stolen automatically', async t => { + const fixture = await makeServerFixture(t), + port = await getOpenPort() + const first = startServer(t, fixture, { port, runtimeEnvironment: 'test', - faultPoint: 'before-command-done-status', - maxCommandCrashAttempts: 2 - } - - const first = startServer(t, fixture, crashOptions) + faultPoint: 'before-command-done-status' + }) await waitForServer(first.child, first.getOutput, port) await postCommandExpectingCrash(port, { - id: commandId, + id: 'A', name: 'addPerson', - value: {name: 'Poison'} + value: { name: 'A' } }) - - assert.equal((await waitForExit(first.child)).signal, 'SIGKILL') - - const second = startServer(t, fixture, crashOptions) - assert.equal((await waitForExit(second.child)).signal, 'SIGKILL') - - const third = startServer(t, fixture, { - port, - maxCommandCrashAttempts: 2 - }) - await waitForServer(third.child, third.getOutput, port) - - const commandStatus = await getCommandStatus(port, commandId) - assert.equal(commandStatus.status, 'unsafe') - assert.equal(commandStatus.attempt, 2) - - const retryStatus = await postCommandStatus(port, { - id: commandId, - name: 'addPerson', - value: {name: 'Retry'} - }) - assert.equal(retryStatus.command, commandId) - assert.equal(retryStatus.status, 'unsafe') - assert.equal(retryStatus.attempt, 2) - - await assertServerStateMatchesOracle(port, fixture, []) - - const statusFile = await fs.readFile(fixture.commandStatus, 'utf8') - assert.match(statusFile, /"status":"active","attempt":1/) - assert.match(statusFile, /"status":"active","attempt":2/) - assert.match(statusFile, /"status":"unsafe"/) + await waitForExit(first.child) + const second = startServer(t, fixture, { port }) + assert.equal((await waitForExit(second.child)).code, 1) + assert.match(second.getOutput(), /Store is locked/) + assert.equal( + (await readCommandStatusRecords(fixture)).filter( + s => s.status === 'active' + ).length, + 1 + ) }) test('hanging command times out unsafe and later accepted command commits', async t => { @@ -304,11 +243,15 @@ test('hanging command times out unsafe and later accepted command commits', asyn const queuedResponse = await postCommand(port, { id: 'after-timeout', name: 'addPerson', - value: {name: 'After Timeout'} + value: { name: 'After Timeout' } }) assert.equal(queuedResponse.status, 202) - const timeoutStatus = await waitForCommandStatus(port, 'timeout-command', 'unsafe') + const timeoutStatus = await waitForCommandStatus( + port, + 'timeout-command', + 'unsafe' + ) assert.equal(timeoutStatus.code, 504) assert.equal(timeoutStatus.attempt, 1) assert.match(timeoutStatus.message, /command worker timed out after 100ms/) @@ -318,7 +261,7 @@ test('hanging command times out unsafe and later accepted command commits', asyn const retryStatus = await postCommandStatus(port, { id: 'timeout-command', name: 'addPerson', - value: {name: 'Retry'} + value: { name: 'Retry' } }) assert.equal(retryStatus.command, 'timeout-command') assert.equal(retryStatus.status, 'unsafe') @@ -338,12 +281,15 @@ test('hanging command times out unsafe and later accepted command commits', asyn test('hanging load worker fails startup explicitly', async t => { const fixture = await makeServerFixture(t) const loadWorker = path.join(fixture.dir, 'hang-load-worker.mjs') - await fs.writeFile(loadWorker, `import { parentPort } from 'node:worker_threads' + await fs.writeFile( + loadWorker, + `import { parentPort } from 'node:worker_threads' parentPort.on('message', () => { while (true) {} }) -`) +` + ) const port = await getOpenPort() const running = startServer(t, fixture, { port, @@ -361,7 +307,11 @@ test('integrity-enabled command writes changeset digest before done status', asy const fixture = await makeServerFixture(t) const port = await getOpenPort() const integrityFile = getDefaultIntegrityFile(fixture.datafile) - await appendIntegrityRecord(integrityFile, fixture.datafile, await fs.readFile(fixture.datafile)) + await appendIntegrityRecord( + integrityFile, + fixture.datafile, + await fs.readFile(fixture.datafile) + ) const running = startServer(t, fixture, { port, @@ -373,7 +323,7 @@ test('integrity-enabled command writes changeset digest before done status', asy const response = await postCommand(port, { id: commandId, name: 'addPerson', - value: {name: 'Ada'} + value: { name: 'Ada' } }) assert.equal(response.status, 202) await waitForCommandStatus(port, commandId, 'done') @@ -399,13 +349,13 @@ test('integrity-enabled command writes changeset digest before done status', asy test('normal restart preserves committed state according to reconstruction oracle', async t => { const fixture = await makeServerFixture(t) const port = await getOpenPort() - const first = startServer(t, fixture, {port}) + const first = startServer(t, fixture, { port }) await waitForServer(first.child, first.getOutput, port) const response = await postCommand(port, { id: 'normal-command', name: 'addPerson', - value: {name: 'Normal'} + value: { name: 'Normal' } }) assert.equal(response.status, 202) await waitForCommandStatus(port, 'normal-command', 'done') @@ -415,7 +365,7 @@ test('normal restart preserves committed state according to reconstruction oracl await assertServerStateMatchesOracle(port, fixture, ['Normal']) await stopServer(first.child) - const second = startServer(t, fixture, {port}) + const second = startServer(t, fixture, { port }) await waitForServer(second.child, second.getOutput, port) await assertServerStateMatchesOracle(port, fixture, ['Normal']) diff --git a/test/durability-helpers.mjs b/test/durability-helpers.mjs index fdc92f6..5957609 100644 --- a/test/durability-helpers.mjs +++ b/test/durability-helpers.mjs @@ -10,7 +10,9 @@ import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import { getChangesetPath } from '../src/recovery.mjs' -export const rootDir = path.dirname(path.dirname(fileURLToPath(import.meta.url))) +export const rootDir = path.dirname( + path.dirname(fileURLToPath(import.meta.url)) +) const serverModule = path.join(rootDir, 'src/server.mjs') export async function getOpenPort() { @@ -26,7 +28,7 @@ export async function getOpenPort() { export async function makeServerFixture(t, options = {}) { const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-crash-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) + t.after(() => fs.rm(dir, { recursive: true, force: true })) const datafile = path.join(dir, 'data.jsontag') const commandsFile = path.join(dir, 'commands.mjs') @@ -35,21 +37,45 @@ export async function makeServerFixture(t, options = {}) { const commandStatus = path.join(dir, 'command-status.jsontag') const runner = path.join(dir, 'run-server.mjs') - await fs.writeFile(datafile, serialize(JSONTag.parse(options.initialData || '{"persons":[]}'))) - await fs.writeFile(commandsFile, options.commandsSource || `export default { + await fs.writeFile(commandLog, '') + await fs.writeFile(commandStatus, '') + await fs.writeFile( + datafile, + serialize(JSONTag.parse(options.initialData || '{"persons":[]}')) + ) + await fs.writeFile( + commandsFile, + options.commandsSource || + `export default { addPerson: (dataspace, command) => { dataspace.persons.push(command.value) } } -`) - await fs.writeFile(indexFile, options.indexSource || 'export default { create() {}, update() {}, load() { return {} } }\n') - await fs.writeFile(runner, `import SimplyStore from ${JSON.stringify(serverModule)} +` + ) + await fs.writeFile( + indexFile, + options.indexSource || + 'export default { create() {}, update() {}, load() { return {} } }\n' + ) + await fs.writeFile( + runner, + `import SimplyStore from ${JSON.stringify(serverModule)} const options = JSON.parse(process.env.SIMPLYSTORE_TEST_OPTIONS) SimplyStore.run(options) -`) +` + ) - return {dir, datafile, commandsFile, indexFile, commandLog, commandStatus, runner} + return { + dir, + datafile, + commandsFile, + indexFile, + commandLog, + commandStatus, + runner + } } export function startServer(t, fixture, options = {}) { @@ -89,7 +115,7 @@ export function startServer(t, fixture, options = {}) { t.after(() => stopServer(child)) - return {child, getOutput: () => output} + return { child, getOutput: () => output } } export async function stopServer(child) { @@ -100,17 +126,32 @@ export async function stopServer(child) { } export async function waitForServer(child, getOutput, port) { + let finished = false await new Promise((resolve, reject) => { const timeout = setTimeout(() => { + finished = true child.off('exit', onExit) - reject(new Error(`Server did not start on port ${port}:\n${getOutput()}`)) + reject( + new Error( + `Server did not start on port ${port}:\n${getOutput()}` + ) + ) }, 5000) const onExit = (code, signal) => { clearTimeout(timeout) - reject(new Error(`Server exited before startup (${code || signal}):\n${getOutput()}`)) + finished = true + reject( + new Error( + `Server exited before startup (${code || signal}):\n${getOutput()}` + ) + ) } const checkReady = () => { + if (finished) { + return + } if (getOutput().includes(`SimplyStore listening on port ${port}`)) { + finished = true clearTimeout(timeout) child.off('exit', onExit) resolve() @@ -126,7 +167,7 @@ export async function waitForServer(child, getOutput, port) { export async function waitForExit(child, timeoutMs = 5000) { return new Promise((resolve, reject) => { if (child.exitCode !== null || child.signalCode) { - resolve({code: child.exitCode, signal: child.signalCode}) + resolve({ code: child.exitCode, signal: child.signalCode }) return } const timeout = setTimeout(() => { @@ -135,7 +176,7 @@ export async function waitForExit(child, timeoutMs = 5000) { }, timeoutMs) const onExit = (code, signal) => { clearTimeout(timeout) - resolve({code, signal}) + resolve({ code, signal }) } child.once('exit', onExit) }) @@ -163,20 +204,27 @@ export async function queryPersons(port) { signal: AbortSignal.timeout(5000) }) if (!response.ok) { - throw new Error(`Query failed with ${response.status}: ${await response.text()}`) + throw new Error( + `Query failed with ${response.status}: ${await response.text()}` + ) } return JSONTag.parse(await response.text()) } export async function getCommandStatus(port, commandId) { - const response = await fetch(`http://127.0.0.1:${port}/command/${commandId}`, { - headers: { - accept: 'application/json' - }, - signal: AbortSignal.timeout(5000) - }) + const response = await fetch( + `http://127.0.0.1:${port}/command/${commandId}`, + { + headers: { + accept: 'application/json' + }, + signal: AbortSignal.timeout(5000) + } + ) if (!response.ok) { - throw new Error(`Command status failed with ${response.status}: ${await response.text()}`) + throw new Error( + `Command status failed with ${response.status}: ${await response.text()}` + ) } return response.json() } @@ -185,7 +233,8 @@ async function readJsonTagLines(file) { let text try { text = await fs.readFile(file, 'utf8') - } catch (error) { + } + catch (error) { if (error.code === 'ENOENT') { return [] } @@ -220,7 +269,9 @@ export async function reconstructCommittedDataset(fixture) { if (applied.has(command.id) || finalStatus.get(command.id) !== 'done') { continue } - data = parser.parse(await fs.readFile(getChangesetPath(fixture.datafile, command.id))) + data = parser.parse( + await fs.readFile(getChangesetPath(fixture.datafile, command.id)) + ) applied.add(command.id) } return data @@ -230,3 +281,20 @@ export async function reconstructCommittedPersonNames(fixture) { const data = await reconstructCommittedDataset(fixture) return data.persons.map(person => person.name) } + +// Explicit administrator action in disposable fixtures whose writer has exited. +export async function unlockStoppedFixture(t, fixture) { + const { releaseOfflineLocks } = await import('../src/admin-recovery.mjs') + const { publishFile } = await import('../src/storage.mjs') + const audit = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-unlock-audit-') + ) + t.after(() => fs.rm(audit, { recursive: true, force: true })) + const release = await releaseOfflineLocks(fixture, { + operator: 'test administrator', + reason: 'Fixture writer has exited; inspect before startup', + confirmedStopped: true + }) + await publishFile(path.join(audit, 'release.json'), JSON.stringify(release)) + await release.finish() +} diff --git a/test/durability-latency.mjs b/test/durability-latency.mjs new file mode 100644 index 0000000..c5d4926 --- /dev/null +++ b/test/durability-latency.mjs @@ -0,0 +1,69 @@ +import process from 'node:process' +import test from 'node:test' +import assert from 'node:assert/strict' +import { performance } from 'node:perf_hooks' +import { execFileSync } from 'node:child_process' +import { + makeServerFixture, + getOpenPort, + startServer, + waitForServer, + postCommand, + getCommandStatus +} from './durability-helpers.mjs' + +test('measure local startup, acceptance, and completion latency', async t => { + const store = await makeServerFixture(t), + port = await getOpenPort() + const boot = performance.now(), + server = startServer(t, store, { port }) + await waitForServer(server.child, server.getOutput, port) + const startupMs = performance.now() - boot, + accepted = [], + completed = [] + for (let i = 0; i < 30; i++) { + const start = performance.now(), + id = `latency-${i}` + assert.equal( + ( + await postCommand(port, { + id, + name: 'addPerson', + value: { name: id } + }) + ).status, + 202 + ) + accepted.push(performance.now() - start) + for (;;) { + const status = await getCommandStatus(port, id) + if (status.status === 'done') { + break + } + assert.equal(['accepted', 'active'].includes(status.status), true) + await new Promise(r => setTimeout(r, 2)) + } + completed.push(performance.now() - start) + } + const stats = values => { + const v = [...values].sort((a, b) => a - b) + return { + p50Ms: v[Math.floor(v.length * 0.5)], + p95Ms: v[Math.floor(v.length * 0.95)], + maxMs: v.at(-1) + } + } + t.diagnostic( + JSON.stringify({ + samples: 30, + startupMs, + acceptance: stats(accepted), + completion: stats(completed), + filesystem: execFileSync('stat', ['-f', '-c', '%T', store.dir], { + encoding: 'utf8' + }).trim(), + node: process.version, + notes: 'Single localhost client; completion includes a fresh worker and polling; no production performance claim' + }) + ) +}) diff --git a/test/durability-startup.mjs b/test/durability-startup.mjs index c8d3d92..b5ad9cc 100644 --- a/test/durability-startup.mjs +++ b/test/durability-startup.mjs @@ -9,18 +9,18 @@ import { fileURLToPath } from 'node:url' import JSONTag from '@muze-nl/jsontag' import Parser from '@muze-nl/od-jsontag/src/parse.mjs' import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' -import { appendIntegrityRecord, getDefaultIntegrityFile } from '../src/integrity.mjs' import { - activeCommandStatus, + appendIntegrityRecord, + getDefaultIntegrityFile +} from '../src/integrity.mjs' +import { getCommittedCommandIds, getChangesetPath, assertOdJsonTagFraming, loadCommandLog, loadCommandStatus, - pendingCommandStatus, recoverActiveCommands, - RecoveryIntegrityError, - unsafeCommandStatus + RecoveryIntegrityError } from '../src/recovery.mjs' const rootDir = path.dirname(path.dirname(fileURLToPath(import.meta.url))) @@ -32,21 +32,29 @@ function parseOd(buffer) { } async function makeFixture(t) { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-durability-')) - t.after(() => fs.rm(dir, {recursive: true, force: true})) + const dir = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-durability-') + ) + t.after(() => fs.rm(dir, { recursive: true, force: true })) const dataFile = path.join(dir, 'data.jsontag') const indexFile = path.join(dir, 'index.mjs') const base = serialize(JSONTag.parse('{"persons":[]}')) await fs.writeFile(dataFile, base) - await fs.writeFile(indexFile, 'export default { create() {}, update() {}, load() { return {} } }\n') + await fs.writeFile( + indexFile, + 'export default { create() {}, update() {}, load() { return {} } }\n' + ) - return {dir, dataFile, indexFile} + return { dir, dataFile, indexFile } } async function writeJsonTagLines(file, records) { - await fs.writeFile(file, records.map(record => JSONTag.stringify(record)).join('\n') + '\n') + await fs.writeFile( + file, + records.map(record => JSONTag.stringify(record)).join('\n') + '\n' + ) } async function writeChangeset(dataFile, commandId, change) { @@ -57,14 +65,26 @@ async function writeChangeset(dataFile, commandId, change) { change(data) const extension = dataFile.split('.').pop() - const basefile = dataFile.substring(0, dataFile.length - (extension.length + 1)) - await fs.writeFile(`${basefile}.${commandId}.${extension}`, serialize(data, { - meta: parser.meta, - changes: true - })) + const basefile = dataFile.substring( + 0, + dataFile.length - (extension.length + 1) + ) + await fs.writeFile( + `${basefile}.${commandId}.${extension}`, + serialize(data, { + meta: parser.meta, + changes: true + }) + ) } -function loadDataset({dataFile, indexFile, commands, integrityFile = null, integrityRequired = false}) { +function loadDataset({ + dataFile, + indexFile, + commands, + integrityFile = null, + integrityRequired = false +}) { return new Promise((resolve, reject) => { const worker = new Worker(loadWorker) worker.on('message', result => { @@ -93,11 +113,14 @@ async function writeIntegrityEntry(integrityFile, file) { test('accepted command changeset is not treated as committed state on startup', async t => { const fixture = await makeFixture(t) await writeChangeset(fixture.dataFile, 'accepted-command', data => { - data.persons.push({name: 'Ada'}) + data.persons.push({ name: 'Ada' }) }) const status = new Map([ - ['accepted-command', {command: 'accepted-command', code: 202, status: 'accepted'}] + [ + 'accepted-command', + { command: 'accepted-command', code: 202, status: 'accepted' } + ] ]) const result = await loadDataset({ ...fixture, @@ -105,13 +128,24 @@ test('accepted command changeset is not treated as committed state on startup', }) const data = parseOd(result.data) - assert.equal(data.persons.length, 0, 'accepted-but-not-done changesets must not become visible after restart') + assert.equal( + data.persons.length, + 0, + 'accepted-but-not-done changesets must not become visible after restart' + ) }) test('done command with missing changeset refuses recovery instead of silently using base data', async t => { const fixture = await makeFixture(t) const status = new Map([ - ['done-command-with-missing-changeset', {command: 'done-command-with-missing-changeset', code: 200, status: 'done'}] + [ + 'done-command-with-missing-changeset', + { + command: 'done-command-with-missing-changeset', + code: 200, + status: 'done' + } + ] ]) await assert.rejects( @@ -140,9 +174,15 @@ test('malformed base OD-JSONTag file refuses recovery', async t => { test('malformed committed changeset refuses recovery', async t => { const fixture = await makeFixture(t) const status = new Map([ - ['malformed-changeset', {command: 'malformed-changeset', code: 200, status: 'done'}] + [ + 'malformed-changeset', + { command: 'malformed-changeset', code: 200, status: 'done' } + ] ]) - await fs.writeFile(getChangesetPath(fixture.dataFile, 'malformed-changeset'), '{"persons":[{"name":"Bad"}]}') + await fs.writeFile( + getChangesetPath(fixture.dataFile, 'malformed-changeset'), + '{"persons":[{"name":"Bad"}]}' + ) await assert.rejects( loadDataset({ @@ -157,14 +197,17 @@ test('truncated committed changeset payload refuses recovery before lazy parsing const fixture = await makeFixture(t) const commandId = 'truncated-changeset' const status = new Map([ - [commandId, {command: commandId, code: 200, status: 'done'}] + [commandId, { command: commandId, code: 200, status: 'done' }] ]) await writeChangeset(fixture.dataFile, commandId, data => { - data.persons.push({name: 'Truncated'}) + data.persons.push({ name: 'Truncated' }) }) const changesetPath = getChangesetPath(fixture.dataFile, commandId) const changeset = await fs.readFile(changesetPath) - await fs.writeFile(changesetPath, changeset.subarray(0, changeset.length - 3)) + await fs.writeFile( + changesetPath, + changeset.subarray(0, changeset.length - 3) + ) await assert.rejects( loadDataset({ @@ -177,9 +220,15 @@ test('truncated committed changeset payload refuses recovery before lazy parsing test('malformed uncommitted changeset is ignored during committed startup reconstruction', async t => { const fixture = await makeFixture(t) - await fs.writeFile(getChangesetPath(fixture.dataFile, 'accepted-command'), '{"persons":[{"name":"Ignored"}]}') + await fs.writeFile( + getChangesetPath(fixture.dataFile, 'accepted-command'), + '{"persons":[{"name":"Ignored"}]}' + ) const status = new Map([ - ['accepted-command', {command: 'accepted-command', code: 202, status: 'accepted'}] + [ + 'accepted-command', + { command: 'accepted-command', code: 202, status: 'accepted' } + ] ]) const result = await loadDataset({ @@ -196,18 +245,18 @@ test('durable status file selects only done command changesets for startup', asy const statusFile = path.join(fixture.dir, 'command-status.jsontag') await writeChangeset(fixture.dataFile, 'accepted-command', data => { - data.persons.push({name: 'Accepted'}) + data.persons.push({ name: 'Accepted' }) }) await writeChangeset(fixture.dataFile, 'done-command', data => { - data.persons.push({name: 'Done'}) + data.persons.push({ name: 'Done' }) }) await writeJsonTagLines(statusFile, [ - {command: 'accepted-command', code: 202, status: 'accepted'}, - {command: 'done-command', code: 202, status: 'accepted'}, - {command: 'done-command', code: 200, status: 'done'}, - {command: 'failed-command', code: 202, status: 'accepted'}, - {command: 'failed-command', code: 500, status: 'failed'} + { command: 'accepted-command', code: 202, status: 'accepted' }, + { command: 'done-command', code: 202, status: 'accepted' }, + { command: 'done-command', code: 200, status: 'done' }, + { command: 'failed-command', code: 202, status: 'accepted' }, + { command: 'failed-command', code: 500, status: 'failed' } ]) const status = loadCommandStatus(statusFile) @@ -217,7 +266,10 @@ test('durable status file selects only done command changesets for startup', asy }) const data = parseOd(result.data) - assert.deepEqual(data.persons.map(person => person.name), ['Done']) + assert.deepEqual( + data.persons.map(person => person.name), + ['Done'] + ) }) test('durable command log replays only accepted commands', async t => { @@ -226,18 +278,22 @@ test('durable command log replays only accepted commands', async t => { const commandLog = path.join(fixture.dir, 'command-log.jsontag') await writeJsonTagLines(statusFile, [ - {command: 'accepted-command', code: 202, status: 'accepted'}, - {command: 'done-command', code: 200, status: 'done'}, - {command: 'failed-command', code: 500, status: 'failed'} + { command: 'accepted-command', code: 202, status: 'accepted' }, + { command: 'done-command', code: 200, status: 'done' }, + { command: 'failed-command', code: 500, status: 'failed' } ]) await writeJsonTagLines(commandLog, [ - {id: 'accepted-command', name: 'addPerson', value: {name: 'Accepted'}}, - {id: 'done-command', name: 'addPerson', value: {name: 'Done'}}, - {id: 'failed-command', name: 'addPerson', value: {name: 'Failed'}} + { + id: 'accepted-command', + name: 'addPerson', + value: { name: 'Accepted' } + }, + { id: 'done-command', name: 'addPerson', value: { name: 'Done' } }, + { id: 'failed-command', name: 'addPerson', value: { name: 'Failed' } } ]) const taskDefaults = { - meta: {source: 'test-meta'}, + meta: { source: 'test-meta' }, data: ['test-data'], commandsFile: '/commands.mjs', indexFile: '/index.mjs', @@ -251,54 +307,32 @@ test('durable command log replays only accepted commands', async t => { assert.deepEqual(commands[0], { ...taskDefaults, id: 'accepted-command', - command: JSONTag.stringify({id: 'accepted-command', name: 'addPerson', value: {name: 'Accepted'}}), - request: null + command: JSONTag.stringify({ + id: 'accepted-command', + name: 'addPerson', + value: { name: 'Accepted' } + }) }) assert.equal(JSONTag.parse(commands[0].command).name, 'addPerson') }) -test('active command below crash threshold is accepted for replay on startup', async t => { - const fixture = await makeFixture(t) - const statusFile = path.join(fixture.dir, 'command-status.jsontag') - const commandLog = path.join(fixture.dir, 'command-log.jsontag') - - await writeJsonTagLines(statusFile, [ - {command: 'crashed-command', code: 102, status: activeCommandStatus, attempt: 1} - ]) - await writeJsonTagLines(commandLog, [ - {id: 'crashed-command', name: 'addPerson', value: {name: 'Retry'}} - ]) - - const status = loadCommandStatus(statusFile) - await recoverActiveCommands(status, statusFile, {maxCrashAttempts: 2}) - const commands = loadCommandLog(status, commandLog) - - assert.equal(status.get('crashed-command').status, pendingCommandStatus) - assert.equal(commands.length, 1) - assert.equal(commands[0].id, 'crashed-command') - assert.match(await fs.readFile(statusFile, 'utf8'), /"status":"accepted"/) -}) - -test('active command at crash threshold is marked unsafe and not replayed', async t => { - const fixture = await makeFixture(t) - const statusFile = path.join(fixture.dir, 'command-status.jsontag') - const commandLog = path.join(fixture.dir, 'command-log.jsontag') - - await writeJsonTagLines(statusFile, [ - {command: 'poison-command', code: 102, status: activeCommandStatus, attempt: 2} - ]) - await writeJsonTagLines(commandLog, [ - {id: 'poison-command', name: 'addPerson', value: {name: 'Unsafe'}} - ]) - - const status = loadCommandStatus(statusFile) - await recoverActiveCommands(status, statusFile, {maxCrashAttempts: 2}) - const commands = loadCommandLog(status, commandLog) - - assert.equal(status.get('poison-command').status, unsafeCommandStatus) - assert.equal(commands.length, 0) - assert.match(await fs.readFile(statusFile, 'utf8'), /"status":"unsafe"/) -}) +for (const attempt of [1, 2]) { + test(`active command at attempt ${attempt} remains unchanged pending administrator assessment`, async t => { + const fixture = await makeFixture(t) + const file = path.join(fixture.dir, 'status.jsontag') + await writeJsonTagLines(file, [ + { command: 'A', status: 'active', attempt } + ]) + const before = await fs.readFile(file) + const status = loadCommandStatus(file) + await assert.rejects( + recoverActiveCommands(status, file, { maxCrashAttempts: 2 }), + /administrator assessment/ + ) + assert.equal(status.get('A').status, 'active') + assert.deepEqual(await fs.readFile(file), before) + }) +} test('malformed durable status record refuses recovery with explicit integrity error', async t => { const fixture = await makeFixture(t) @@ -308,7 +342,8 @@ test('malformed durable status record refuses recovery with explicit integrity e let error try { loadCommandStatus(statusFile) - } catch (caught) { + } + catch (caught) { error = caught } @@ -322,14 +357,13 @@ test('malformed durable status record refuses recovery with explicit integrity e test('structurally invalid durable status record refuses recovery', async t => { const fixture = await makeFixture(t) const statusFile = path.join(fixture.dir, 'command-status.jsontag') - await writeJsonTagLines(statusFile, [ - {code: 202, status: 'accepted'} - ]) + await writeJsonTagLines(statusFile, [{ code: 202, status: 'accepted' }]) let error try { loadCommandStatus(statusFile) - } catch (caught) { + } + catch (caught) { error = caught } @@ -349,7 +383,8 @@ test('malformed durable command log record refuses recovery with explicit integr let error try { loadCommandLog(status, commandLog) - } catch (caught) { + } + catch (caught) { error = caught } @@ -365,13 +400,14 @@ test('structurally invalid durable command log record refuses recovery', async t const commandLog = path.join(fixture.dir, 'command-log.jsontag') const status = new Map() await writeJsonTagLines(commandLog, [ - {name: 'addPerson', value: {name: 'No id'}} + { name: 'addPerson', value: { name: 'No id' } } ]) let error try { loadCommandLog(status, commandLog) - } catch (caught) { + } + catch (caught) { error = caught } @@ -383,12 +419,20 @@ test('structurally invalid durable command log record refuses recovery', async t }) test('OD-JSONTag framing validation catches truncated lazy records', () => { - const buffer = Buffer.from('(16){"persons":[~1]}\n(14){"name":"Ada"', 'utf8') + const buffer = Buffer.from( + '(16){"persons":[~1]}\n(14){"name":"Ada"', + 'utf8' + ) let error try { - assertOdJsonTagFraming(buffer, 'data.truncated.jsontag', 'changeset OD-JSONTag data') - } catch (caught) { + assertOdJsonTagFraming( + buffer, + 'data.truncated.jsontag', + 'changeset OD-JSONTag data' + ) + } + catch (caught) { error = caught } @@ -399,10 +443,17 @@ test('OD-JSONTag framing validation catches truncated lazy records', () => { }) test('OD-JSONTag framing validation accepts changeset skip records', () => { - const buffer = Buffer.from('(18){"persons":[~1-2]}\n+1\n(15){"name":"Once"}', 'utf8') + const buffer = Buffer.from( + '(18){"persons":[~1-2]}\n+1\n(15){"name":"Once"}', + 'utf8' + ) assert.doesNotThrow(() => { - assertOdJsonTagFraming(buffer, 'data.patch.jsontag', 'changeset OD-JSONTag data') + assertOdJsonTagFraming( + buffer, + 'data.patch.jsontag', + 'changeset OD-JSONTag data' + ) }) }) @@ -429,12 +480,12 @@ test('integrity manifest detects same-length altered committed changeset payload const fixture = await makeFixture(t) const commandId = 'tampered-changeset' const status = new Map([ - [commandId, {command: commandId, code: 200, status: 'done'}] + [commandId, { command: commandId, code: 200, status: 'done' }] ]) const integrityFile = getDefaultIntegrityFile(fixture.dataFile) await writeIntegrityEntry(integrityFile, fixture.dataFile) await writeChangeset(fixture.dataFile, commandId, data => { - data.persons.push({name: 'Ada'}) + data.persons.push({ name: 'Ada' }) }) const changesetPath = getChangesetPath(fixture.dataFile, commandId) await writeIntegrityEntry(integrityFile, changesetPath) diff --git a/test/power-loss-baseline.mjs b/test/power-loss-baseline.mjs new file mode 100644 index 0000000..e88b00d --- /dev/null +++ b/test/power-loss-baseline.mjs @@ -0,0 +1,339 @@ +import { Buffer } from 'node:buffer' +// The original baseline is retained at EVD-20260919-TTZ7C-23's Git version. +// These assertions implement the corrected command-only / authoritative-log +// contracts. +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import path from 'node:path' +import JSONTag from '@muze-nl/jsontag' +import { + fixture, + launch, + submit, + completion, + eventually, + rows, + crash, + copyImage, + loseUnpublishedName +} from './power-loss-helpers.mjs' +import { + getOpenPort, + waitForServer, + waitForExit, + queryPersons, + readCommandLogRecords, + readCommandStatusRecords, + unlockStoppedFixture +} from './durability-helpers.mjs' +import { inspectStore } from '../src/store-inspection.mjs' + +async function ready(t, store, options = {}) { + const port = await getOpenPort(), + server = await launch(t, store, port, options) + await waitForServer(server.child, server.getOutput, port) + return { ...server, port } +} +const command = id => ({ id, name: 'addPerson', value: { name: id } }) +async function commit(store, server, id = 'A') { + assert.equal((await submit(store, server.port, command(id))).http, 202) + assert.equal((await completion(store, server.port, id)).status, 'done') +} + +test('PL01-05 durable publication survives modeled loss of unsynced names and restart', async t => { + const store = await fixture(t), + server = await ready(t, store) + await commit(store, server) + const trace = await rows(store.trace) + const changeset = path.join(store.dir, 'data.A.jsontag') + const rename = trace.findIndex( + e => e.op === 'rename' && e.file === changeset + ) + assert.ok(rename > 0) + assert.ok( + trace + .slice(0, rename) + .some(e => e.op === 'sync' && e.file === trace[rename].from) + ) + const dirSync = trace.findIndex( + (e, i) => i > rename && e.op === 'sync' && e.file === store.dir + ) + const doneSync = trace.findIndex( + (e, i) => + i > dirSync && + e.op === 'datasync' && + e.file === store.commandStatus && + Buffer.from(e.bytes, 'base64') + .toString() + .includes('"status":"done"') + ) + assert.ok(dirSync > rename && doneSync > dirSync) + await crash(server.child) + const image = await copyImage(t, store) + assert.equal(await loseUnpublishedName(store, image, changeset), false) + await unlockStoppedFixture(t, image) + const restarted = await ready(t, image) + assert.deepEqual( + (await queryPersons(restarted.port)).map(p => p.name), + ['A'] + ) +}) + +test('PL10 an actual short changeset write is completed before acknowledgment', async t => { + const store = await fixture(t), + server = await ready(t, store, { + fault: { + op: 'write', + file: 'data.A.jsontag.', + action: 'short-write' + } + }) + await commit(store, server) + const trace = await rows(store.trace) + assert.ok( + trace.some(e => e.op === 'injected' && e.action === 'short-write') + ) + assert.ok( + trace.filter( + e => + e.op === 'write' && + path.basename(e.file).startsWith('data.A.jsontag.') + ).length >= 2 + ) + assert.deepEqual( + (await inspectStore(store)).data.persons.map(p => p.name), + ['A'] + ) +}) +for (const [name, op, file, contains] of [ + ['log sync', 'datasync', 'command-log.jsontag', '"id":"A"'], + [ + 'acceptance sync', + 'datasync', + 'command-status.jsontag', + '"status":"accepted"' + ], + ['log close', 'close:after', 'command-log.jsontag', '"id":"A"'], + ['changeset sync', 'sync', 'data.A.jsontag.'], + ['changeset rename', 'rename', 'data.A.jsontag'], + ['done sync', 'datasync', 'command-status.jsontag', '"status":"done"'] +]) { + test(`PL07-09/11/18-20 ${name} failure stops mutation without false terminal success`, async t => { + const store = await fixture(t), + server = await ready(t, store, { + fault: { op, file, contains, action: 'error' } + }) + try { + await submit(store, server.port, command('A')) + } + catch (error) { + assert.match(error.message, /fetch failed|terminated|socket/) + } + assert.equal((await waitForExit(server.child)).code, 1) + assert.match(server.getOutput(), /Storage outcome uncertain/) + assert.ok((await rows(store.trace)).some(e => e.op === 'injected')) + const status = await readCommandStatusRecords(store) + // Failed sync can leave a complete but uncertain done record; never + // append failed after it. + const done = status.findIndex(s => s.status === 'done') + if (done >= 0) { + assert.equal( + status.slice(done + 1).some(s => s.status === 'failed'), + false + ) + } + else { + assert.equal( + status.some(s => s.status === 'done'), + false + ) + } + }) +} +test('PL15 configured workers cannot receive HTTP execution inputs; command values roundtrip', async t => { + const store = await fixture(t, { + commandsSource: `export default {addPerson(data,command,request,meta){ + if(request!==undefined)throw new Error('unlogged input leaked') + if(!meta.index.id)throw new Error('index unavailable') + data.persons.push(command.value) + }}` + }) + const worker = path.join(store.dir, 'custom-worker.mjs') + await fs.writeFile( + worker, + `import {parentPort} from 'node:worker_threads' +import runCommand,{initialize} from ${JSON.stringify(new URL('../src/command-worker-module.mjs', import.meta.url).href)} +parentPort.on('message',async task=>{if('request' in task)throw new Error('request leaked to worker');await initialize(task);parentPort.postMessage(await runCommand(task.command,{query:{name:'wrong'}}))})` + ) + const server = await ready(t, store, { server: { commandWorker: worker } }) + const input = { + id: 'A', + name: 'addPerson', + value: { name: 'café 漢字 😀', date: new Date('2026-09-19T00:00:00Z') } + } + assert.equal( + (await submit(store, server.port, input, '?name=wrong')).http, + 202 + ) + assert.equal((await completion(store, server.port, 'A')).status, 'done') + assert.equal( + JSONTag.stringify((await readCommandLogRecords(store))[0]), + JSONTag.stringify(input) + ) + assert.equal((await queryPersons(server.port))[0].name, input.value.name) +}) +for (const duplicate of [false, true]) { + test(`PL16 delayed append cannot be overtaken${duplicate ? ' by duplicate ID' : ''}`, async t => { + const store = await fixture(t), + server = await ready(t, store, { + fault: { + op: 'datasync:after', + file: 'command-log.jsontag', + contains: '"id":"A"', + action: 'pause' + } + }) + const a = submit(store, server.port, command('A')) + await eventually( + async () => + (await rows(store.trace)).some(e => e.op === 'injected'), + 'A append gate' + ) + const b = submit(store, server.port, command(duplicate ? 'A' : 'B')) + await new Promise(resolve => setTimeout(resolve, 30)) + assert.deepEqual( + (await readCommandLogRecords(store)).map(c => c.id), + ['A'] + ) + await fs.writeFile(store.release, '') + assert.equal((await a).http, 202) + assert.ok([200, 202].includes((await b).http)) + await completion(store, server.port, duplicate ? 'A' : 'B') + const expected = duplicate ? ['A'] : ['A', 'B'] + assert.deepEqual( + (await readCommandLogRecords(store)).map(c => c.id), + expected + ) + assert.deepEqual( + (await readCommandStatusRecords(store)) + .filter(s => s.status === 'active') + .map(s => s.command), + expected + ) + assert.deepEqual( + (await queryPersons(server.port)).map(p => p.name), + expected + ) + }) +} +for (const artifact of [ + 'command-log.jsontag', + 'command-status.jsontag', + 'data.A.jsontag' +]) { + test(`PL02-04/06 missing ${artifact} fails open rather than recreating history`, async t => { + const store = await fixture(t), + server = await ready(t, store) + await commit(store, server) + await crash(server.child) + await unlockStoppedFixture(t, store) + await fs.unlink(path.join(store.dir, artifact)) + const before = await fs.readdir(store.dir) + const restarted = await launch(t, store, await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code, 1) + assert.match(restarted.getOutput(), /Administrative recovery required/) + assert.deepEqual((await fs.readdir(store.dir)).sort(), before.sort()) + }) +} +test('PL12/17 startup preserves uncertain active work without repeating external effects', async t => { + const store = await fixture(t, { + commandsSource: `import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify('/tmp/placeholder-effect')},'') ;data.persons.push(command.value)}}` + }) + const witness = path.join(store.audit, 'effects') + await fs.writeFile( + store.commandsFile, + `import fs from 'node:fs'; export default {addPerson(data,command){fs.appendFileSync(${JSON.stringify(witness)},command.id+'\\n');data.persons.push(command.value)}}` + ) + const server = await ready(t, store, { + fault: { + op: 'write', + file: 'command-status.jsontag', + contains: '"status":"done"', + action: 'error' + } + }) + await submit(store, server.port, command('A')) + await waitForExit(server.child).catch(error => { + throw new Error(server.getOutput(), { cause: error }) + }) + const before = await fs.readFile(store.commandStatus) + await unlockStoppedFixture(t, store) + const restarted = await launch(t, store, await getOpenPort()) + assert.equal((await waitForExit(restarted.child)).code, 1) + assert.deepEqual(await fs.readFile(store.commandStatus), before) + assert.equal(await fs.readFile(witness, 'utf8'), 'A\n') +}) + +for (const operation of ['sync', 'close:after']) { + test(`directory ${operation} failure after rename prevents done`, async t => { + const store = await fixture(t) + const server = await ready(t, store, { + fault: { + op: operation, + file: path.basename(store.dir), + action: 'error', + after: { op: 'rename', file: 'data.A.jsontag' } + } + }) + await submit(store, server.port, command('A')) + assert.equal((await waitForExit(server.child)).code, 1) + assert.ok( + (await rows(store.trace)).some( + e => e.op === 'injected' && e.file === store.dir + ) + ) + assert.equal( + (await readCommandStatusRecords(store)).some( + c => c.status === 'done' + ), + false + ) + }) +} +for (const operation of ['open', 'write', 'close:after']) { + test(`changeset ${operation} error is not masked or acknowledged`, async t => { + const store = await fixture(t) + const server = await ready(t, store, { + fault: { + op: operation, + file: 'data.A.jsontag.', + action: 'error', + code: operation === 'write' ? 'ENOSPC' : 'EIO' + } + }) + await submit(store, server.port, command('A')) + assert.equal((await waitForExit(server.child)).code, 1) + assert.equal( + (await readCommandStatusRecords(store)).some( + c => c.status === 'done' + ), + false + ) + assert.match(server.getOutput(), /Storage outcome uncertain/) + }) +} +test('short command-log append preserves its entire JSONTag record', async t => { + const store = await fixture(t) + const server = await ready(t, store, { + fault: { + op: 'write', + file: 'command-log.jsontag', + contains: '"id":"A"', + action: 'short-write' + } + }) + await commit(store, server) + assert.deepEqual((await readCommandLogRecords(store))[0], command('A')) + assert.ok((await rows(store.trace)).some(e => e.op === 'injected')) +}) diff --git a/test/power-loss-helpers.mjs b/test/power-loss-helpers.mjs new file mode 100644 index 0000000..7b2d390 --- /dev/null +++ b/test/power-loss-helpers.mjs @@ -0,0 +1,199 @@ +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import process from 'node:process' +import { spawn } from 'node:child_process' +import { fileURLToPath } from 'node:url' +import JSONTag from '@muze-nl/jsontag' +import { + makeServerFixture, + rootDir, + stopServer, + waitForExit, + getCommandStatus +} from './durability-helpers.mjs' + +export async function fixture(t, options = {}) { + const store = await makeServerFixture(t, options) + const audit = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-observer-') + ) + t.after(() => fs.rm(audit, { recursive: true, force: true })) + const trace = path.join(audit, 'io.jsonl') + const journal = path.join(audit, 'client.jsonl') + const release = path.join(audit, 'release') + await fs.writeFile(trace, '') + await fs.writeFile(journal, '') + return { ...store, audit, trace, journal, release } +} + +export async function rows(file) { + return (await fs.readFile(file, 'utf8')) + .split('\n') + .filter(Boolean) + .map(line => JSON.parse(line)) +} + +export async function observe(store, event) { + await fs.appendFile(store.journal, JSON.stringify(event) + '\n') +} + +export async function submit(store, port, command, query = '') { + await observe(store, { event: 'issued', command, query }) + const response = await fetch(`http://127.0.0.1:${port}/command${query}`, { + method: 'POST', + headers: { + accept: 'application/json', + 'content-type': 'application/jsontag' + }, + body: JSONTag.stringify(command), + signal: AbortSignal.timeout(5000) + }) + const text = await response.text() + let body + try { + body = JSON.parse(text) + } + catch { + body = { raw: text } + } + await observe(store, { + event: 'response', + id: command.id, + http: response.status, + body + }) + return { http: response.status, body } +} + +export async function eventually(fn, description, timeout = 6000) { + const end = Date.now() + timeout + while (Date.now() < end) { + const result = await fn() + if (result) { + return result + } + await new Promise(resolve => setTimeout(resolve, 10)) + } + assert.fail(`Timed out: ${description}`) +} + +export async function completion(store, port, id) { + const result = await eventually(async () => { + const status = await getCommandStatus(port, id) + return ['done', 'failed', 'unsafe'].includes(status.status) && status + }, `completion of ${id}`) + await observe(store, { event: 'completion', id, body: result }) + return result +} + +export async function launch(t, store, port, options = {}) { + const config = path.join( + store.audit, + `config-${Math.random().toString(16).slice(2)}.json` + ) + await fs.writeFile( + config, + JSON.stringify({ + root: store.dir, + trace: store.trace, + release: store.release, + fault: options.fault + }) + ) + const child = spawn( + process.execPath, + [ + '--import', + fileURLToPath(new URL('./power-loss-preload.mjs', import.meta.url)), + store.runner + ], + { + cwd: store.dir, + env: { + ...process.env, + SIMPLYSTORE_ENV: 'test', + SIMPLYSTORE_FAULT_POINT: '', + SIMPLYSTORE_BASELINE_IO: config, + SIMPLYSTORE_TEST_OPTIONS: JSON.stringify({ + port, + datafile: store.datafile, + commandsFile: store.commandsFile, + indexFile: store.indexFile, + commandLog: store.commandLog, + commandStatus: store.commandStatus, + wwwroot: path.join(rootDir, 'www'), + maxWorkers: 1, + commandTimeout: 10000, + ...options.server + }) + }, + stdio: ['ignore', 'pipe', 'pipe'] + } + ) + let output = '' + child.stdout.on('data', data => { + output += data + }) + child.stderr.on('data', data => { + output += data + }) + t.after(() => stopServer(child)) + return { child, getOutput: () => output } +} + +export async function crash(child) { + child.kill('SIGKILL') + const exit = await waitForExit(child) + assert.equal(exit.signal, 'SIGKILL') +} + +export async function copyImage(t, store) { + const dir = await fs.mkdtemp( + path.join(os.tmpdir(), 'simplystore-crash-image-') + ) + t.after(() => fs.rm(dir, { recursive: true, force: true })) + await fs.cp(store.dir, dir, { recursive: true }) + const image = { ...store, dir } + for (const key of [ + 'datafile', + 'commandsFile', + 'indexFile', + 'commandLog', + 'commandStatus', + 'runner' + ]) { + image[key] = path.join(dir, path.basename(store[key])) + } + return image +} + +export async function loseUnpublishedName(store, image, original) { + const trace = await rows(store.trace) + const publication = trace.findLastIndex( + e => + e.file === original && + (e.op === 'rename' || (e.op === 'open' && !e.existed)) + ) + assert.ok(publication >= 0, `trace contains publication of ${original}`) + const durable = trace + .slice(publication + 1) + .some(e => e.file === path.dirname(original) && e.op === 'sync') + if (!durable) { + await fs.rm(path.join(image.dir, path.basename(original))) + } + return !durable +} + +export async function requirement(t, id, actual, expected, message) { + let todo = 'Known baseline violation; run test:power-loss:strict' + if (process.env.SIMPLYSTORE_BASELINE_STRICT === '1') { + todo = false + } + await t.test( + `${id}: ${message}`, + { todo }, + () => assert.deepEqual(actual, expected, message) + ) +} diff --git a/test/power-loss-preload.mjs b/test/power-loss-preload.mjs new file mode 100644 index 0000000..410e862 --- /dev/null +++ b/test/power-loss-preload.mjs @@ -0,0 +1,232 @@ +// Test-only I/O observation/injection. Loaded explicitly in disposable servers. +import fs from 'node:fs' +import fsp from 'node:fs/promises' +import path from 'node:path' +import process from 'node:process' +import { threadId } from 'node:worker_threads' +import { syncBuiltinESMExports } from 'node:module' + +const configFile = process.env.SIMPLYSTORE_BASELINE_IO +if (configFile) { + const config = JSON.parse(fs.readFileSync(configFile, 'utf8')) + const root = path.resolve(config.root) + const raw = Object.fromEntries( + [ + 'open', + 'write', + 'fsync', + 'close', + 'rename', + 'appendFileSync', + 'readFileSync', + 'existsSync', + 'statSync', + 'writeFileSync' + ].map(key => [key, fs[key].bind(fs)]) + ) + const open = fsp.open.bind(fsp) + const rename = fsp.rename.bind(fsp) + const handles = new Map() + let injected = false + const scoped = file => + typeof file === 'string' && + (file === root || file.startsWith(root + path.sep)) + function record(op, file, extra = {}) { + if (scoped(file)) { + raw.appendFileSync( + config.trace, + JSON.stringify({ op, file, threadId, ...extra }) + '\n' + ) + } + } + function snapshot(file) { + if (raw.statSync(file).isDirectory()) { + return {} + } + const bytes = raw.readFileSync(file).toString('base64') + return { bytes } + } + async function fault(op, file, content = '') { + const rule = config.fault + if ( + injected || + !rule || + rule.op !== op || + !scoped(file) || + !path.basename(file).startsWith(rule.file) || + (rule.contains && !content.includes(rule.contains)) + ) { + return false + } + if ( + rule.after && + !raw + .readFileSync(config.trace, 'utf8') + .split('\n') + .filter(Boolean) + .map(JSON.parse) + .some( + event => + event.op === rule.after.op && + path.basename(event.file) === rule.after.file + ) + ) { + return false + } + injected = true + record('injected', file, { operation: op, action: rule.action }) + if (rule.action === 'pause') { + const deadline = Date.now() + 15000 + while (!raw.existsSync(config.release)) { + if (Date.now() > deadline) { + throw new Error('Baseline gate was not released') + } + await new Promise(resolve => setTimeout(resolve, 5)) + } + } + else if (rule.action === 'short-write') { + return true + } + else { + throw Object.assign( + new Error(`Injected ${rule.code || 'EIO'} at ${op}`), + { code: rule.code || 'EIO' } + ) + } + return false + } + + fsp.rename = async function (from, to) { + const file = path.resolve(String(to)) + await fault('rename', file) + await rename(from, to) + record('rename', file, { from: path.resolve(String(from)) }) + } + fsp.open = async function (file, ...args) { + const absolute = path.resolve(String(file)) + const existed = raw.existsSync(absolute) + await fault('open', absolute) + const handle = await open(file, ...args) + if (!scoped(absolute)) { + return handle + } + record('open', absolute, { existed, flags: args[0] }) + let content = '' + for (const method of [ + 'appendFile', + 'write', + 'datasync', + 'sync', + 'close' + ]) { + const original = handle[method].bind(handle) + handle[method] = async (...params) => { + if (method === 'appendFile') { + content = String(params[0]) + } + if (method === 'write') { + content = String( + params[0].subarray(params[1], params[1] + params[2]) + ) + } + const short = await fault(method, absolute, content) + if (short && method === 'write') { + params[2] = Math.max(1, Math.floor(params[2] / 2)) + } + const result = await original(...params) + let extra = {} + if (method === 'datasync' || method === 'sync') { + extra = snapshot(absolute) + } + else if (method === 'write') { + extra = { bytesWritten: result.bytesWritten } + } + record(method, absolute, extra) + await fault(method + ':after', absolute, content) + return result + } + } + return handle + } + fs.open = function (file, ...args) { + const callback = args.pop() + const absolute = path.resolve(String(file)) + const existed = raw.existsSync(absolute) + return raw.open(file, ...args, (error, fd) => { + if (!error && scoped(absolute)) { + handles.set(fd, absolute) + record('open', absolute, { existed, flags: args[0] }) + } + callback(error, fd) + }) + } + fs.write = function (fd, ...args) { + const callback = args.pop() + const file = handles.get(fd) + void fault('write', file) + .then(short => { + if (short) { + // The fixture targets write-file-atomic's Uint8Array + // overload. + if ( + !ArrayBuffer.isView(args[0]) || + typeof args[2] !== 'number' + ) { + throw new Error('Unsupported short-write fixture') + } + args[2] = Math.max(1, Math.floor(args[2] / 2)) + } + raw.write(fd, ...args, (error, bytes, buffer) => { + if (!error) { + record('write', file, { bytesWritten: bytes }) + } + callback(error, bytes, buffer) + }) + }) + .catch(callback) + } + fs.fsync = function (fd, callback) { + const file = handles.get(fd) + void fault('fsync', file) + .then(() => + raw.fsync(fd, error => { + if (!error && scoped(file)) { + record('sync', file, snapshot(file)) + } + callback(error) + }) + ) + .catch(callback) + } + fs.close = function (fd, callback) { + const file = handles.get(fd) + return raw.close(fd, error => { + if (!error) { + record('close', file) + handles.delete(fd) + } + callback(error) + }) + } + fs.rename = function (from, to, callback) { + const file = path.resolve(String(to)) + void fault('rename', file) + .then(() => + raw.rename(from, to, error => { + if (!error) { + record('rename', file, { + from: path.resolve(String(from)) + }) + } + callback(error) + }) + ) + .catch(callback) + } + fs.writeFileSync = function (file, ...args) { + const result = raw.writeFileSync(file, ...args) + record('writeFileSync', path.resolve(String(file))) + return result + } + syncBuiltinESMExports() +} diff --git a/test/storage.mjs b/test/storage.mjs new file mode 100644 index 0000000..8442784 --- /dev/null +++ b/test/storage.mjs @@ -0,0 +1,82 @@ +import { Buffer } from 'node:buffer' +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import { + writeAll, + publishFile, + publishFileSync, + appendRecord, + serialWriter +} from '../src/storage.mjs' + +test('complete-write loop preserves bytes through repeated short writes and rejects zero progress', async () => { + const bytes = Buffer.from('café 漢字 😀'), + written = [] + await writeAll( + { + async write(buffer, offset, length) { + const n = Math.min(3, length) + written.push(buffer.subarray(offset, offset + n)) + return { bytesWritten: n } + } + }, + bytes + ) + assert.deepEqual(Buffer.concat(written), bytes) + for (const count of [0, -1, NaN, 1000]) { + await assert.rejects( + writeAll( + { + async write() { + return { bytesWritten: count } + } + }, + bytes + ), + /progress write/ + ) + } +}) +test('serialized writer preserves order and stops after an uncertain operation', async () => { + const run = serialWriter(), + seen = [] + let release + const gate = new Promise(resolve => { + release = resolve + }) + const first = run(async () => { + seen.push('A') + await gate + throw new Error('uncertain') + }) + const second = run(async () => seen.push('B')) + release() + await assert.rejects(first, /uncertain/) + await assert.rejects(second, /uncertain/) + assert.deepEqual(seen, ['A']) +}) +test('durable publication replaces entire contents and concurrent append retains each record once', async t => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-storage-')) + t.after(() => fs.rm(dir, { recursive: true, force: true })) + const file = path.join(dir, 'data') + publishFileSync(file, 'old') + await publishFile(file, 'new café') + assert.equal(await fs.readFile(file, 'utf8'), 'new café') + await publishFile(file, '') + assert.equal((await fs.stat(file)).size, 0) + await Promise.all( + Array.from({ length: 40 }, (_, i) => appendRecord(file, String(i))) + ) + assert.equal( + await fs.readFile(file, 'utf8'), + Array.from({ length: 40 }, (_, i) => i + '\n').join('') + ) + await assert.rejects( + publishFile(path.join(dir, 'missing', 'x'), 'x'), + error => error.code === 'ENOENT' && error.storageFailure + ) + assert.deepEqual((await fs.readdir(dir)).sort(), ['data']) +})