diff --git a/.spiral/cycles/CYC-20260926-TTZ7C-67.md b/.spiral/cycles/CYC-20260926-TTZ7C-67.md new file mode 100644 index 0000000..98a2fd3 --- /dev/null +++ b/.spiral/cycles/CYC-20260926-TTZ7C-67.md @@ -0,0 +1,57 @@ +# Self-healing store ownership + +Goal: REQ-20260926-TTZ7C-66. A stop during startup releases ownership, and a +crashed owner that is proven gone no longer needs manual lock release. + +Status: Accepted on 2026-09-26. Human commitment in SRC-20260926-TTZ7C-65: the maintainer asked +for self-healing, reviewed the complexity estimate, and chose this cycle as the +SimplyStore part of a chain ending in curriculum-restapi-dev's +CYC-20260926-1ZG1H-6. + +Classification: deliberate deviation from ROADMAP.md; object history remains +the next target. It revises the accepted ownership rule of +DES-20260919-TTZ7C-24 through that design's named kernel-backed alternative, +by explicit maintainer decision. + +Branch: spiral/CYC-20260926-TTZ7C-67-self-healing-ownership, created from +authoritative master 039748e (SimplyStore 0.11.1). Branch identity checked when +opening the cycle. + +In scope: ownership liveness and takeover, startup signal handling, the +administrator unlock's handling of the new lock contents, tests (including the +existing crash-ownership test, whose expectation changes from the lock to the +unfinished command), recovery and durability documentation. + +Non-goals: automatic restart, automatic command recovery, multi-machine +stores, native lock dependencies, a release (the maintainer publishes), and +changes in curriculum-store or curriculum-restapi-dev. + +## Evaluation ready + +The goal can be judged. A server takes over the lock of a runtime owner whose +liveness socket refuses connections, audited and serialized, then applies the +unchanged startup assessment; all other locks still need administrator +release. Stops during startup release ownership. EVD-20260926-TTZ7C-68 records +the regressions, 215 passing tests, and a Docker run through the actual +compose file: after SIGKILL the next `up` heals and serves. + +Retained for later: Windows and socket-less filesystems keep manual release; +real-device power loss is not exercised; SimplyStore's release is the +maintainer's; curriculum-store's lockfile and curriculum-restapi-dev's pin and +end-to-end evaluation follow in CYC-20260926-1ZG1H-6. Existing 0.11.1 locks, +including the local dataset's, still need one administrator release. + +## Human acceptance and version + +On 2026-09-26 the maintainer instructed: "lets bump simplystore to 0.11.2 and +mark simplystore's cycle as accepted." + +This accepts the cycle at a635f793b12ecbcb58a4fb8bbc8e764df43cc74e: liveness-socket ownership, audited +and serialized takeover of provably dead runtime owners, startup signal +handling and administrator release of the new lock contents, with the evidence +and limits in EVD-20260926-TTZ7C-68. SimplyStore is bumped from 0.11.1 to +0.11.2, a patch release: formats and APIs are unchanged, and older locks stay +under administrator release. No tag or npm publication is made by this change. +Acceptance authorizes proposing integration; merging and publishing remain the +maintainer's. The chain continues in curriculum-store and +CYC-20260926-1ZG1H-6. Branch identity was verified before recording acceptance. diff --git a/.spiral/cycles/CYC-20260926-TTZ7C-67.ttl b/.spiral/cycles/CYC-20260926-TTZ7C-67.ttl new file mode 100644 index 0000000..a92de36 --- /dev/null +++ b/.spiral/cycles/CYC-20260926-TTZ7C-67.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:CYC-20260926-TTZ7C-67 a sd:Cycle ; + dcterms:identifier "CYC-20260926-TTZ7C-67" ; + sd:repositoryPath ".spiral/cycles/CYC-20260926-TTZ7C-67.md" ; + sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:REQ-20260926-TTZ7C-66 ; sd:gitCommit "26bd6babd3d2fd9af191a861cc166c271ee62c89" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260926-TTZ7C-67 ; sd:gitCommit "a635f793b12ecbcb58a4fb8bbc8e764df43cc74e" ] ; + sd:status sd:Accepted . diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.md b/.spiral/designs/DES-20260919-TTZ7C-24.md index 287f165..55ac61c 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.md +++ b/.spiral/designs/DES-20260919-TTZ7C-24.md @@ -62,19 +62,47 @@ directory, acquired in sorted path order. This deliberately treats stores sharin the default index/log directory as conflicting. Resolve existing parent paths; do not allow symlink aliases to create independent locks on the same directory. -Write owner diagnostics (random token, PID, host, start time) and sync the lock -entry before any store mutation. The token must match on release; clean shutdown -removes the lock and syncs its parent. Crash leftovers require an explicit offline -administrator release after confirming no writer remains. PID age, a timeout, or -a failed signal probe alone must never authorize automatic stealing. A crash -between mkdir and metadata publication is still a lock, not an empty slot. +Write owner diagnostics (random token, PID, host, start time, purpose) and sync +the lock entry before any store mutation. The token must match on release; clean +shutdown removes the lock and syncs its parent. PID age, a timeout, or a failed +signal probe alone must never authorize automatic stealing. A crash between +mkdir and metadata publication is still a lock, not an empty slot. + +Revised 2026-09-26 for REQ-20260926-TTZ7C-66: the owner also proves it is alive. +Before publishing its diagnostics, it listens on a Unix socket `owner.sock` in +the lock and records `liveness: "socket"`; release closes and removes it. The +kernel refuses connections to a socket whose process has ended, however it +ended. A server start (`purpose: "runtime"`) that finds such a runtime lock +connects to it: a refused connection proves the owner is gone. Any other +outcome, including success, a missing or partial lock, a lock without +`liveness`, another purpose, an error or a timeout, is no proof and startup +refuses as before. Paths too long for a socket or filesystems without sockets +publish no liveness and keep administrator release. + +Takeover serializes on an atomically created `.simplystore-lock.takeover` +directory beside the lock. Holding it, the starter re-reads the lock, requires +the same token and a refused connection again, writes a takeover audit record +with the previous owner's diagnostics into the store directory, moves the dead +lock aside, creates its own lock as usual, removes the old one and then the +takeover directory. A leftover takeover directory blocks takeover and is +removed by the administrator release. Ownership taken this way proves only +that no writer remains: the unchanged strict startup assessment still refuses +unfinished commands, and the new owner's failed start releases its ownership +as any failed start does. Other crash leftovers, including administrative +locks, still require an explicit offline administrator release. The proof +assumes one machine uses a store directory. This is cooperative exclusion, not protection against arbitrary file writers. Read-only inspection does not create source lock files. Backup and recovery apply require a stopped/quiescent source plus ownership; verify source fingerprints after acquiring it. Tests must demonstrate competing startup/apply exclusion and -crash-leftover handling. Kernel advisory locks are an alternative, but would need -a proven Node integration; no new native dependency is chosen in this step. +crash-leftover handling. Kernel advisory locks were the named alternative; a +listening socket provides the same kernel-held evidence of process death in +Node without a native dependency. + +A server handles stop signals from its start. A stop that arrives while the +store is opening lets opening finish (or fail), then closes and releases +ownership normally. Within the owner process use one acceptance FIFO and one serialized status writer. Worker messages do not release the command-runner slot until the parent finishes diff --git a/.spiral/designs/DES-20260919-TTZ7C-24.ttl b/.spiral/designs/DES-20260919-TTZ7C-24.ttl index efe6ab7..6020fef 100644 --- a/.spiral/designs/DES-20260919-TTZ7C-24.ttl +++ b/.spiral/designs/DES-20260919-TTZ7C-24.ttl @@ -24,4 +24,6 @@ project:DES-20260919-TTZ7C-24 a sd:Design ; sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ] ; sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260919-TTZ7C-18 ; - sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] . + sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] ; + sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260926-TTZ7C-66 ; sd:gitCommit "26bd6babd3d2fd9af191a861cc166c271ee62c89" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] . diff --git a/.spiral/evidence/EVD-20260926-TTZ7C-68.md b/.spiral/evidence/EVD-20260926-TTZ7C-68.md new file mode 100644 index 0000000..6092086 --- /dev/null +++ b/.spiral/evidence/EVD-20260926-TTZ7C-68.md @@ -0,0 +1,56 @@ +# Self-healing ownership evidence + +Verifies the implementation at d393787 (design 2cfb2ee) for +REQ-20260926-TTZ7C-66. Node 24.7.0 on Linux; Docker `node:24-alpine`. + +## Regressions + +`test/store-ownership.mjs` holds ownership in separate processes killed with +SIGKILL: a running owner is refused; a killed runtime owner is taken over with +an audit record naming the previous token and PID; recovery locks, a legacy +lock without liveness, a lock whose socket is missing, and non-runtime starts +are refused; a store path too long for a socket publishes no liveness and is +refused; six concurrent contenders for a dead lock leave exactly one owner +(20 of 20 repeated runs); a leftover takeover directory blocks takeover and +administrator release removes it together with a socket-bearing lock. + +`test/durability-crash.mjs`: an idle server killed with SIGKILL starts again +and serves queries; a server killed during a command now starts only as far as +"Administrative recovery required", with its active status intact and no lock +left; a SIGTERM while a slowed load is opening exits 0 without listening and +leaves no lock. + +At baseline 460ef5b, eight of these failed; the running-owner and long-path +refusals already held. + +`fs.cp` cannot copy socket files, which failed PL01-05's crash-image copy. The +helper now skips sockets, as `tar` and `rsync` do; the image and its +administrator unlock are otherwise unchanged. SimplyStore's backup, restore +and recovery copy individual files and never copy lock directories. + +The 15-file suite passes 215 tests; `npm test` passes 141. Every changed file +lints clean; `src/` has 52 pre-existing problems against 53 on master. + +## Docker, end to end + +The actual curriculum-restapi-dev compose file ran with an override naming a +disposable curriculum-store copy, whose installed SimplyStore `src/` was +replaced by this branch's: + +| Case | Result | +| --- | --- | +| SIGKILL (exit 137), then `up` | takeover logged, audit written, query HTTP 200 | +| Ctrl-C after healing | exit 0, lock released | +| Ctrl-C 0.5, 1, 1.5 s after the lock appears | exit 0, not listening, lock released | +| lock in the 0.11.1 format | `Store is locked`, exit 1 | + +## Size and limits + +Product code: +253/−30 lines (ownership +181, administrator release +33, +server +30/−16, runtime +8); tests +236; documentation in `docs/recovery.md` +and `DURABILITY.md`. This exceeds the pre-design estimate of about 150 lines, +mainly for the serialized takeover and administrator handling. + +Unproven: filesystems without Unix sockets other than the long-path case, +Windows, and power loss on a real device (the modeled power-loss suite passes; +its images carry no socket). Stores shared between machines are unsupported. diff --git a/.spiral/evidence/EVD-20260926-TTZ7C-68.ttl b/.spiral/evidence/EVD-20260926-TTZ7C-68.ttl new file mode 100644 index 0000000..42290a3 --- /dev/null +++ b/.spiral/evidence/EVD-20260926-TTZ7C-68.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260926-TTZ7C-68 a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260926-TTZ7C-68" ; + sd:repositoryPath ".spiral/evidence/EVD-20260926-TTZ7C-68.md" ; + sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "d39378771b5eb8720bfb606b8d69e089273df624" ] ; + sd:verifies [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "2cfb2ee6d202ce63828af985c26569a7c78b32c2" ] ; + sd:status sd:Active . diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.md b/.spiral/implementations/IMP-20260919-TTZ7C-26.md index 2873002..2332292 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.md +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.md @@ -211,3 +211,20 @@ The example exposes npm run init-integrity with a declarative store.json. Regression coverage and disposable-copy example verification preserve existing commands. Performance comparison uses the accepted optional baseline and identical 30-command runs; canonical hashes are reused and index entries share one barrier. + +## Self-healing ownership + +Implements the 2026-09-26 revision of DES-20260919-TTZ7C-24 for +REQ-20260926-TTZ7C-66. `acquireOwnership` listens on `owner.sock` in each lock +before publishing `liveness: "socket"` (skipped when the path exceeds 107 bytes +or listening fails) and closes and removes it on release. For +`purpose: "runtime"`, an existing lock whose owner is a runtime with socket +liveness and whose socket refuses connections is taken over under +`.simplystore-lock.takeover`: re-verify the same token, append the previous +owner to `.simplystore-takeovers.jsonl`, move the lock aside, create a new one, +then remove the old lock and the takeover directory. `StoreRuntime.open` takes +runtime ownership and logs takeovers. `run()` handles SIGTERM/SIGINT from its +start; a stop while opening closes after opening settles. The administrator +release accepts `owner.sock` and removes interrupted takeover directories, +retaining any previous owner record in its preview. The power-loss image helper +skips sockets, as `tar` and `rsync` do. diff --git a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl index 7827b87..27f1e34 100644 --- a/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl +++ b/.spiral/implementations/IMP-20260919-TTZ7C-26.ttl @@ -34,14 +34,14 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ; sd:implementationLocation "docs/recovery.md" ; sd:implementationLocation "DURABILITY.md" ; sd:implementationLocation "eslint.config.js" ; - sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; + sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "2cfb2ee6d202ce63828af985c26569a7c78b32c2" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-25 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-47 ; sd:gitCommit "83edb03bc406b9e651ef93bed69b87ece17dcf47" ] ; - sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:DEF-20260923-TTZ7C-50 ; sd:gitCommit "4bb10c2013fd2a7cb989b02586b992b936121362" ] ; - sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ; + sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "1e119dd101dd231a7cf24ba1b9d6c3e4999e8f60" ] ; sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-54 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ; - sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-53 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ; + sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:REQ-20260926-TTZ7C-66 ; sd:gitCommit "26bd6babd3d2fd9af191a861cc166c271ee62c89" ] ; + sd:implementationLocation "src/store-runtime.mjs", "test/store-ownership.mjs", "test/durability-crash.mjs", "test/power-loss-helpers.mjs" ; sd:implementationChangeKind sd:SemanticChange . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 7c5deca..094df9d 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,18 +18,21 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Latest accepted cycle: CYC-20260926-TTZ7C-63, object identity across the query -isolate, on spiral/CYC-20260926-TTZ7C-63-schema-identity, accepted on 2026-09-26 -at fa6c791. It corrects DEF-20260926-TTZ7C-62 (curriculum-store's shared schema -exhausted query limits) and, at the maintainer's request, identity for all query -data: stable private schema handles, schema ids in query workers, record-derived -output ids for shared objects without an `id`, and per-query tagged scalar -identity. EVD-20260926-TTZ7C-64 records 207 passing tests and about 4% added -cost on large scans. It was a deliberate deviation from the roadmap; object -history remains the next roadmap target. Integration is pending; no cycle is -active. - -The preceding cycle, CYC-20260924-TTZ7C-56 (isolated-vm query execution, SimplyStore +Latest accepted cycle: CYC-20260926-TTZ7C-67, self-healing store ownership, on +spiral/CYC-20260926-TTZ7C-67-self-healing-ownership, accepted on 2026-09-26 with +SimplyStore 0.11.2. A server takes over the lock of a runtime owner whose +liveness socket refuses connections (audited, serialized) and then applies the +unchanged startup assessment; other locks still need administrator release; +stops during startup release ownership. It revises DES-20260919-TTZ7C-24 by +maintainer decision and was a deliberate deviation from the roadmap; object +history remains the next roadmap target. EVD-20260926-TTZ7C-68 records 215 +passing tests and a Docker heal after SIGKILL. Integration is pending; no cycle +is active. + +CYC-20260926-TTZ7C-63, object identity across the query isolate, is accepted and +integrated (pull request #28, SimplyStore 0.11.1). + +Earlier, CYC-20260924-TTZ7C-56 (isolated-vm query execution, SimplyStore 0.11.0), is accepted and integrated at bd61d9c (pull request #27). Queries must have no import, Node filesystem/network/database authority. Preserve diff --git a/.spiral/requests/REQ-20260926-TTZ7C-66.md b/.spiral/requests/REQ-20260926-TTZ7C-66.md new file mode 100644 index 0000000..a16950f --- /dev/null +++ b/.spiral/requests/REQ-20260926-TTZ7C-66.md @@ -0,0 +1,25 @@ +# Self-healing store ownership + +Derived from SRC-20260926-TTZ7C-65. + +1. A server that stops during startup releases its ownership. A stop request + before the store is open lets opening finish, then closes normally. +2. When a previous server's ownership remains after a crash, SIGKILL or power + loss, the next server start takes it over only when the previous owner is + proven gone, records the takeover, and then applies the unchanged strict + startup assessment. Unfinished commands still require administrator + recovery; nothing is rerun automatically. +3. When proof is unavailable (the owner may be alive, the lock predates this + mechanism, the lock belongs to an administrative operation, the filesystem + or path cannot hold the proof, or its state is incomplete), startup refuses + exactly as before and the documented administrator release still works. +4. Two concurrent starts never both own the store. + +Assumption made explicit: one machine uses a store directory at a time. The +proof is local to the kernel that runs the owner. + +Acceptance: a server killed with SIGKILL while idle starts again without manual +release; a server killed during a command starts no further than administrator +recovery required; a running owner, a legacy lock and a recovery lock are never +taken over; concurrent takeovers leave exactly one owner; a stop during startup +leaves no lock; administrator unlock handles the new lock contents. diff --git a/.spiral/requests/REQ-20260926-TTZ7C-66.ttl b/.spiral/requests/REQ-20260926-TTZ7C-66.ttl new file mode 100644 index 0000000..cd7f8d9 --- /dev/null +++ b/.spiral/requests/REQ-20260926-TTZ7C-66.ttl @@ -0,0 +1,9 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260926-TTZ7C-66 a sd:Request ; + dcterms:identifier "REQ-20260926-TTZ7C-66" ; + sd:repositoryPath ".spiral/requests/REQ-20260926-TTZ7C-66.md" ; + sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:SRC-20260926-TTZ7C-65 ; sd:gitCommit "70a2b2a58fde152b9d5b8dae0b7e4aaa270ddd54" ] ; + sd:status sd:Active . diff --git a/.spiral/sources/SRC-20260926-TTZ7C-65.md b/.spiral/sources/SRC-20260926-TTZ7C-65.md new file mode 100644 index 0000000..261768b --- /dev/null +++ b/.spiral/sources/SRC-20260926-TTZ7C-65.md @@ -0,0 +1,42 @@ +# Store lock left after Docker shutdown; self-healing request + +Retained source: maintainer conversation on 2026-09-26. + +Testing SimplyStore 0.11.1 in curriculum-restapi-dev's curriculum-store, the +store lock remained after shutdown. The maintainer: "What I'd like is for +shutting down docker compose to clear the lock. Currently that appears not to +happen." + +Investigation on a disposable copy found two SimplyStore gaps: a stop during +startup always leaves the lock, because `run()` installs its SIGTERM/SIGINT +handlers only after `StoreRuntime.open` has taken ownership; and a running slow +query delays close to about 10 s, Docker's default grace period. After startup, +SimplyStore released the lock within about 1 s. + +The maintainer then asked: "What I'd like is for the system to self-heal. So if +the it crashed, it should restart automatically and check if the lock can be +safely released." The assistant reported that DES-20260919-TTZ7C-24 forbids +automatic stealing on PID age, timeouts or signal probes, and that it names +kernel-backed exclusion as the alternative. A cross-container probe showed a +Unix socket in a bind-mounted directory accepts connections while its owner +runs and refuses them (ECONNREFUSED) after the owner is killed with SIGKILL. +The assistant proposed a liveness socket in each lock, automatic takeover only +when the socket refuses, fallback to administrator release otherwise, and +retained administrator assessment for unfinished commands; with the stated +assumption that one machine uses a store directory. Asked "How much complexity +does this add?", the assistant estimated about 150 lines of code and 250 of +tests, and named takeover races, recovery locks, unsupported filesystems and +the single-machine rule as the conceptual cost. + +The maintainer asked to investigate the actual incident first ("I pressed +ctrl-c once in the docker compose process, which should have started normal +shutdown"). Docker's event log showed every service was killed after 10 s +because `sh` stayed PID 1; that was corrected in curriculum-restapi-dev +(CYC-20260926-1ZG1H-6). The maintainer then kept that cycle open: "I think the +self healing part should be included in it", chose a chain of cycles with the +mechanism in SimplyStore, and declined a restart policy because the Docker +environment is only for development. + +This settles the commitment: SimplyStore proves a crashed owner is gone and +takes over its lock on the next start; otherwise it refuses as before. Starting +again remains the operator's action. diff --git a/.spiral/sources/SRC-20260926-TTZ7C-65.ttl b/.spiral/sources/SRC-20260926-TTZ7C-65.ttl new file mode 100644 index 0000000..e92d013 --- /dev/null +++ b/.spiral/sources/SRC-20260926-TTZ7C-65.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:SRC-20260926-TTZ7C-65 a sd:Source ; + dcterms:identifier "SRC-20260926-TTZ7C-65" ; + sd:repositoryPath ".spiral/sources/SRC-20260926-TTZ7C-65.md" ; + sd:sourceAvailability sd:Retained ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . diff --git a/DURABILITY.md b/DURABILITY.md index 9550488..607f5ce 100644 --- a/DURABILITY.md +++ b/DURABILITY.md @@ -43,7 +43,11 @@ suppression remains a separate follow-up contract. Ownership locks exclude cooperating server/admin writers across configured paths. They are not protection against arbitrary filesystem writers or older -binaries. Stale locks require explicit offline assessment, never PID-age stealing. +binaries. A server takes over a stale server lock only when that owner's +liveness socket refuses connections, proving its process has ended on this +machine; startup assessment still applies, and all other stale locks require +explicit offline assessment, never PID-age stealing. Stores must not be shared +between machines. Custom hooks must await their work; declare additional required artifacts with `requiredFiles`. Default indexes are reconstructible; custom dependencies need their own conformance checks. Unsupported directory-sync operations fail rather diff --git a/docs/recovery.md b/docs/recovery.md index 12af6ed..960bfa8 100644 --- a/docs/recovery.md +++ b/docs/recovery.md @@ -99,11 +99,33 @@ backup or retain the diagnosis; do not claim a safe rerun. ## Deal with ownership after a crash Each mutable directory has a `.simplystore-lock` during operation. Normal shutdown -drains accepted work before releasing ownership. A crash or storage fault leaves -locks. Neither PID age nor an apparently dead process authorizes automatic stealing. -A hung shutdown can be terminated, but then requires this offline procedure. - -After confirming that no writer remains, inspect and explicitly release locks: +drains accepted work before releasing ownership, including a stop that arrives +while the server is still opening the store. A crash, `SIGKILL`, power loss or +storage fault leaves locks. + +A server lock contains `owner.sock`, a Unix socket its process listens on. The +kernel refuses connections to it once that process has ended, however it ended. +When a server starts and finds a server lock whose socket refuses connections, +it takes the lock over, logs `Store ownership took over …`, and appends the +previous owner's diagnostics to `.simplystore-takeovers.jsonl` in the store +directory. The normal startup checks then still apply: a crash during a command +stops startup with `Administrative recovery required`, and nothing is rerun. + +Anything else is not proof, and startup refuses with `Store is locked`: a +running owner, a lock written by an older SimplyStore, a lock from an +administrative tool, a missing or incomplete lock or socket, a store path too +long for a socket (107 bytes including `/.simplystore-lock/owner.sock`), or a +filesystem without sockets. A leftover `.simplystore-lock.takeover` directory +means a takeover was interrupted and also blocks startup. Copies of a store made +with tools that skip sockets, such as `tar` or `rsync` without `--specials`, +carry locks that need administrator release. + +The proof is local to one machine. Do not open a store directory from more +than one machine, for example over a network filesystem. PID age, timeouts and +signal probes never authorize a takeover. + +After confirming that no writer remains, inspect and explicitly release locks +and any interrupted takeover: ```sh node scripts/recover.mjs unlock --store store.json --confirmed-stopped \ diff --git a/package.json b/package.json index c4cac9e..716d3d4 100644 --- a/package.json +++ b/package.json @@ -1,11 +1,11 @@ { "name": "@muze-nl/simplystore", - "version": "0.11.1", + "version": "0.11.2", "main": "src/server.mjs", "type": "module", "scripts": { "start": "node --no-node-snapshot src/run.mjs", - "test": "node --no-node-snapshot --test test/query-isolate.mjs test/required-integrity.mjs test/file-data.mjs test/index-ids.mjs test/store-runtime.mjs test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs", + "test": "node --no-node-snapshot --test test/query-isolate.mjs test/store-ownership.mjs test/required-integrity.mjs test/file-data.mjs test/index-ids.mjs test/store-runtime.mjs test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs", "test:acid": "node --no-node-snapshot --test test/acid-baseline.mjs", "test:durability": "node --no-node-snapshot --test test/durability-crash.mjs", "test:handlers": "node --no-node-snapshot --test test/after-change-handlers.mjs", diff --git a/src/admin-recovery.mjs b/src/admin-recovery.mjs index b5146ae..0a1e1ae 100644 --- a/src/admin-recovery.mjs +++ b/src/admin-recovery.mjs @@ -851,6 +851,7 @@ class OfflineLockRelease { this.config = null this.report = null this.removed = [] + this.takeovers = [] } async preview() { @@ -860,6 +861,7 @@ class OfflineLockRelease { const directories = await this.findLockDirectories() for (const directory of directories) { await this.inspectLock(directory) + await this.inspectTakeover(directory) } return this.createPreview() } @@ -912,7 +914,8 @@ class OfflineLockRelease { return } const unknownContents = entries.some( - entry => entry !== 'owner.json' && !entry.endsWith('.tmp') + entry => entry !== 'owner.json' && entry !== 'owner.sock' && + !entry.endsWith('.tmp') ) if (unknownContents) { throw new Error(`Unknown lock contents: ${lock}`) @@ -937,6 +940,11 @@ class OfflineLockRelease { async readOwnerFiles(lock, entries) { const ownerFiles = [] for (const entry of entries) { + // A liveness socket has no contents to preserve. + if (entry === 'owner.sock') { + ownerFiles.push({ name: entry, socket: true }) + continue + } const bytes = await fs.readFile(path.join(lock, entry)) ownerFiles.push({ name: entry, @@ -996,6 +1004,7 @@ class OfflineLockRelease { operator: this.operator, reason: this.reason, removed: this.removed, + takeovers: this.takeovers, finish: () => this.finish() } } @@ -1009,10 +1018,33 @@ class OfflineLockRelease { await fs.rmdir(lock) await syncDirectory(path.dirname(lock)) } + for (const { takeover } of this.takeovers) { + await fs.rm(takeover, { recursive: true }) + await syncDirectory(path.dirname(takeover)) + } + } + + // An interrupted automatic takeover leaves its directory, possibly with + // the previous owner's lock inside. Preserve what it holds, then remove it. + async inspectTakeover(directory) { + const takeover = path.join(directory, '.simplystore-lock.takeover') + const previous = path.join(takeover, 'previous') + if (await this.readLockEntries(takeover) === null) { + return + } + const entries = await this.readLockEntries(previous) + let ownerFiles = [] + if (entries !== null) { + ownerFiles = await this.readOwnerFiles(previous, entries) + } + this.takeovers.push({ takeover, ownerFiles }) } async verifyLockUnchanged(lock, ownerFiles) { for (const entry of ownerFiles) { + if (entry.socket) { + continue + } const current = await fs.readFile(path.join(lock, entry.name)) if (current.toString('base64') !== entry.bytes) { throw new Error('Lock changed after preview') diff --git a/src/server.mjs b/src/server.mjs index 5b19a17..8ae03a7 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -23,7 +23,31 @@ async function run(server, options = {}) { } const port = options.port || 3000 const wwwroot = options.wwwroot || rootDirectory + '/www' - const runtime = await StoreRuntime.open(options, { + let runtime = null + let listener = null + let stopRequested = false + + // Handle stop signals from the start: opening takes ownership, so a stop + // while opening lets it finish (or fail) and then closes normally. + async function shutdown() { + stopRequested = true + if (!runtime) { + return + } + listener?.close() + try { + await runtime.close() + process.exit(runtime.storageFailed ? 1 : 0) + } + catch (error) { + runtime.failStorage(error) + } + } + + process.once('SIGTERM', shutdown) + process.once('SIGINT', shutdown) + + runtime = await StoreRuntime.open(options, { onStorageFailure(error) { console.error( 'Storage outcome uncertain; stopping mutation. ' + @@ -33,28 +57,18 @@ async function run(server, options = {}) { setImmediate(() => process.exit(1)) } }) + if (stopRequested) { + await shutdown() + return + } configureRoutes(server, runtime, wwwroot) - const listener = server.listen(port, () => { + listener = server.listen(port, () => { console.log('SimplyStore listening on port ' + port) }) listener.on('error', error => { runtime.failStorage(error) }) - - async function shutdown() { - listener.close() - try { - await runtime.close() - process.exit(runtime.storageFailed ? 1 : 0) - } - catch (error) { - runtime.failStorage(error) - } - } - - process.once('SIGTERM', shutdown) - process.once('SIGINT', shutdown) } function configureRoutes(server, runtime, wwwroot) { diff --git a/src/store-ownership.mjs b/src/store-ownership.mjs index 0ee6c5e..bc00f1f 100644 --- a/src/store-ownership.mjs +++ b/src/store-ownership.mjs @@ -1,10 +1,152 @@ +import { Buffer } from 'node:buffer' import fs from 'node:fs/promises' +import net from 'node:net' import path from 'node:path' import os from 'node:os' import process from 'node:process' import { randomUUID } from 'node:crypto' import { publishFile, syncDirectory, syncAncestors } from './storage.mjs' +const lockName = '.simplystore-lock' +const takeoverName = '.simplystore-lock.takeover' +const auditName = '.simplystore-takeovers.jsonl' +// A Unix socket path holds at most 107 bytes plus a terminating zero. +const maxSocketPath = 107 +const probeTimeout = 1000 + +// The owner listens on a socket in its lock. The kernel refuses connections +// once the owning process has ended, however it ended: that is the proof a +// restarting server needs. Without a socket, only an administrator may +// release the lock. +async function listenForLiveness(lock) { + const socket = path.join(lock, 'owner.sock') + if (Buffer.byteLength(socket) > maxSocketPath) { + return null + } + const server = net.createServer(connection => { + connection.end() + }) + try { + await new Promise((resolve, reject) => { + server.once('error', reject) + server.listen(socket, resolve) + }) + } + catch { + return null + } + server.unref() + return server +} + +function probeLiveness(lock) { + return new Promise(resolve => { + const connection = net.connect(path.join(lock, 'owner.sock')) + const timer = setTimeout(() => { + connection.destroy() + resolve('timeout') + }, probeTimeout) + connection.once('connect', () => { + clearTimeout(timer) + connection.destroy() + resolve('alive') + }) + connection.once('error', error => { + clearTimeout(timer) + resolve(error.code || 'error') + }) + }) +} + +async function readOwner(lock) { + try { + return JSON.parse( + await fs.readFile(path.join(lock, 'owner.json'), 'utf8') + ) + } + catch { + return null + } +} + +// Only a runtime owner with a liveness socket that refuses connections is +// proven gone. Everything else, including errors and timeouts, is no proof. +async function provenDeadOwner(lock) { + const owner = await readOwner(lock) + if (owner?.purpose !== 'runtime' || owner.liveness !== 'socket') { + return null + } + if (await probeLiveness(lock) !== 'ECONNREFUSED') { + return null + } + return owner +} + +async function createLock(lock) { + try { + await fs.mkdir(lock) + return true + } + catch (error) { + if (error.code === 'EEXIST') { + return false + } + throw error + } +} + +async function appendTakeoverAudit(directory, record) { + const file = await fs.open(path.join(directory, auditName), 'a') + try { + await file.write(JSON.stringify(record) + '\n') + await file.sync() + } + finally { + await file.close() + } +} + +// Takeovers serialize on a takeover directory. Holding it, re-check that the +// same owner is still proven gone, record the takeover, move its lock aside +// and create ours. A leftover takeover directory blocks until an +// administrator releases it. +async function takeOverDeadLock(directory, token) { + const lock = path.join(directory, lockName) + const dead = await provenDeadOwner(lock) + if (!dead) { + return null + } + const takeover = path.join(directory, takeoverName) + if (!await createLock(takeover)) { + throw new Error( + `Store lock takeover is in progress or was interrupted: ${takeover}. Offline administrator release is required if no server is starting.` + ) + } + const again = await provenDeadOwner(lock) + if (again?.token !== dead.token) { + await fs.rmdir(takeover) + return null + } + await appendTakeoverAudit(directory, { + time: new Date().toISOString(), + token, + pid: process.pid, + host: os.hostname(), + proof: 'ECONNREFUSED', + previous: again + }) + const previous = path.join(takeover, 'previous') + await fs.rename(lock, previous) + const created = await createLock(lock) + await fs.rm(previous, { recursive: true }) + await fs.rmdir(takeover) + await syncDirectory(directory) + if (!created) { + return null + } + return again +} + export async function acquireOwnership( directories, { ancestorToken, purpose, auditDir } = {} @@ -52,28 +194,39 @@ export async function acquireOwnership( } } const token = randomUUID(), - held = [] + held = [], + servers = new Map(), + takeovers = [] try { for (const directory of canonical) { - const lock = path.join(directory, '.simplystore-lock') - try { - await fs.mkdir(lock) - } - catch (error) { - if (error.code === 'EEXIST') { - throw new Error( - `Store is locked: ${lock}. Offline administrator inspection is required; do not automatically steal a lock.` - ) + const lock = path.join(directory, lockName) + let created = await createLock(lock) + if (!created && purpose === 'runtime') { + const previous = await takeOverDeadLock(directory, token) + if (previous) { + takeovers.push({ lock, previous }) + created = true } - throw error + } + if (!created) { + throw new Error( + `Store is locked: ${lock}. The owner may still run; offline administrator inspection is required.` + ) } held.push(lock) + const server = await listenForLiveness(lock) + let liveness + if (server) { + servers.set(lock, server) + liveness = 'socket' + } await publishFile( path.join(lock, 'owner.json'), JSON.stringify({ token, purpose, auditDir, + liveness, pid: process.pid, host: os.hostname(), started: new Date().toISOString() @@ -86,6 +239,7 @@ export async function acquireOwnership( // No store writes have begun, so only our own acquired locks can be // released. for (const lock of held.reverse()) { + await closeLiveness(servers, lock) await fs.rm(lock, { recursive: true }) await syncDirectory(path.dirname(lock)) } @@ -94,6 +248,7 @@ export async function acquireOwnership( return { token, directories: canonical, + takeovers, async release() { for (const lock of [...held].reverse()) { const owner = JSON.parse( @@ -102,6 +257,7 @@ export async function acquireOwnership( if (owner.token !== token) { throw new Error(`Ownership changed: ${lock}`) } + await closeLiveness(servers, lock) await fs.unlink(path.join(lock, 'owner.json')) await fs.rmdir(lock) await syncDirectory(path.dirname(lock)) @@ -110,3 +266,17 @@ export async function acquireOwnership( } } } + +async function closeLiveness(servers, lock) { + const server = servers.get(lock) + if (!server) { + return + } + servers.delete(lock) + await new Promise(resolve => { + server.close(() => { + resolve() + }) + }) + await fs.rm(path.join(lock, 'owner.sock'), { force: true }) +} diff --git a/src/store-runtime.mjs b/src/store-runtime.mjs index 6ad4b23..a55c532 100644 --- a/src/store-runtime.mjs +++ b/src/store-runtime.mjs @@ -192,8 +192,15 @@ export class StoreRuntime { async open() { const { store } = this.configuration this.ownership = await this.mechanisms.acquireOwnership( - mutableDirectories(store) + mutableDirectories(store), { purpose: 'runtime' } ) + for (const { lock, previous } of this.ownership.takeovers ?? []) { + console.warn( + `Store ownership took over ${lock} from stopped owner ` + + `${previous.pid}@${previous.host} (started ` + + `${previous.started}); the owner's socket refused connections` + ) + } try { const inspection = await this.mechanisms.inspectStore(store) this.assertStoreReady(inspection) diff --git a/test/durability-crash.mjs b/test/durability-crash.mjs index ef1152c..81ffbd0 100644 --- a/test/durability-crash.mjs +++ b/test/durability-crash.mjs @@ -189,7 +189,7 @@ test('crash after done status but before query update recovers committed state w assert.equal((await getCommandStatus(port, command.id)).status, 'done') }) -test('crash ownership cannot be stolen automatically', async t => { +test('crash during a command still requires administrator recovery', async t => { const fixture = await makeServerFixture(t), port = await getOpenPort() const first = startServer(t, fixture, { @@ -204,9 +204,12 @@ test('crash ownership cannot be stolen automatically', async t => { value: { name: 'A' } }) await waitForExit(first.child) + // The dead owner's lock is taken over, but the unfinished command is not + // resolved automatically. const second = startServer(t, fixture, { port }) assert.equal((await waitForExit(second.child)).code, 1) - assert.match(second.getOutput(), /Store is locked/) + assert.match(second.getOutput(), /Administrative recovery required/) + await assert.rejects(fs.stat(path.join(fixture.dir, '.simplystore-lock'))) assert.equal( (await readCommandStatusRecords(fixture)).filter( s => s.status === 'active' @@ -215,6 +218,57 @@ test('crash ownership cannot be stolen automatically', async t => { ) }) +test('an idle server killed with SIGKILL starts again without release', async t => { + const fixture = await makeServerFixture(t), + port = await getOpenPort() + const first = startServer(t, fixture, { port }) + await waitForServer(first.child, first.getOutput, port) + first.child.kill('SIGKILL') + await waitForExit(first.child) + const second = startServer(t, fixture, { port }) + await waitForServer(second.child, second.getOutput, port) + assert.match(second.getOutput(), /took over/) + assert.deepEqual(await queryPersons(port), []) + await stopServer(second.child) + await assert.rejects(fs.stat(path.join(fixture.dir, '.simplystore-lock'))) +}) + +test('a stop during startup releases ownership', async t => { + const fixture = await makeServerFixture(t) + const loadWorker = path.join(fixture.dir, 'slow-load-worker.mjs') + const fileData = new URL('../src/file-data.mjs', import.meta.url) + await fs.writeFile( + loadWorker, + `import { parentPort } from 'node:worker_threads' +import { loadFileData } from ${JSON.stringify(fileData.href)} + +parentPort.on('message', async files => { + await new Promise(resolve => setTimeout(resolve, 1000)) + parentPort.postMessage(await loadFileData(files)) +}) +` + ) + const lock = path.join(fixture.dir, '.simplystore-lock') + const running = startServer(t, fixture, { + port: await getOpenPort(), + loadWorker + }) + for (let attempt = 0; attempt < 100; attempt++) { + try { + await fs.stat(path.join(lock, 'owner.json')) + break + } + catch { + await new Promise(resolve => setTimeout(resolve, 20)) + } + } + running.child.kill('SIGTERM') + const exit = await waitForExit(running.child) + assert.equal(exit.code, 0, running.getOutput()) + assert.doesNotMatch(running.getOutput(), /SimplyStore listening/) + await assert.rejects(fs.stat(lock)) +}) + test('hanging command times out unsafe and later accepted command commits', async t => { const fixture = await makeServerFixture(t, { commandsSource: `export default { diff --git a/test/power-loss-helpers.mjs b/test/power-loss-helpers.mjs index 7b2d390..7860db6 100644 --- a/test/power-loss-helpers.mjs +++ b/test/power-loss-helpers.mjs @@ -154,7 +154,12 @@ export async function copyImage(t, store) { path.join(os.tmpdir(), 'simplystore-crash-image-') ) t.after(() => fs.rm(dir, { recursive: true, force: true })) - await fs.cp(store.dir, dir, { recursive: true }) + // Like tar or rsync, skip an owner's liveness socket: it holds no data, + // and a copied lock without it still needs administrator release. + await fs.cp(store.dir, dir, { + recursive: true, + filter: async source => !(await fs.lstat(source)).isSocket() + }) const image = { ...store, dir } for (const key of [ 'datafile', diff --git a/test/store-ownership.mjs b/test/store-ownership.mjs new file mode 100644 index 0000000..c914df6 --- /dev/null +++ b/test/store-ownership.mjs @@ -0,0 +1,180 @@ +import assert from 'node:assert/strict' +import { spawn } from 'node:child_process' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import process from 'node:process' +import test from 'node:test' +import { fileURLToPath } from 'node:url' +import { acquireOwnership } from '../src/store-ownership.mjs' +import { releaseOfflineLocks } from '../src/admin-recovery.mjs' +import { makeServerFixture } from './durability-helpers.mjs' + +const ownershipModule = fileURLToPath( + new URL('../src/store-ownership.mjs', import.meta.url) +) + +// Hold ownership in a separate process, so it can be killed like a server. +function holdOwnership(t, directory, options = { purpose: 'runtime' }) { + const script = ` + import { acquireOwnership } from ${JSON.stringify(ownershipModule)} + try { + await acquireOwnership( + [${JSON.stringify(directory)}], ${JSON.stringify(options)} + ) + process.stdout.write('owned\\n') + setInterval(() => {}, 1000) + } + catch (error) { + process.stdout.write('refused: ' + error.message + '\\n') + process.exit(1) + } + ` + const child = spawn(process.execPath, ['--input-type=module', '-e', script]) + t.after(() => child.kill('SIGKILL')) + let output = '' + const settled = new Promise(resolve => { + child.stdout.on('data', data => { + output += data + if (/owned|refused/.test(output)) { + resolve(output.trim()) + } + }) + child.on('exit', () => resolve(output.trim())) + }) + return { child, settled } +} + +async function killed(owner) { + const exit = new Promise(resolve => owner.child.once('exit', resolve)) + owner.child.kill('SIGKILL') + await exit +} + +async function deadRuntimeLock(t, directory) { + const owner = holdOwnership(t, directory) + assert.equal(await owner.settled, 'owned') + await killed(owner) +} + +async function lockEntries(directory) { + const entries = await fs.readdir(directory) + return entries.filter(entry => entry.startsWith('.simplystore-lock')) +} + +async function readOwner(directory) { + return JSON.parse(await fs.readFile( + path.join(directory, '.simplystore-lock', 'owner.json'), 'utf8' + )) +} + +test('a running owner is never taken over', async t => { + const { dir } = await makeServerFixture(t) + const owner = holdOwnership(t, dir) + assert.equal(await owner.settled, 'owned') + await assert.rejects( + acquireOwnership([dir], { purpose: 'runtime' }), /Store is locked/ + ) + assert.equal((await readOwner(dir)).pid, owner.child.pid) +}) + +test('a killed runtime owner is taken over with an audit record', async t => { + const { dir } = await makeServerFixture(t) + const owner = holdOwnership(t, dir) + assert.equal(await owner.settled, 'owned') + const dead = await readOwner(dir) + assert.equal(dead.liveness, 'socket') + await killed(owner) + + const ownership = await acquireOwnership([dir], { purpose: 'runtime' }) + assert.equal((await readOwner(dir)).token, ownership.token) + const audit = (await fs.readFile( + path.join(dir, '.simplystore-takeovers.jsonl'), 'utf8' + )).trim().split('\n').map(line => JSON.parse(line)) + assert.equal(audit.length, 1) + assert.equal(audit[0].previous.token, dead.token) + assert.equal(audit[0].previous.pid, dead.pid) + assert.equal(audit[0].token, ownership.token) + assert.equal(audit[0].proof, 'ECONNREFUSED') + await ownership.release() + assert.deepEqual(await lockEntries(dir), []) +}) + +test('locks without proof of a dead runtime owner are kept', async t => { + const { dir } = await makeServerFixture(t) + const lock = path.join(dir, '.simplystore-lock') + + const recovery = holdOwnership(t, dir, { purpose: 'recovery' }) + assert.equal(await recovery.settled, 'owned') + await killed(recovery) + await assert.rejects( + acquireOwnership([dir], { purpose: 'runtime' }), /Store is locked/ + ) + + await fs.rm(lock, { recursive: true }) + await fs.mkdir(lock) + await fs.writeFile(path.join(lock, 'owner.json'), JSON.stringify({ + token: 'legacy', pid: 1, host: os.hostname(), purpose: 'runtime' + })) + await assert.rejects( + acquireOwnership([dir], { purpose: 'runtime' }), /Store is locked/ + ) + + await fs.rm(lock, { recursive: true }) + await deadRuntimeLock(t, dir) + await fs.unlink(path.join(lock, 'owner.sock')) + await assert.rejects( + acquireOwnership([dir], { purpose: 'runtime' }), /Store is locked/ + ) + + await fs.rm(lock, { recursive: true }) + await deadRuntimeLock(t, dir) + await assert.rejects( + acquireOwnership([dir], { purpose: 'recovery' }), /Store is locked/ + ) +}) + +test('paths too long for a socket publish no liveness', async t => { + const base = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-long-')) + t.after(() => fs.rm(base, { recursive: true, force: true })) + const dir = path.join(base, 'x'.repeat(100)) + await fs.mkdir(dir) + const owner = holdOwnership(t, dir) + assert.equal(await owner.settled, 'owned') + assert.equal((await readOwner(dir)).liveness, undefined) + await killed(owner) + await assert.rejects( + acquireOwnership([dir], { purpose: 'runtime' }), /Store is locked/ + ) +}) + +test('concurrent takeovers leave exactly one owner', async t => { + const { dir } = await makeServerFixture(t) + await deadRuntimeLock(t, dir) + const contenders = Array.from({ length: 6 }, () => holdOwnership(t, dir)) + const results = await Promise.all(contenders.map(owner => owner.settled)) + assert.equal(results.filter(result => result === 'owned').length, 1, + results.join('\n')) + const winner = contenders[results.indexOf('owned')] + assert.equal((await readOwner(dir)).pid, winner.child.pid) + assert.deepEqual(await lockEntries(dir), ['.simplystore-lock']) +}) + +test('an interrupted takeover blocks until administrator release', async t => { + const fixture = await makeServerFixture(t) + const { dir } = fixture + await deadRuntimeLock(t, dir) + await fs.mkdir(path.join(dir, '.simplystore-lock.takeover')) + await assert.rejects( + acquireOwnership([dir], { purpose: 'runtime' }), /takeover/ + ) + const release = await releaseOfflineLocks(fixture, { + operator: 'test administrator', + reason: 'Owner killed during takeover test', + confirmedStopped: true + }) + await release.finish() + assert.deepEqual(await lockEntries(dir), []) + const ownership = await acquireOwnership([dir], { purpose: 'runtime' }) + await ownership.release() +})