Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions .spiral/cycles/CYC-20260926-TTZ7C-67.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Self-healing store ownership

Goal: REQ-20260926-TTZ7C-66. A stop during startup releases ownership, and a
crashed owner that is proven gone no longer needs manual lock release.

Status: Accepted on 2026-09-26. Human commitment in SRC-20260926-TTZ7C-65: the maintainer asked
for self-healing, reviewed the complexity estimate, and chose this cycle as the
SimplyStore part of a chain ending in curriculum-restapi-dev's
CYC-20260926-1ZG1H-6.

Classification: deliberate deviation from ROADMAP.md; object history remains
the next target. It revises the accepted ownership rule of
DES-20260919-TTZ7C-24 through that design's named kernel-backed alternative,
by explicit maintainer decision.

Branch: spiral/CYC-20260926-TTZ7C-67-self-healing-ownership, created from
authoritative master 039748e (SimplyStore 0.11.1). Branch identity checked when
opening the cycle.

In scope: ownership liveness and takeover, startup signal handling, the
administrator unlock's handling of the new lock contents, tests (including the
existing crash-ownership test, whose expectation changes from the lock to the
unfinished command), recovery and durability documentation.

Non-goals: automatic restart, automatic command recovery, multi-machine
stores, native lock dependencies, a release (the maintainer publishes), and
changes in curriculum-store or curriculum-restapi-dev.

## Evaluation ready

The goal can be judged. A server takes over the lock of a runtime owner whose
liveness socket refuses connections, audited and serialized, then applies the
unchanged startup assessment; all other locks still need administrator
release. Stops during startup release ownership. EVD-20260926-TTZ7C-68 records
the regressions, 215 passing tests, and a Docker run through the actual
compose file: after SIGKILL the next `up` heals and serves.

Retained for later: Windows and socket-less filesystems keep manual release;
real-device power loss is not exercised; SimplyStore's release is the
maintainer's; curriculum-store's lockfile and curriculum-restapi-dev's pin and
end-to-end evaluation follow in CYC-20260926-1ZG1H-6. Existing 0.11.1 locks,
including the local dataset's, still need one administrator release.

## Human acceptance and version

On 2026-09-26 the maintainer instructed: "lets bump simplystore to 0.11.2 and
mark simplystore's cycle as accepted."

This accepts the cycle at a635f793b12ecbcb58a4fb8bbc8e764df43cc74e: liveness-socket ownership, audited
and serialized takeover of provably dead runtime owners, startup signal
handling and administrator release of the new lock contents, with the evidence
and limits in EVD-20260926-TTZ7C-68. SimplyStore is bumped from 0.11.1 to
0.11.2, a patch release: formats and APIs are unchanged, and older locks stay
under administrator release. No tag or npm publication is made by this change.
Acceptance authorizes proposing integration; merging and publishing remain the
maintainer's. The chain continues in curriculum-store and
CYC-20260926-1ZG1H-6. Branch identity was verified before recording acceptance.
10 changes: 10 additions & 0 deletions .spiral/cycles/CYC-20260926-TTZ7C-67.ttl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
@prefix sd: <https://muze.nl/ns/spiral-developer#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .

project:CYC-20260926-TTZ7C-67 a sd:Cycle ;
dcterms:identifier "CYC-20260926-TTZ7C-67" ;
sd:repositoryPath ".spiral/cycles/CYC-20260926-TTZ7C-67.md" ;
sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:REQ-20260926-TTZ7C-66 ; sd:gitCommit "26bd6babd3d2fd9af191a861cc166c271ee62c89" ] ;
sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260926-TTZ7C-67 ; sd:gitCommit "a635f793b12ecbcb58a4fb8bbc8e764df43cc74e" ] ;
sd:status sd:Accepted .
44 changes: 36 additions & 8 deletions .spiral/designs/DES-20260919-TTZ7C-24.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,19 +62,47 @@ directory, acquired in sorted path order. This deliberately treats stores sharin
the default index/log directory as conflicting. Resolve existing parent paths;
do not allow symlink aliases to create independent locks on the same directory.

Write owner diagnostics (random token, PID, host, start time) and sync the lock
entry before any store mutation. The token must match on release; clean shutdown
removes the lock and syncs its parent. Crash leftovers require an explicit offline
administrator release after confirming no writer remains. PID age, a timeout, or
a failed signal probe alone must never authorize automatic stealing. A crash
between mkdir and metadata publication is still a lock, not an empty slot.
Write owner diagnostics (random token, PID, host, start time, purpose) and sync
the lock entry before any store mutation. The token must match on release; clean
shutdown removes the lock and syncs its parent. PID age, a timeout, or a failed
signal probe alone must never authorize automatic stealing. A crash between
mkdir and metadata publication is still a lock, not an empty slot.

Revised 2026-09-26 for REQ-20260926-TTZ7C-66: the owner also proves it is alive.
Before publishing its diagnostics, it listens on a Unix socket `owner.sock` in
the lock and records `liveness: "socket"`; release closes and removes it. The
kernel refuses connections to a socket whose process has ended, however it
ended. A server start (`purpose: "runtime"`) that finds such a runtime lock
connects to it: a refused connection proves the owner is gone. Any other
outcome, including success, a missing or partial lock, a lock without
`liveness`, another purpose, an error or a timeout, is no proof and startup
refuses as before. Paths too long for a socket or filesystems without sockets
publish no liveness and keep administrator release.

Takeover serializes on an atomically created `.simplystore-lock.takeover`
directory beside the lock. Holding it, the starter re-reads the lock, requires
the same token and a refused connection again, writes a takeover audit record
with the previous owner's diagnostics into the store directory, moves the dead
lock aside, creates its own lock as usual, removes the old one and then the
takeover directory. A leftover takeover directory blocks takeover and is
removed by the administrator release. Ownership taken this way proves only
that no writer remains: the unchanged strict startup assessment still refuses
unfinished commands, and the new owner's failed start releases its ownership
as any failed start does. Other crash leftovers, including administrative
locks, still require an explicit offline administrator release. The proof
assumes one machine uses a store directory.

This is cooperative exclusion, not protection against arbitrary file writers.
Read-only inspection does not create source lock files. Backup and recovery apply
require a stopped/quiescent source plus ownership; verify source fingerprints
after acquiring it. Tests must demonstrate competing startup/apply exclusion and
crash-leftover handling. Kernel advisory locks are an alternative, but would need
a proven Node integration; no new native dependency is chosen in this step.
crash-leftover handling. Kernel advisory locks were the named alternative; a
listening socket provides the same kernel-held evidence of process death in
Node without a native dependency.

A server handles stop signals from its start. A stop that arrives while the
store is opening lets opening finish (or fail), then closes and releases
ownership normally.

Within the owner process use one acceptance FIFO and one serialized status writer.
Worker messages do not release the command-runner slot until the parent finishes
Expand Down
4 changes: 3 additions & 1 deletion .spiral/designs/DES-20260919-TTZ7C-24.ttl
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,6 @@ project:DES-20260919-TTZ7C-24 a sd:Design ;
sd:gitCommit "9958f0e296e84169c21c02aa478ac4aae5a06a41" ] ;
sd:changeCausedBy [ a sd:ArtifactReference ;
sd:artifact project:SRC-20260919-TTZ7C-18 ;
sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] .
sd:gitCommit "9fecb10ba1b58c79779394130e1c5cf500206ea9" ] ;
sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260926-TTZ7C-66 ; sd:gitCommit "26bd6babd3d2fd9af191a861cc166c271ee62c89" ] ;
sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] .
56 changes: 56 additions & 0 deletions .spiral/evidence/EVD-20260926-TTZ7C-68.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Self-healing ownership evidence

Verifies the implementation at d393787 (design 2cfb2ee) for
REQ-20260926-TTZ7C-66. Node 24.7.0 on Linux; Docker `node:24-alpine`.

## Regressions

`test/store-ownership.mjs` holds ownership in separate processes killed with
SIGKILL: a running owner is refused; a killed runtime owner is taken over with
an audit record naming the previous token and PID; recovery locks, a legacy
lock without liveness, a lock whose socket is missing, and non-runtime starts
are refused; a store path too long for a socket publishes no liveness and is
refused; six concurrent contenders for a dead lock leave exactly one owner
(20 of 20 repeated runs); a leftover takeover directory blocks takeover and
administrator release removes it together with a socket-bearing lock.

`test/durability-crash.mjs`: an idle server killed with SIGKILL starts again
and serves queries; a server killed during a command now starts only as far as
"Administrative recovery required", with its active status intact and no lock
left; a SIGTERM while a slowed load is opening exits 0 without listening and
leaves no lock.

At baseline 460ef5b, eight of these failed; the running-owner and long-path
refusals already held.

`fs.cp` cannot copy socket files, which failed PL01-05's crash-image copy. The
helper now skips sockets, as `tar` and `rsync` do; the image and its
administrator unlock are otherwise unchanged. SimplyStore's backup, restore
and recovery copy individual files and never copy lock directories.

The 15-file suite passes 215 tests; `npm test` passes 141. Every changed file
lints clean; `src/` has 52 pre-existing problems against 53 on master.

## Docker, end to end

The actual curriculum-restapi-dev compose file ran with an override naming a
disposable curriculum-store copy, whose installed SimplyStore `src/` was
replaced by this branch's:

| Case | Result |
| --- | --- |
| SIGKILL (exit 137), then `up` | takeover logged, audit written, query HTTP 200 |
| Ctrl-C after healing | exit 0, lock released |
| Ctrl-C 0.5, 1, 1.5 s after the lock appears | exit 0, not listening, lock released |
| lock in the 0.11.1 format | `Store is locked`, exit 1 |

## Size and limits

Product code: +253/−30 lines (ownership +181, administrator release +33,
server +30/−16, runtime +8); tests +236; documentation in `docs/recovery.md`
and `DURABILITY.md`. This exceeds the pre-design estimate of about 150 lines,
mainly for the serialized takeover and administrator handling.

Unproven: filesystems without Unix sockets other than the long-path case,
Windows, and power loss on a real device (the modeled power-loss suite passes;
its images carry no socket). Stores shared between machines are unsupported.
10 changes: 10 additions & 0 deletions .spiral/evidence/EVD-20260926-TTZ7C-68.ttl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
@prefix sd: <https://muze.nl/ns/spiral-developer#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .

project:EVD-20260926-TTZ7C-68 a sd:VerificationEvidence ;
dcterms:identifier "EVD-20260926-TTZ7C-68" ;
sd:repositoryPath ".spiral/evidence/EVD-20260926-TTZ7C-68.md" ;
sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "d39378771b5eb8720bfb606b8d69e089273df624" ] ;
sd:verifies [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "2cfb2ee6d202ce63828af985c26569a7c78b32c2" ] ;
sd:status sd:Active .
17 changes: 17 additions & 0 deletions .spiral/implementations/IMP-20260919-TTZ7C-26.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,3 +211,20 @@ The example exposes npm run init-integrity with a declarative store.json.
Regression coverage and disposable-copy example verification preserve existing
commands. Performance comparison uses the accepted optional baseline and identical
30-command runs; canonical hashes are reused and index entries share one barrier.

## Self-healing ownership

Implements the 2026-09-26 revision of DES-20260919-TTZ7C-24 for
REQ-20260926-TTZ7C-66. `acquireOwnership` listens on `owner.sock` in each lock
before publishing `liveness: "socket"` (skipped when the path exceeds 107 bytes
or listening fails) and closes and removes it on release. For
`purpose: "runtime"`, an existing lock whose owner is a runtime with socket
liveness and whose socket refuses connections is taken over under
`.simplystore-lock.takeover`: re-verify the same token, append the previous
owner to `.simplystore-takeovers.jsonl`, move the lock aside, create a new one,
then remove the old lock and the takeover directory. `StoreRuntime.open` takes
runtime ownership and logs takeovers. `run()` handles SIGTERM/SIGINT from its
start; a stop while opening closes after opening settles. The administrator
release accepts `owner.sock` and removes interrupted takeover directories,
retaining any previous owner record in its preview. The power-loss image helper
skips sockets, as `tar` and `rsync` do.
8 changes: 4 additions & 4 deletions .spiral/implementations/IMP-20260919-TTZ7C-26.ttl
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@ project:IMP-20260919-TTZ7C-26 a sd:Implementation ;
sd:implementationLocation "docs/recovery.md" ;
sd:implementationLocation "DURABILITY.md" ;
sd:implementationLocation "eslint.config.js" ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-24 ; sd:gitCommit "2cfb2ee6d202ce63828af985c26569a7c78b32c2" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260919-TTZ7C-25 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-003 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-14 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260819-09ZEF-7 ; sd:gitCommit "59179b6c93b7e21c3b2c69db46c53fbe706927c3" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-47 ; sd:gitCommit "83edb03bc406b9e651ef93bed69b87ece17dcf47" ] ;
sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:DEF-20260923-TTZ7C-50 ; sd:gitCommit "4bb10c2013fd2a7cb989b02586b992b936121362" ] ;
sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ;
sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "1e119dd101dd231a7cf24ba1b9d6c3e4999e8f60" ] ;
sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-54 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ;
sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-53 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ;
sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:REQ-20260926-TTZ7C-66 ; sd:gitCommit "26bd6babd3d2fd9af191a861cc166c271ee62c89" ] ;
sd:implementationLocation "src/store-runtime.mjs", "test/store-ownership.mjs", "test/durability-crash.mjs", "test/power-loss-helpers.mjs" ;
sd:implementationChangeKind sd:SemanticChange .
27 changes: 15 additions & 12 deletions .spiral/project-context.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,21 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira

Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`.

Latest accepted cycle: CYC-20260926-TTZ7C-63, object identity across the query
isolate, on spiral/CYC-20260926-TTZ7C-63-schema-identity, accepted on 2026-09-26
at fa6c791. It corrects DEF-20260926-TTZ7C-62 (curriculum-store's shared schema
exhausted query limits) and, at the maintainer's request, identity for all query
data: stable private schema handles, schema ids in query workers, record-derived
output ids for shared objects without an `id`, and per-query tagged scalar
identity. EVD-20260926-TTZ7C-64 records 207 passing tests and about 4% added
cost on large scans. It was a deliberate deviation from the roadmap; object
history remains the next roadmap target. Integration is pending; no cycle is
active.

The preceding cycle, CYC-20260924-TTZ7C-56 (isolated-vm query execution, SimplyStore
Latest accepted cycle: CYC-20260926-TTZ7C-67, self-healing store ownership, on
spiral/CYC-20260926-TTZ7C-67-self-healing-ownership, accepted on 2026-09-26 with
SimplyStore 0.11.2. A server takes over the lock of a runtime owner whose
liveness socket refuses connections (audited, serialized) and then applies the
unchanged startup assessment; other locks still need administrator release;
stops during startup release ownership. It revises DES-20260919-TTZ7C-24 by
maintainer decision and was a deliberate deviation from the roadmap; object
history remains the next roadmap target. EVD-20260926-TTZ7C-68 records 215
passing tests and a Docker heal after SIGKILL. Integration is pending; no cycle
is active.

CYC-20260926-TTZ7C-63, object identity across the query isolate, is accepted and
integrated (pull request #28, SimplyStore 0.11.1).

Earlier, CYC-20260924-TTZ7C-56 (isolated-vm query execution, SimplyStore
0.11.0), is accepted and integrated at bd61d9c (pull request #27).

Queries must have no import, Node filesystem/network/database authority. Preserve
Expand Down
25 changes: 25 additions & 0 deletions .spiral/requests/REQ-20260926-TTZ7C-66.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Self-healing store ownership

Derived from SRC-20260926-TTZ7C-65.

1. A server that stops during startup releases its ownership. A stop request
before the store is open lets opening finish, then closes normally.
2. When a previous server's ownership remains after a crash, SIGKILL or power
loss, the next server start takes it over only when the previous owner is
proven gone, records the takeover, and then applies the unchanged strict
startup assessment. Unfinished commands still require administrator
recovery; nothing is rerun automatically.
3. When proof is unavailable (the owner may be alive, the lock predates this
mechanism, the lock belongs to an administrative operation, the filesystem
or path cannot hold the proof, or its state is incomplete), startup refuses
exactly as before and the documented administrator release still works.
4. Two concurrent starts never both own the store.

Assumption made explicit: one machine uses a store directory at a time. The
proof is local to the kernel that runs the owner.

Acceptance: a server killed with SIGKILL while idle starts again without manual
release; a server killed during a command starts no further than administrator
recovery required; a running owner, a legacy lock and a recovery lock are never
taken over; concurrent takeovers leave exactly one owner; a stop during startup
leaves no lock; administrator unlock handles the new lock contents.
9 changes: 9 additions & 0 deletions .spiral/requests/REQ-20260926-TTZ7C-66.ttl
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
@prefix sd: <https://muze.nl/ns/spiral-developer#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .

project:REQ-20260926-TTZ7C-66 a sd:Request ;
dcterms:identifier "REQ-20260926-TTZ7C-66" ;
sd:repositoryPath ".spiral/requests/REQ-20260926-TTZ7C-66.md" ;
sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:SRC-20260926-TTZ7C-65 ; sd:gitCommit "70a2b2a58fde152b9d5b8dae0b7e4aaa270ddd54" ] ;
sd:status sd:Active .
Loading
Loading