Spiral/cyc 20260923 ttz7 c 51 required integrity

.spiral/cycles/CYC-20260923-TTZ7C-44.md

@@ -0,0 +1,65 @@
+---
+id: CYC-20260923-TTZ7C-44
+---
+
+# File-backed dataset runtime
+
+Goal: replace shared-memory dataset transport and retention with file-backed
+reads throughout live operation and administrative replay, preserving remote
+behavior, disk formats, durable command ordering and recovery restrictions.
+
+Branch: spiral/CYC-20260923-TTZ7C-44-file-backed, from master dbb0703.
+Status: Accepted on 2026-09-23. Authorization is retained in SRC-45;
+human acceptance and independent example checks are recorded below.
+Continue the file-only roadmap direction; prioritize it ahead of rebuild under
+the maintainer's explicit direction. Library readiness was established first.
+
+Evaluate with existing durability/recovery tests, file-only worker messages,
+ID lookup of unread records, sparse overlays, no-op commands, worker replacement
+and shutdown, index/framing/integrity failures and a bounded-memory large-file
+probe. Source changes must remain confined to disposable fixtures during tests.
+Client version metadata/audit history and sandbox replacement remain later work.
+
+## Evaluation
+
+Implementation, including requested persisted-index loading, is complete at
+`46fb22fe5c4fecbb23c3b8d3b59e5ff6e58e11c2`. EVD-20260923-TTZ7C-49
+records 178 passing tests, clean focused lint, fresh-package installation, a
+329 MB/48 MB-heap runtime probe and read-only curriculum evidence.
+See [the integration guide](../../docs/file-data.md) for the source contract and
+limits. Human acceptance is recorded below; integration remains pending.
+
+Final review added DEF-20260923-TTZ7C-50 and two reproducing tests. The correction
+stops mutation after late lazy-read failures, including caught errors, and keeps
+query-supplied flags from masquerading as parser failures. EVD-49 records the
+full 142-test pass and the final 23-test response-format check.
+
+The maintainer's example-folder verification and subsequent corrections are
+retained in SRC-45. REQ-46/DES-47 now require direct trusted ID loading, reliable
+final ID publication, errors for duplicate IDs and no legacy index compatibility.
+Missing indexes reconstruct; validation/reconstruction can be explicitly
+requested. EVD-49 records the 170-test final pass and direct-index memory probe.
+This evaluation preceded the human acceptance recorded below.
+
+The maintainer then confirmed offset trust and file hashes. Normal startup now
+loads both indexes directly, including inspection; explicit validation or missing
+offsets alone reconstruct framing. Existing integrity manifests cover final
+standard indexes too. EVD-49 records 178 passing tests, clean lint, and the updated
+large-file probe. Human review concluded with the acceptance below.
+
+## Human acceptance
+
+On 2026-09-23, after the completed implementation and evaluation at
+`8013c0c74fb1f01bb882b0b221c4502ef4970b85`, the maintainer confirmed:
+
+> confirmed tests, example works, ran a command, change was applied, and restarting kept it. Happy.
+
+This accepts the file-backed runtime cycle, including trusted committed offset
+and ID indexes and independent file integrity hashes. The maintainer reports
+confirming the tests and independently observing the example apply a command
+and retain the change after restart. These are human-reported checks, separate
+from the 178 automated tests retained in EVD-20260923-TTZ7C-49.
+
+The cycle is closed and ready to propose for integration into master with a
+normal merge commit. Integration remains a separate decision; no next cycle
+is selected. The branch identity was rechecked when recording acceptance.

.spiral/cycles/CYC-20260923-TTZ7C-44.ttl

@@ -0,0 +1,11 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:CYC-20260923-TTZ7C-44
+    a sd:Cycle ;
+    dcterms:identifier "CYC-20260923-TTZ7C-44" ;
+    sd:repositoryPath ".spiral/cycles/CYC-20260923-TTZ7C-44.md" ;
+    sd:status sd:Accepted  ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-46 ; sd:gitCommit "46fb22fe5c4fecbb23c3b8d3b59e5ff6e58e11c2" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260923-TTZ7C-44 ; sd:gitCommit "8013c0c74fb1f01bb882b0b221c4502ef4970b85" ] .

.spiral/cycles/CYC-20260923-TTZ7C-51.md

@@ -0,0 +1,46 @@
+# Required integrity and existing-store initialization
+
+Status: Accepted on 2026-09-23. Continue the roadmap durability/file-backed direction under
+SRC-20260923-TTZ7C-52. The confirmed goal is mandatory artifact integrity,
+explicit initialization for existing stores, and measured command latency.
+
+Branch: spiral/CYC-20260923-TTZ7C-51-required-integrity.
+Base exception: start from accepted CYC-44 tip `4680c76c8faabde53ecd712e1b903f2e8b0eddb6` because it supplies the
+required file/index behavior and awaits separate integration authorization.
+Do not merge it into master as part of this task. Preserve accepted history.
+
+Scope: conversion, startup/loader, commands, recovery, stopped-store initializer,
+example documentation, regression coverage and before/after latency evidence.
+Keep trusted index loading; no automatic re-baselining, command replay during
+initialization, data rewrite, version/audit feature, or new hash format.
+Preserve the maintainer's example data and command. Test with disposable stores.
+
+## Evaluation
+
+Implementation is complete at `1e119dd101dd231a7cf24ba1b9d6c3e4999e8f60`. EVD-55 records 193 passing tests,
+clean focused lint, successful initialization of a disposable copy of the actual
+example with its command retained, and a 329 MB/48 MB-heap probe. Three alternating
+30-command latency runs on local ext4 measured roughly 55 ms baseline versus
+57 ms required-integrity median completion. There is one durable hash append
+per artifact set. Human acceptance is recorded below; integration is pending.
+
+The example now documents npm run init-integrity for a stopped existing store.
+Actual user data/index/log files were preserved. Integration of accepted CYC-44
+and this cycle is still a separate decision.
+
+## Human acceptance
+
+After testing the evaluated branch tip `9515993531c55176e1c3444f71dc3a23d5172df0`, the maintainer confirmed
+on 2026-09-23:
+
+> works as expected, fails to start without integrity now, init works, commands work.
+
+This accepts the mandatory-integrity and existing-store initialization cycle.
+The maintainer independently reports startup refusal without integrity, successful
+initialization, and working commands. These are human-reported checks alongside
+the 193 automated tests and latency evidence in EVD-20260923-TTZ7C-55.
+
+The cycle is closed and ready to propose for integration with a normal merge
+commit. This branch includes accepted CYC-44 and CYC-51. Integration remains a
+separate decision; no next cycle is selected. The actual branch identity was
+rechecked when recording acceptance.

.spiral/cycles/CYC-20260923-TTZ7C-51.ttl

@@ -0,0 +1,11 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:CYC-20260923-TTZ7C-51
+    a sd:Cycle ;
+    dcterms:identifier "CYC-20260923-TTZ7C-51" ;
+    sd:repositoryPath ".spiral/cycles/CYC-20260923-TTZ7C-51.md" ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-53 ; sd:gitCommit "1e119dd101dd231a7cf24ba1b9d6c3e4999e8f60" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260923-TTZ7C-51 ; sd:gitCommit "9515993531c55176e1c3444f71dc3a23d5172df0" ] ;
+    sd:status sd:Accepted .

.spiral/defects/DEF-20260923-TTZ7C-50.md

@@ -0,0 +1,35 @@
+---
+id: DEF-20260923-TTZ7C-50
+---
+
+# Late file reads need storage-failure semantics
+
+Final review of the file-backed migration identified a changed failure boundary:
+source reads can now fail inside a command or query after initialization. The
+prior buffered runtime read source bytes before execution. DES-47 preserved
+publication/storage failures but did not explicitly carry late read failures
+through the ordinary command/query exception handlers.
+
+Two new regressions on implementation bfeed2c fail (21 other focused tests pass):
+a command's unread record raises Unexpected end of file without storageFailure;
+a query that catches its unread-record error can return success, leaving future
+command acceptance enabled. The fixtures truncate only disposable data files
+after both query pools have initialized. This models unavailable backing bytes;
+normal operation still requires immutable source files.
+
+Required correction within the existing migration goal: the file parser records
+read/decode/reference failures in host-owned session state. Commands must refuse
+publication even if handler code catches that failure. Query workers consult
+that host state, not a user-thrown storageFailure property; a spoofed flag must
+remain an ordinary query error. The runtime marks storage failure and refuses
+further commands. Preserve ownership/recovery evidence on failure.
+
+This corrects the migration design's failure classification, not the sandbox or
+remote query language. The baseline tests are committed before the repair.
+
+## Correction
+
+Implemented at faa3e89 and refined at cdc1564. EVD-20260923-TTZ7C-49 records
+the full 142-test pass and focused response-format verification. The tests also
+show that swallowing the read error cannot permit command publication, and
+query-thrown error properties cannot trigger host storage-failure state.

.spiral/defects/DEF-20260923-TTZ7C-50.ttl

@@ -0,0 +1,13 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:DEF-20260923-TTZ7C-50
+    a sd:Defect ;
+    dcterms:identifier "DEF-20260923-TTZ7C-50" ;
+    sd:repositoryPath ".spiral/defects/DEF-20260923-TTZ7C-50.md" ;
+    sd:status sd:Active ;
+    sd:provenanceConfidence sd:Evidenced ;
+    sd:observes [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "bfeed2c5c45cd3ac58dd42aac868585f31c59d19" ]  ;
+    sd:observes [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "cdc1564744989c955a187fdce4baed22ec5d0fd7" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DEF-20260923-TTZ7C-50 ; sd:gitCommit "4bb10c2013fd2a7cb989b02586b992b936121362" ] .

.spiral/designs/DES-20260923-TTZ7C-47.md

@@ -0,0 +1,88 @@
+---
+id: DES-20260923-TTZ7C-47
+---
+
+# Ordered file sources and owned reader sessions
+
+Transport ordered source descriptors (path, byte offsets, size and digest), not
+open descriptors or dataset bytes. A small file-reader session owns handles and
+an od-jsontag parser; queries reuse it, commands open the committed prefix for
+one execution, and all exits close handles. Resolve IDs through getLineProxy.
+
+Load base and log-authoritative committed ID sidecars directly. Validate their
+JSON shape, record numbers and uniqueness using catalog metadata; do not read
+record tags to check a present index's contents/completeness in normal startup.
+Missing ID sidecars are reconstructed from their source's headers. Explicit
+validateIndexes checks persisted content; rebuildIndexes ignores persisted IDs
+and reconstructs in memory. Malformed present ID indexes fail normally. Offset
+framing/hash scans and configured integrity validation remain unchanged.
+
+Use the existing compact ID-to-record JSON map; no legacy format path or
+duplicate-ID winner selection. Apply each changeset by removing mappings whose
+records it replaces, then inserting its IDs. Two distinct current records with
+the same ID are an error; swaps and same-record updates remain valid.
+
+JSONTag attribute setters mutate the tag container in place; the reviewed
+od-jsontag revision can therefore miss ID-only changes. Before calling update
+and again after it returns, compare only materialized records' IDs with the
+committed ID map and mark differences dirty. Conversion performs the same
+check around create. This belongs at the store's ID publication boundary and
+requires no full-record traversal or dependency revision. Regression tests must
+use real JSONTag setters, including attribute-only commands.
+
+Extract IDs from final serialized bytes after mutation hooks. Prepare the ID
+map and check global uniqueness before publishing the canonical changeset.
+The core finalizeIndex boundary awaits custom finalization, durably writes the
+canonical ID sidecar, and updates returned metadata. Custom finalizers retain
+their call contract and control their own offset behavior, but cannot bypass
+core ID finalization. Conversion does the same prepublication preparation.
+Capture the expected serialized digest before custom finalization and reject
+changes to canonical bytes before command or conversion success. Empty commands publish {}.
+
+The loader and commit preparation share removal/uniqueness rules. Command
+preparation uses a snapshot of committed IDs, not a mutable hook/parser map.
+Tests must compare live results with restart and prove complete indexes avoid
+header parsing. No persistent reconstruction or canonical rewrite occurs at
+startup. The only byte scan remains framing/hash validation unless an ID index
+is missing or maintenance validation/reconstruction is requested.
+
+Startup inspection returns plain file sources instead of retained data/buffers.
+Administrative recovery uses the same loader/session. Temporary command output
+may remain buffered for the existing custom finalize(serialized, meta, uuid)
+hook; it is written durably and returned as a file descriptor object. This
+preserves the hook and avoids holding the whole dataset in memory. Streaming
+full-dataset serialization is unnecessary once startup stops compacting it.
+
+Worker updates must serialize with queries and replacement workers must receive
+the current committed source list. Await termination before releasing store
+ownership. OS/Node worker teardown closes descriptors on timeout/failure.
+
+Alternative considered: keep a buffered fallback or merely replace query reads.
+That would leave startup/recovery retaining entire datasets and duplicate modes;
+a shared file-session boundary gives one implementation and test surface.
+
+Pin the reviewed od-jsontag Git commit until a separately authorized release.
+No dependency publish, remote deployment or user data mutation is part of this
+implementation. Validate behavior with independent byte-offset oracles and the
+existing durability suite.
+
+## Offset trust correction, 2026-09-23
+
+Share source loading between inspection, loader and publication: load offsets,
+validate structural ranges, hash canonical bytes with a fixed buffer and retain
+the stable file identity. Only missing offsets or explicit maintenance parse
+framing. When IDs alone are missing, read headers at the stored offsets.
+
+Hash the exact index bytes that are parsed, using the existing SHA-256 manifest.
+With integrity enabled, every present standard sidecar requires a manifest entry;
+missing sidecars can reconstruct in memory from verified canonical data. Explicit
+rebuild ignores sidecars including their hashes but still verifies canonical data.
+No automatic mismatch repair and no legacy compatibility path.
+
+Conversion and command/recovery publication append hashes for final standard
+sidecars after hooks and durable index writes, before done/success. Custom
+finalizers may omit offsets; hash only present files. Inspection checks committed
+sidecars too without importing custom hooks. Existing inventory byte hashes and
+immutable source identity checks remain independent of offset reconstruction.
+No claim that hashing proves index semantics: publication establishes consistency;
+explicit validation diagnoses externally generated or modified indexes.

.spiral/designs/DES-20260923-TTZ7C-47.ttl

@@ -0,0 +1,12 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:DES-20260923-TTZ7C-47
+    a sd:Design ;
+    dcterms:identifier "DES-20260923-TTZ7C-47" ;
+    sd:repositoryPath ".spiral/designs/DES-20260923-TTZ7C-47.md" ;
+    sd:status sd:Active ;
+    sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-46 ; sd:gitCommit "15088d395ae6466231edaf932ec7e889171729de" ] ;
+    sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260923-TTZ7C-45 ; sd:gitCommit "7e1ea39dc4056beec311c8e6d832d97d2a550d58" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-47 ; sd:gitCommit "7e1ea39dc4056beec311c8e6d832d97d2a550d58" ] .

.spiral/designs/DES-20260923-TTZ7C-54.md

@@ -0,0 +1,35 @@
+# Mandatory verification with explicit initialization
+
+Always resolve a manifest path in runtime, converter, loader and inspection.
+Ignore the former integrity=false/required=false toggles; they cannot disable
+verification. Keep --integrity as a harmless redundant converter flag. Missing
+manifest diagnostics direct operators to the explicit initialization workflow.
+
+Factor artifact hash collection and manifest serialization: canonical digest
+already verified after finalization plus SHA-256 of final index bytes. Append
+all entries with a single durable appendRecord before done; conversion uses the
+same set. Administrative replay defers its worker's manifest append to its
+coordinator so retained expected digests are checked before replacement hashes
+and done. That is an internal publication boundary, not a store opt-out.
+
+Add initializeIntegrity and recover.mjs init-integrity --store config.json.
+Acquire normal ownership on all mutable directories, refuse any existing
+manifest, inspect an unsealed store via a separate explicit internal entry point,
+validate framing/index contents/IDs and decode records through FileDataset.
+Require a ready, stable committed prefix. Compare inventory after validation,
+then atomically and durably publish hashes from that snapshot for base and
+committed changesets and present standard sidecars. Release ownership on read-only
+validation failures; retain it if durable publication becomes uncertain.
+Never invoke custom hooks or replay commands; never modify data/index/log files.
+This establishes trust in current validated bytes, not proof of past correctness.
+
+Normal inspect/load functions remain mandatory; only the named unsealed
+validation entry points used by initialization can operate without a manifest.
+Existing stores require this operation explicitly. Tests create manifests when
+constructing fixtures, not as a side effect of opening the product.
+
+Evaluate mandatory startup and false-toggle rejection, same-length tampering,
+index verification, missing index fallback, lifecycle and crash barriers,
+initialization/rejection/no rewrite/lock cases and real command/restart behavior.
+Compare repeated local latency runs against accepted 4680c76 with the same
+30-command fixture; report observed variance and filesystem limitations.

.spiral/designs/DES-20260923-TTZ7C-54.ttl

@@ -0,0 +1,8 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+project:DES-20260923-TTZ7C-54 a sd:Design ;
+    dcterms:identifier "DES-20260923-TTZ7C-54" ;
+    sd:repositoryPath ".spiral/designs/DES-20260923-TTZ7C-54.md" ;
+    sd:status sd:Active ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:SRC-20260923-TTZ7C-52 ; sd:gitCommit "44827fde518ed8eca5fbcb606f731aa5e95d5167" ] .