Spiral/cyc 20260923 ttz7 c 44 file backed

.spiral/cycles/CYC-20260923-TTZ7C-44.md

@@ -0,0 +1,40 @@
+---
+id: CYC-20260923-TTZ7C-44
+---
+
+# File-backed dataset runtime
+
+Goal: replace shared-memory dataset transport and retention with file-backed
+reads throughout live operation and administrative replay, preserving remote
+behavior, disk formats, durable command ordering and recovery restrictions.
+
+Branch: spiral/CYC-20260923-TTZ7C-44-file-backed, from master dbb0703.
+Status: Active. Human authorization is retained in SRC-20260923-TTZ7C-45.
+Continue the file-only roadmap direction; prioritize it ahead of rebuild under
+the maintainer's explicit direction. Library readiness was established first.
+
+Evaluate with existing durability/recovery tests, file-only worker messages,
+ID lookup of unread records, sparse overlays, no-op commands, worker replacement
+and shutdown, index/framing/integrity failures and a bounded-memory large-file
+probe. Source changes must remain confined to disposable fixtures during tests.
+Client version metadata/audit history and sandbox replacement remain later work.
+
+## Evaluation
+
+Implementation, including requested persisted-index loading, is complete at
+`d9d8ff860ed2b74585ea66454382c9baf4a09250`. EVD-20260923-TTZ7C-49
+records 155 passing tests, clean focused lint, fresh-package installation, a
+329 MB/48 MB-heap runtime probe and read-only curriculum evidence.
+See [the integration guide](../../docs/file-data.md) for the source contract and
+limits. Human acceptance remains pending; the cycle stays Active.
+
+Final review added DEF-20260923-TTZ7C-50 and two reproducing tests. The correction
+stops mutation after late lazy-read failures, including caught errors, and keeps
+query-supplied flags from masquerading as parser failures. EVD-49 records the
+full 142-test pass and the final 23-test response-format check.
+
+The maintainer's example-folder verification and requirement to finish persisted
+index loading are retained in the revised SRC-45. REQ-46/DES-47 now require
+validated sidecar use and fallback. The extension removes full-body ID decoding
+from normal startup, while retaining canonical byte scans and lazy read-failure
+handling. EVD-49 records exact full/focused tests and new measurements.

.spiral/cycles/CYC-20260923-TTZ7C-44.ttl

@@ -0,0 +1,11 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:CYC-20260923-TTZ7C-44
+    a sd:Cycle ;
+    dcterms:identifier "CYC-20260923-TTZ7C-44" ;
+    sd:repositoryPath ".spiral/cycles/CYC-20260923-TTZ7C-44.md" ;
+    sd:status sd:Active  ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-46 ; sd:gitCommit "b279aefe78a262a041b59dff2e2c199233fbcf7d" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260923-TTZ7C-44 ; sd:gitCommit "d9d8ff860ed2b74585ea66454382c9baf4a09250" ] .

.spiral/defects/DEF-20260923-TTZ7C-50.md

@@ -0,0 +1,35 @@
+---
+id: DEF-20260923-TTZ7C-50
+---
+
+# Late file reads need storage-failure semantics
+
+Final review of the file-backed migration identified a changed failure boundary:
+source reads can now fail inside a command or query after initialization. The
+prior buffered runtime read source bytes before execution. DES-47 preserved
+publication/storage failures but did not explicitly carry late read failures
+through the ordinary command/query exception handlers.
+
+Two new regressions on implementation bfeed2c fail (21 other focused tests pass):
+a command's unread record raises Unexpected end of file without storageFailure;
+a query that catches its unread-record error can return success, leaving future
+command acceptance enabled. The fixtures truncate only disposable data files
+after both query pools have initialized. This models unavailable backing bytes;
+normal operation still requires immutable source files.
+
+Required correction within the existing migration goal: the file parser records
+read/decode/reference failures in host-owned session state. Commands must refuse
+publication even if handler code catches that failure. Query workers consult
+that host state, not a user-thrown storageFailure property; a spoofed flag must
+remain an ordinary query error. The runtime marks storage failure and refuses
+further commands. Preserve ownership/recovery evidence on failure.
+
+This corrects the migration design's failure classification, not the sandbox or
+remote query language. The baseline tests are committed before the repair.
+
+## Correction
+
+Implemented at faa3e89 and refined at cdc1564. EVD-20260923-TTZ7C-49 records
+the full 142-test pass and focused response-format verification. The tests also
+show that swallowing the read error cannot permit command publication, and
+query-thrown error properties cannot trigger host storage-failure state.

.spiral/defects/DEF-20260923-TTZ7C-50.ttl

@@ -0,0 +1,13 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:DEF-20260923-TTZ7C-50
+    a sd:Defect ;
+    dcterms:identifier "DEF-20260923-TTZ7C-50" ;
+    sd:repositoryPath ".spiral/defects/DEF-20260923-TTZ7C-50.md" ;
+    sd:status sd:Active ;
+    sd:provenanceConfidence sd:Evidenced ;
+    sd:observes [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "bfeed2c5c45cd3ac58dd42aac868585f31c59d19" ]  ;
+    sd:observes [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "cdc1564744989c955a187fdce4baed22ec5d0fd7" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DEF-20260923-TTZ7C-50 ; sd:gitCommit "4bb10c2013fd2a7cb989b02586b992b936121362" ] .

.spiral/designs/DES-20260923-TTZ7C-47.md

@@ -0,0 +1,52 @@
+---
+id: DES-20260923-TTZ7C-47
+---
+
+# Ordered file sources and owned reader sessions
+
+Transport ordered source descriptors (path, byte offsets, size and digest), not
+open descriptors or dataset bytes. A small file-reader session owns handles and
+an od-jsontag parser; queries reuse it, commands open the committed prefix for
+one execution, and all exits close handles. Resolve IDs through getLineProxy.
+
+Load default offset/ID sidecars for the base and log-authoritative committed
+commands. Keep the fixed-buffer framing/hash scan: legacy sidecars have no
+cryptographic binding to canonical bytes. Validate every persisted offset range
+and record key against framing, and pass the loaded offsets to od-jsontag only
+when the complete map agrees. Missing, malformed or stale maps use derived
+offsets. Do not import custom index code during inspection/loading.
+
+During the same scan, collect only each record's leading JSONTag metadata,
+respecting quoted/escaped delimiters, and use JSONTag's own tag parser for IDs.
+Skip record bodies. Merge per-record IDs in committed source order, then apply
+ascending record-number precedence. Compare the composed persisted ID map with
+this complete final map; use persisted IDs when equal and reconstructed IDs
+otherwise. This handles removed/renamed IDs, duplicate IDs and sparse holes
+without trusting an incomplete sidecar. Unusual reference-only records use the
+existing parser reconstruction fallback. Lazy body errors remain storage errors
+when read; startup validates framing, headers and configured full-file integrity,
+not all record-body syntax.
+
+Read-only load never rewrites sidecars or requires a new format/cache certificate.
+The scan remains O(file bytes), and catalogs/IDs O(record count); the improvement
+is avoiding full live-record decoding, not skipping integrity reads.
+
+Startup inspection returns plain file sources instead of retained data/buffers.
+Administrative recovery uses the same loader/session. Temporary command output
+may remain buffered for the existing custom finalize(serialized, meta, uuid)
+hook; it is written durably and returned as a file descriptor object. This
+preserves the hook and avoids holding the whole dataset in memory. Streaming
+full-dataset serialization is unnecessary once startup stops compacting it.
+
+Worker updates must serialize with queries and replacement workers must receive
+the current committed source list. Await termination before releasing store
+ownership. OS/Node worker teardown closes descriptors on timeout/failure.
+
+Alternative considered: keep a buffered fallback or merely replace query reads.
+That would leave startup/recovery retaining entire datasets and duplicate modes;
+a shared file-session boundary gives one implementation and test surface.
+
+Pin the reviewed od-jsontag Git commit until a separately authorized release.
+No dependency publish, remote deployment or user data mutation is part of this
+implementation. Validate behavior with independent byte-offset oracles and the
+existing durability suite.

.spiral/designs/DES-20260923-TTZ7C-47.ttl

@@ -0,0 +1,11 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:DES-20260923-TTZ7C-47
+    a sd:Design ;
+    dcterms:identifier "DES-20260923-TTZ7C-47" ;
+    sd:repositoryPath ".spiral/designs/DES-20260923-TTZ7C-47.md" ;
+    sd:status sd:Active ;
+    sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-46 ; sd:gitCommit "b279aefe78a262a041b59dff2e2c199233fbcf7d" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-47 ; sd:gitCommit "b279aefe78a262a041b59dff2e2c199233fbcf7d" ] .

.spiral/evidence/EVD-20260923-TTZ7C-49.md

@@ -0,0 +1,146 @@
+---
+id: EVD-20260923-TTZ7C-49
+---
+
+# File-backed runtime evaluation
+
+Implementation: `cdc1564744989c955a187fdce4baed22ec5d0fd7` on
+`spiral/CYC-20260923-TTZ7C-44-file-backed`.
+
+## Evaluation
+
+The authorized migration is implemented and ready for human review. Runtime,
+query workers, command workers, startup inspection and administrative replay use
+ordered file sources. The remote query/command API and stored data/log formats
+remain unchanged. No production data, deployment, merge or release was changed.
+Existing untracked example indexes and package backup were left untouched.
+
+- Full regression command below: **142 tests pass**, zero failures/cancellations
+  (118 existing cases plus 24 new). Runtime/startup, conversion and offsets,
+  ACID, process crashes, handlers, power-loss baseline, storage and administrative
+  recovery remain covered. Adaptations to old tests change internal source/report
+  fixtures; existing remote behavior and durability assertions remain.
+- New regressions cover sparse overlays and record allocation, untouched IDs,
+  byte-free worker messages, stale/missing sidecars, UTF-8 and short reads,
+  malformed framing/overflow/premature EOF, source identity changes and handle
+  cleanup, integrity, index loader paths, ID-based command edits, no-op changes,
+  update ordering while workers initialize, replacement at current head,
+  failed update refusal, awaited shutdown, real commit/query/reopen, and finalizer
+  tampering. Duplicate-ID precedence follows record number even when a patch
+  fills an earlier hole.
+- Array `map`, JAQT projection and mutation/defineProperty refusal are exercised
+  through the real query sandbox. Those new checks exposed vm2's temporary array
+  constructor mutation, which the QueryParser compatibility adapter avoids
+  using vm2's already-safe constructor shape. No sandbox replacement or broader
+  security assessment is claimed.
+- Focused ESLint for all changed runtime modules, new tests and benchmark:
+  **zero errors and warnings**. `git diff --check` passes.
+- Local package creation and installation into a fresh temporary consumer pass;
+  the installed package reads an indexed file. New source, documentation and
+  benchmark are included. The dependency resolves the pinned upstream od-jsontag
+  revision. No package was published.
+- End-to-end generated store: **328,877,813 bytes / 20,000 records**, with a
+  **48 MB JavaScript heap limit**. Startup and a direct last-record ID query
+  pass. One run measured 17,498 ms opening and 93 ms for the query including
+  worker readiness. Parent retained heap: 8,905,216 bytes; query-worker heap:
+  24,427,720 bytes. Array buffers: 24,823 and 57,562 bytes respectively. This is
+  evidence against retained dataset buffers, not a total-process-memory or
+  cold-storage bound. The benchmark uses and removes only a disposable fixture.
+- Read-only curriculum probe: **77,952,430 bytes / 181,724 records / 180,416 IDs**,
+  1,003 sampled ID reads, clean cache capped at 256 records, about 7.9 seconds
+  including ID reconstruction. Source SHA-256 before and after matched. No
+  dataset or sidecar was edited.
+
+```sh
+node --test --test-reporter=tap test/file-data.mjs test/store-runtime.mjs \
+  test/durability-startup.mjs test/conversion-offsets.mjs \
+  test/command-offsets.mjs test/acid-baseline.mjs test/durability-crash.mjs \
+  test/after-change-handlers.mjs test/power-loss-baseline.mjs \
+  test/storage.mjs test/admin-recovery.mjs
+node --expose-gc --max-old-space-size=48 scripts/benchmark-file-data.mjs
+```
+
+## Final read-failure correction
+
+DEF-20260923-TTZ7C-50 records two failing regressions introduced during final
+review. Late lazy reads could be treated as ordinary command errors or swallowed
+inside query code. FileParser now records read/decode/reference failure in
+host-owned session state. Commands refuse publication even when a handler catches
+the failure; query workers signal it to the runtime even when query code catches
+it. Forged user-thrown storageFailure properties do not change that state.
+The runtime stops mutation and retains ownership/recovery evidence.
+
+The complete suite passes 142 tests after that correction (faa3e89). The final
+response-format refinement (cdc1564) additionally passes all 23 focused file-data
+tests, including JSONTag error negotiation. Focused lint remains clean. Earlier
+memory, curriculum and package probes establish the unchanged file-source and
+distribution contracts; they are not separate security or hardware claims.
+
+## Persisted-index completion, 2026-09-23
+
+The maintainer required persisted index loading in this version after review
+identified its absence, and reported successful operation with the example
+folder. SRC-45 retains that feedback. This is supporting evidence, not cycle
+acceptance; the agent did not modify or rerun the live example folder.
+
+Implementation d9d8ff8 reads base and committed-command offset/ID sidecars.
+Valid offsets are passed directly to the parser; a complete validated composed
+ID map is used. Canonical framing and record headers supply validation and
+fallback, without decoding every record body or writing sidecars.
+
+- Complete suite: **155 tests pass** (142 previous plus 13 index/lazy-body cases),
+  `/tmp/simplystore-index-full-final.log`. The final edit only removes a redundant
+  escape in a test string without changing its value; all **36 focused tests**
+  pass afterwards (`/tmp/simplystore-index-focused-final.log`).
+- Focused ESLint on file-data, its tests and the benchmark: zero errors/warnings
+  (`/tmp/simplystore-index-lint-final.log`). Whitespace checks pass.
+- Tests observe actual base/committed-sidecar loader calls and the returned
+  offset object's identity; root-only parsing is checked with a parser spy and
+  full-record ID reconstruction is forbidden in that case. Missing, malformed,
+  partial, redirected and extra index entries fall back correctly. Overlays
+  handle ID removal/renaming and restoration of a lower duplicate.
+- Header tests cross buffer boundaries with short reads and 70 KB metadata,
+  Unicode, backslashes, quoted delimiters and prototype-named IDs. They compare
+  results with full parser reconstruction. Invalid tags fail; unread invalid
+  bodies raise host-owned storage failures when accessed. JSONTag's attribute
+  API is used as well as its tag parser, preserving attribute validation.
+- Same 328,877,813-byte / 20,000-record fixture and 48 MB JS heap limit, with
+  independently generated persisted offsets/IDs: open **1,076.6 ms**, query
+  **91.2 ms**, parent heap **8,990,544 bytes**, worker heap **24,561,248 bytes**.
+  Retained buffers: **24,823 / 57,562 bytes**. Log:
+  `/tmp/simplystore-index-memory.log`. Earlier full-body ID reconstruction was
+  about 17.5 seconds. The main improvement is header-only validation.
+- Same fixture with `--indexes=missing`: open **1,123.1 ms**, query **97.6 ms**;
+  passes the heap limit and ID lookup. Log:
+  `/tmp/simplystore-index-missing-memory.log`.
+- Real curriculum file, read-only: **77,952,430 bytes / 181,724 records / 180,416
+  IDs**. Loader took **1,047.2 ms**; independent full-parser ID reconstruction
+  took **7,436.6 ms**. The complete Maps agree and before/after source digests
+  match (`/tmp/simplystore-index-curriculum.log`). No dataset or sidecar changed.
+
+These local warm measurements do not establish a general latency or total-RAM
+bound. Startup still scans every canonical byte; no new index format, trusted
+certificate cache, dependency change or separate power-cut campaign was added.
+The earlier fresh-package probe remains applicable to the distribution contract;
+this extension changes existing packaged files only.
+
+## Boundaries and review disposition
+
+Startup hashes all canonical bytes and validates persisted IDs from record
+headers, with reconstruction as fallback. Index catalogs remain O(records), handles O(source files per worker),
+and mutable sessions retain touched data. Command output/custom hooks/query
+results may be large. Schemas and logs remain in memory; administrative copying
+still temporarily buffers individual files. These are documented boundaries,
+not a constant-memory claim. The large-file result does not establish a broader
+filesystem or hardware durability envelope; the separate QEMU power-cut campaign
+was not repeated.
+
+Internal custom worker implementations must use the new source contract;
+`inspectStore` returns `sources` instead of live `data`/`buffers`. Existing custom
+index finalizers retain their call contract. Source files must be immutable
+while open under store ownership. A Git dependency pin is used until a separate
+od-jsontag npm release. Client version metadata/audit history and query sandbox
+selection remain later work.
+
+Human outcome acceptance is pending. This Active cycle is not eligible for
+integration until accepted and its exact prospective merge is validated.

.spiral/evidence/EVD-20260923-TTZ7C-49.ttl

@@ -0,0 +1,13 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:EVD-20260923-TTZ7C-49
+    a sd:VerificationEvidence ;
+    dcterms:identifier "EVD-20260923-TTZ7C-49" ;
+    sd:repositoryPath ".spiral/evidence/EVD-20260923-TTZ7C-49.md" ;
+    sd:status sd:Active ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-26 ; sd:gitCommit "d9d8ff860ed2b74585ea66454382c9baf4a09250" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "d9d8ff860ed2b74585ea66454382c9baf4a09250" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-34 ; sd:gitCommit "d9d8ff860ed2b74585ea66454382c9baf4a09250" ]  ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:EVD-20260923-TTZ7C-49 ; sd:gitCommit "d9d8ff860ed2b74585ea66454382c9baf4a09250" ] .