Spiral/cyc 20260924 ttz7 c 56 isolated vm

.spiral/cycles/CYC-20260924-TTZ7C-56.md

@@ -0,0 +1,59 @@
+# Isolated query execution
+
+Goal: register the selected isolated-vm engine and replace VM2 query execution
+with an evaluable, restricted integration that preserves SimplyStore behavior.
+
+Status: Accepted on 2026-09-24. Human commitment: explicit engine choice after benchmark and
+capability evidence, retained in SRC-20260924-TTZ7C-57. Classification: continue the
+file-backed/sandbox roadmap; engine investigation has produced a selection.
+Object history remains the next target, with UI deferred.
+
+Branch: spiral/CYC-20260924-TTZ7C-56-isolated-vm, created from authoritative master
+029b36f. Branch identity checked when opening the cycle. CYC-44 and CYC-51 are
+accepted and integrated on master; the previous context's pending-integration
+text was stale.
+
+In scope: query engine, controlled data/grant interface, supplied query helpers,
+JSON/JSONTag responses, resource limits, dependency/runtime documentation and
+regression/performance evidence. No imports or general host capabilities for
+queries. Trusted command execution and file/integrity formats remain governed
+by their accepted contracts. Hostile queries may lose their worker/session.
+
+Non-goals: history implementation, UI work, broad public-deployment claims,
+engine fallback framework, snapshots, full-history rebuild and unrelated cleanup.
+
+Evaluation ready: isolated-vm replaces VM2, with a fresh isolate per query and
+host-authorized read-only views. The host retains the existing parser and grants;
+this resolves the authority gap in the raw-record probe. Execution/serialization,
+memory, output and outer-worker limits are covered. EVD-20260924-TTZ7C-60 records
+202 passing tests, clean focused lint, package checks, copied-example queries,
+and actual worker-path timings. Implementation is at bd507554a2fed6f77322ef38a98839f127d810db.
+
+The final scan benchmark is about 453 ms for 10,000 records; filtering/sorting/
+projection is about 498 ms. The measured result supports the selected engine,
+with the synthetic/warm-cache scope retained. No snapshots or whole-process
+memory/crash boundary is claimed. Human acceptance is recorded below; integration
+remains pending.
+
+## Human acceptance and version
+
+On 2026-09-24, the maintainer explicitly instructed:
+
+> mark this cycle as accepted, bump simplystore version to 0.11.0 (if that is correct)
+
+This accepts the implementation at `bd8f8eb3ee617c5f9e9198f234725ef7ab5b4201`, including the
+published od-jsontag 0.5.0 dependency and Node parser entry point. The full suite
+was rerun against that published package: 202 tests passed, focused lint passed,
+and queries against a disposable copy of the example store succeeded.
+
+The maintainer emphasized that increased security is the purpose of the change;
+keeping performance comparable to VM2, with modest improvements on larger
+queries, is a welcome secondary result. Acceptance retains the documented limits
+on whole-process memory/crash isolation and public deployment claims.
+
+SimplyStore is bumped from 0.10.3 to 0.11.0. A minor increment is appropriate for
+this pre-1.0 release's engine replacement, Node 22+ requirement, startup flag,
+and query capability/limit changes. No release tag or npm publication is made
+by this version change. The cycle branch identity was verified before recording
+acceptance. The branch is ready for a separately authorized normal merge into
+master; no next cycle has been opened.

.spiral/cycles/CYC-20260924-TTZ7C-56.ttl

@@ -0,0 +1,9 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+project:CYC-20260924-TTZ7C-56 a sd:Cycle ;
+    dcterms:identifier "CYC-20260924-TTZ7C-56" ;
+    sd:repositoryPath ".spiral/cycles/CYC-20260924-TTZ7C-56.md" ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:REQ-20260924-TTZ7C-58 ; sd:gitCommit "fdef01a844e14f50c0af1aac6e89461714736f6d" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260924-TTZ7C-56 ; sd:gitCommit "b176d4553c4f4b62c71571ed1a56475a06f9f975" ] ;
+    sd:status sd:Accepted .

.spiral/designs/DES-20260924-TTZ7C-59.md

@@ -0,0 +1,58 @@
+# Fresh isolates and a host-authorized data view
+
+Use isolated-vm 6.2.0, without snapshots, for each query. Keep the existing
+worker pool and trusted command/storage runtime. The worker owns the established
+FileDataset, offset/ID indexes and access module. Bundle the pure JAQT/JSONTag
+query helpers for the isolate; do not supply an import resolver or Node globals.
+
+The benchmark used a parser inside the guest and raw record callbacks. That is
+not an authorization boundary: parser internals and same-realm mutable library
+objects must not expose raw denied fields. Existing access modules are trusted
+Node modules. Rather than moving them into a hostile realm or stringifying
+functions and silently losing their imports/closures, keep parsing and access
+checks on the host. Supply an isolate-local read-only view with copied values
+and private synchronous callbacks. A disposable prototype scanned 10,000 records
+in about 362–430 ms through this view; the chosen boundary still warrants real
+integration measurements.
+
+References identify record numbers plus property paths (or schema paths), not
+host objects or paths on disk. Resolve them through the host parser for every
+operation so bounded record caches remain effective. Preserve guest identity
+with weak references. Return scalars, tagged scalar copies and descriptions of
+object/array views. Expose only ordinary property names and JSONTag's public
+metadata symbols. Do not forward parser-private symbols, prototypes, methods,
+raw buffers, source handles or isolated-vm References. Mutating the view throws;
+mutating a copied tagged scalar cannot modify stored data. Enforce grants on
+reads, existence, enumeration and descriptors in the host realm.
+
+Each query receives fresh globals/root/data/meta/request and helper instances.
+Run pointer selection, the user script and result serialization inside the
+isolate's deadline. Capture the serializer privately so its Reference never
+enters guest code. Only primitive response text crosses back. Bound UTF-8 result
+size before copying it to Node; reject asynchronous query results. Configure
+64 MiB isolate memory and 10 MiB result bytes by default, with explicit positive
+finite validation. Also limit individual bridge copies. Keep callback values
+small and never pass an untrusted function to a host API.
+
+An outer pool deadline replaces a stuck query worker and initializes its
+replacement from the current committed head. Updates remain serialized with
+queries. Initialization/update failures keep the existing fail-stop behavior.
+Storage failure remains host-owned FileParser state, including when a query
+catches its exception. Guest-thrown properties never classify storage failure.
+Always release references and dispose the isolate, including errors/timeouts.
+
+The native engine's heap limit is not an OS process/RSS limit, and this worker
+thread integration does not claim resilience against every native V8 failure.
+It is the authorized query-engine step, not internet-exposure certification.
+A separate OS process boundary remains a deployment-hardening consideration.
+
+Validate the actual runtime API, grant reflection, Unicode/tagged values, JAQT,
+updates/restart, hostile serialization, imports and post-failure queries. Keep
+accepted durability tests. Benchmarks must distinguish this view from the prior
+raw-record bridge. The portable od-jsontag branch remains separate work; this
+integration can preserve the accepted host parser dependency without publishing
+or depending on an uncommitted cross-repository change.
+
+Muze's replaceable-boundary and library-inspectability preferences favour one
+small engine adapter and an explicit data-view contract over a general sandbox
+framework. No engine selection option or VM2 fallback is needed.

.spiral/designs/DES-20260924-TTZ7C-59.ttl

@@ -0,0 +1,8 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+project:DES-20260924-TTZ7C-59 a sd:Design ;
+    dcterms:identifier "DES-20260924-TTZ7C-59" ;
+    sd:repositoryPath ".spiral/designs/DES-20260924-TTZ7C-59.md" ;
+    sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260924-TTZ7C-58 ; sd:gitCommit "fdef01a844e14f50c0af1aac6e89461714736f6d" ] ;
+    sd:status sd:Active .

.spiral/evidence/EVD-20260924-TTZ7C-60.md

@@ -0,0 +1,105 @@
+# Isolated query worker evaluation
+
+Implementation evaluated: bd507554a2fed6f77322ef38a98839f127d810db. This initial evaluation preceded human acceptance.
+The maintainer accepted the cycle on 2026-09-24; the cycle record retains that
+acceptance and the subsequently evaluated od-jsontag dependency update.
+
+## Verification
+
+202 tests pass, with no failures/skips, using Node 24.7.0 and isolated-vm 6.2.0:
+
+```sh
+node --no-node-snapshot --test \
+  test/required-integrity.mjs test/file-data.mjs test/index-ids.mjs \
+  test/store-runtime.mjs test/durability-startup.mjs \
+  test/conversion-offsets.mjs test/command-offsets.mjs \
+  test/query-isolate.mjs test/acid-baseline.mjs test/durability-crash.mjs \
+  test/after-change-handlers.mjs test/storage.mjs test/admin-recovery.mjs \
+  test/power-loss-baseline.mjs
+```
+
+The nine new query tests exercise real workers and cover:
+
+- JAQT, JSON/JSONTag, Unicode, typed URLs, ID identity/lookups, schema, browsing;
+- host access modules retaining Node imports, reflected property grants,
+  private parser symbols, readonly objects/arrays and metadata;
+- independently allowed existence versus denied reads, and the converse;
+- absent Node/network/shared-memory APIs, rejected static/dynamic/generated
+  imports, asynchronous result rejection and primitive throws;
+- query-global/prototype isolation, inherited toJSON/reference/tagged attacks;
+- execution/getter/toJSON timeouts, follow-up queries, UTF-8 result limits,
+  serializer tampering and bounded error messages;
+- isolate memory exhaustion followed by a successful query;
+- invalid limit configuration and outer-timeout worker replacement initialized
+  at the newly committed head.
+
+Existing tests retain command durability, integrity, publication ordering,
+updates/restart and caught versus forged storage-failure behavior. The power-loss
+baseline here is the existing modeled fault suite; no new QEMU/ext4 power-cut
+campaign was run for this engine change.
+
+Focused ESLint passes without warnings/errors. git diff --check passes. npm
+pack --dry-run includes all engine/view/guest source and the runtime guide.
+Staged provenance validation passed before implementation commits; range/tree
+validation is run again for the completed branch.
+
+## Example
+
+Opened a disposable copy of the current example data, command history, offsets,
+ID indexes and integrity manifest. Root collections, JSON projections and tagged
+person serialization succeeded with the example access module. A projection of
+name/gender returned names and omitted gender. typeof process/require/fetch all
+returned undefined. The original example data was not modified.
+
+## Actual worker measurements
+
+Reproduction: node --no-node-snapshot scripts/benchmark-query-runtime.mjs.
+Five measured samples after two warmups; sequential datasets and warm filesystem
+pages, Intel Core i7-1195G7, Node 24.7.0. All results match independent host
+expectations. Times include worker messaging, a fresh isolate/helper setup,
+host grant callbacks, lazy file reads and response serialization. They exclude
+store startup and are not physical-disk or total-RSS measurements.
+
+| Records | Query | Median ms |
+|---:|---|---:|
+| 1,000 | scan | 48.69 |
+| 1,000 | filter_sort_project | 52.85 |
+| 1,000 | serialize_1000 | 81.33 |
+| 10,000 | scan | 453.27 |
+| 10,000 | filter_sort_project | 497.90 |
+| 10,000 | serialize_1000 | 81.00 |
+
+Raw samples are retained in benchmark/query-engines/worker-results.json. This
+is a different data boundary from the initial raw-parser comparison: the
+production parser/grants stay on the host. The measured scan/query advantage
+remains substantial; do not treat either experiment as a universal speedup or
+public-access security certification.
+
+## Limits and next boundary
+
+Queries use 64 MiB isolate memory and 10 MiB result limits by default, configurable
+with positive validated bounds. Native memory limits are not whole-process RSS
+limits. Worker threads still share the server process; OS process isolation,
+broader adversarial review and concurrent deployment capacity remain outside
+this evaluated slice. No snapshots are used. Trusted storage/command code and
+the accepted parser revision remain unchanged in role; portable od-jsontag is
+separate work, not a silently published dependency.
+
+Source SRC-57, request REQ-58 and design DES-59 retain the rationale. The cycle
+continues the roadmap's sandbox replacement. Queryable object history and audit
+metadata remain the next target after this cycle's review/integration; UI is not
+required for that target.
+
+## Published dependency verification and acceptance
+
+At `bd8f8eb3ee617c5f9e9198f234725ef7ab5b4201`, SimplyStore pins published
+od-jsontag 0.5.0 and imports its Node parser adapter for file-descriptor input.
+The same full test command above passed 202 tests with no failures or skips
+against the installed npm release. Focused lint on src/file-data.mjs and
+git diff --check passed. A disposable copy of the current example store opened
+and returned root collections, permitted person fields and tagged values;
+process/require/fetch remained unavailable. Original example files were preserved.
+
+The maintainer explicitly accepted the cycle after this evaluation and requested
+version 0.11.0. Only package metadata and acceptance records change after this
+evaluated implementation; the existing regression result remains applicable.

.spiral/evidence/EVD-20260924-TTZ7C-60.ttl

@@ -0,0 +1,10 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+project:EVD-20260924-TTZ7C-60 a sd:VerificationEvidence ;
+    dcterms:identifier "EVD-20260924-TTZ7C-60" ;
+    sd:repositoryPath ".spiral/evidence/EVD-20260924-TTZ7C-60.md" ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "bd507554a2fed6f77322ef38a98839f127d810db" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:DES-20260924-TTZ7C-59 ; sd:gitCommit "1e2c7851a12ddc1b4c236d557a70479f411dfb8d" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "bd8f8eb3ee617c5f9e9198f234725ef7ab5b4201" ] ;
+    sd:status sd:Active .

.spiral/implementations/IMP-20260919-TTZ7C-33.md

@@ -214,3 +214,28 @@ The example exposes npm run init-integrity with a declarative store.json.
 Regression coverage and disposable-copy example verification preserve existing
 commands. Performance comparison uses the accepted optional baseline and identical
 30-command runs; canonical hashes are reused and index entries share one barrier.
+
+
+## Isolated query execution (CYC-56)
+
+VM2 is replaced by isolated-vm 6.2.0. Each query gets a fresh isolate and bundled
+JAQT/JSONTag helpers, with no module loading or Node/network APIs. A private
+copied-value bridge exposes read-only views while the established FileDataset,
+access module and storage-failure state remain on the trusted host. Record/path
+references preserve bounded parser caching without leaking parser-private
+symbols or isolated-vm References. This deliberately differs from the initial
+raw-record performance probe and does not require the uncommitted portable
+od-jsontag dependency branch.
+
+Execution and serialization share the configured deadline. The pool replaces
+queries stuck beyond the deadline plus a 250 ms response allowance. Memory and
+UTF-8 result/bridge bounds are explicit options; default values are 64 MiB and
+10 MiB. Typed scalar copies cannot mutate the dataset. Reflection applies host
+grants, with existence checks using the host has policy independently of the
+read/descriptor policy. Primitive throws are bounded errors, prototype changes stay within one
+query, and serialization getters/toJSON remain under the isolate timeout.
+
+Query tests, actual-worker performance measurement, runtime documentation and
+Node startup requirements are included. Native isolates remain inside Node
+worker threads; no OS-process memory/crash boundary or public safety certification
+is claimed. Trusted command/storage/integrity/recovery contracts remain effective.

.spiral/implementations/IMP-20260919-TTZ7C-33.ttl

@@ -32,8 +32,10 @@ project:IMP-20260919-TTZ7C-33
     sd:implementationLocation "test/index-ids.mjs" ;
     sd:implementationLocation "scripts/benchmark-file-data.mjs" ;
     sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-47 ; sd:gitCommit "83edb03bc406b9e651ef93bed69b87ece17dcf47" ] ;
-    sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:DEF-20260923-TTZ7C-50 ; sd:gitCommit "4bb10c2013fd2a7cb989b02586b992b936121362" ] ;
-    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ;
     sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260923-TTZ7C-54 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ;
-    sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:REQ-20260923-TTZ7C-53 ; sd:gitCommit "c8a3fd9464abfd5e7b699a2752789d82af4da2da" ] ;
+    sd:implements [ a sd:ArtifactReference ; sd:artifact project:DES-20260924-TTZ7C-59 ; sd:gitCommit "1e2c7851a12ddc1b4c236d557a70479f411dfb8d" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "2595efb946993f1b088a430c46dd04927fa27868" ] ;
+    sd:changeCausedBy [ a sd:ArtifactReference ; sd:artifact project:REQ-20260924-TTZ7C-58 ; sd:gitCommit "fdef01a844e14f50c0af1aac6e89461714736f6d" ] ;
+    sd:shapedBy [ a sd:ArtifactReference ; sd:artifact <https://muze.nl/ns/spiral-developer/culture/CUL-MUZE-001> ; sd:gitCommit "8d4b2c738a413abd4cccca740ce958f486e5f7af" ] ;
+    sd:implementationLocation "src/query-isolate.mjs", "src/query-view.mjs", "src/query-guest.mjs", "test/query-isolate.mjs", "scripts/benchmark-query-runtime.mjs" ;
     sd:implementationChangeKind sd:SemanticChange .