Spiral/cyc 20260926 ttz7 c 63 schema identity

.spiral/cycles/CYC-20260926-TTZ7C-63.md

@@ -0,0 +1,91 @@
+# Schema identity across the query isolate
+
+Goal: restore object identity for non-record objects across the query isolate
+boundary, so `meta.schema` queries work and shared schema objects keep their
+JSONTag links. Correct DEF-20260926-TTZ7C-62.
+
+Status: Accepted on 2026-09-26. Human commitment: the maintainer confirmed the understanding
+and defect cycle on 2026-09-26 and set the identity direction retained in
+SRC-20260926-TTZ7C-61: each object gets a stable identifier that outlives a
+single query and is not exposed to query code.
+
+Classification: deliberate deviation from ROADMAP.md. Object history remains
+the next roadmap target. This defect is in accepted, integrated work from
+CYC-20260924-TTZ7C-56 and blocks curriculum-store's SimplyStore 0.11 upgrade.
+
+Branch: spiral/CYC-20260926-TTZ7C-63-schema-identity, created from
+authoritative master bd61d9c. Branch identity checked when opening the cycle.
+
+In scope: the identity rule in DES-20260924-TTZ7C-59, its implementation in the
+query view/guest bridge, a regression with a shared and cyclic schema, and
+measurement of the curriculum-store schema through the real runtime. Guest-side
+read caching is in scope only if measurements show identity alone leaves the
+query outside its limits.
+
+Scope refinement during Act: with stable handles, curriculum-store's full
+schema returns as JSON in 2.7 s, above the 1 s default timeout, while the host
+serializes it in 11 ms. Guest-side read caching is therefore in scope. JSONTag
+responses failed because the schema reaches workers without its JSONTag ids;
+per the maintainer's decision in SRC-20260926-TTZ7C-61, query workers receive
+and parse the schema's original JSONTag text.
+
+Non-goals: object history, schema editing, changes to data record identity,
+query limits or grants, curriculum-store's own Spiral repair and unrelated
+cleanup.
+
+## Goal extension
+
+On 2026-09-26, after evaluation-ready evidence for the schema, the maintainer
+extended the goal to object identity for all query data (SRC-20260926-TTZ7C-61):
+JSONTag responses must link shared objects without an `id`, using ids derived
+from record numbers for stored objects and per-response ids for objects a query
+builds; tagged scalars keep identity within a query. Shared arrays and the
+jsontag fallback bug remain out of scope. The cycle is back in Act; the
+evaluation-ready section below describes the earlier schema-only state.
+
+## Evaluation ready
+
+The goal can be judged. Schema objects keep identity across the isolate by
+stable private handles, JSONTag responses keep the schema file's ids and links,
+and guest views read each host property once per query. EVD-20260926-TTZ7C-64
+records 205 passing tests, failing baselines, and curriculum-store's full schema
+returning in 34 ms as JSONTag and 271 ms as JSON through the actual runtime,
+with record-query benchmarks unchanged within variation.
+
+Retained for next-cycle planning, not absorbed: command workers still receive
+the schema without JSONTag ids; guest JSONTag serialization of query-built
+shared objects without ids needs `crypto`, which the isolate lacks;
+curriculum-store's own Spiral state needs repair; nine older non-implementation
+artifacts use `sd:changeCausedBy`, which the ontology reserves for
+implementations, and the validators do not flag it. Human evaluation,
+acceptance and integration remain separate decisions.
+
+A full causal-chain review before acceptance corrected this cycle's own use of
+that relation on DES-20260924-TTZ7C-59 (now `sd:constrainedBy`), clarified the
+EVD-20260926-TTZ7C-64 timing column, added SRC-20260926-TTZ7C-61 as a cause of
+the IMP-20260919-TTZ7C-33 transition, and refreshed the stale project context.
+
+## Evaluation ready (extended goal)
+
+The extended goal can be judged. Stored records keep identity across paths and
+cycles; JSONTag responses link shared objects without an `id` as `~<record>`,
+stable across queries and updates, or `~query-<n>` for query-built objects,
+without exposing those ids to queries; tagged scalars keep identity within a
+query. EVD-20260926-TTZ7C-64 records the regressions, 207 passing tests and
+about 4% added cost on large scans. Shared-array identity and the jsontag
+fallback bug remain deferred, with the earlier retained discoveries.
+
+## Human acceptance
+
+On 2026-09-26, after reviewing the extended evaluation, the maintainer
+instructed: "Accepted."
+
+This accepts the cycle at fa6c791991381beb5fac40103dbc9dba15233ee8: stable
+schema handles, schema JSONTag text for query workers, per-query read caching,
+record-derived and per-response output ids, and per-query tagged scalar
+identity, with the evidence and limits in EVD-20260926-TTZ7C-64. Deferred:
+shared-array identity, the jsontag `createId` fallback bug, schema ids in
+command workers, the nine older `sd:changeCausedBy` uses and curriculum-store's
+Spiral repair. No version change was requested. Acceptance authorizes proposing
+integration; merging remains a separate decision. Branch identity was verified
+before recording acceptance.

.spiral/cycles/CYC-20260926-TTZ7C-63.ttl

@@ -0,0 +1,11 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:CYC-20260926-TTZ7C-63 a sd:Cycle ;
+    dcterms:identifier "CYC-20260926-TTZ7C-63" ;
+    sd:repositoryPath ".spiral/cycles/CYC-20260926-TTZ7C-63.md" ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:SRC-20260926-TTZ7C-61 ; sd:gitCommit "bcf64e82a3a4290a1079d1029ec6bd86c67734bb" ] ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:DEF-20260926-TTZ7C-62 ; sd:gitCommit "d34a97f400e5708d0a7aa23d4f601f3ab51ae7d6" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:CYC-20260926-TTZ7C-63 ; sd:gitCommit "fa6c791991381beb5fac40103dbc9dba15233ee8" ] ;
+    sd:status sd:Accepted .

.spiral/defects/DEF-20260926-TTZ7C-62.md

@@ -0,0 +1,37 @@
+# Schema objects lose identity across the query isolate
+
+Observed on master bd61d9c with curriculum-store's 92 KB `schema.jsontag`.
+A query for `meta.schema` fails in both JSON and JSONTag modes. With default
+limits it exhausted the 64 MiB isolate heap after about 2.5 s; the maintainer
+observed the 10 s slow-query timeout. Smaller contexts succeed, but their
+JSONTag responses expand shared objects instead of linking them: the
+`kerndoelen` context returns 28,227 bytes where host serialization returns
+6,957.
+
+The schema is acyclic. It contains 281 distinct objects and 734 shared
+references. Treating every path as a distinct object expands it to 99,001
+objects. Host-side, the whole schema serializes to 49,844 bytes of JSONTag in
+about 1 ms (4,012,890 bytes of JSON).
+
+Cause: DES-20260924-TTZ7C-59 identifies query objects by "record numbers plus
+property paths (or schema paths)". A record number is a stable identity, and
+od-jsontag stores every data object as its own record, so data arrays have one
+owning path. Schema objects are ordinary JSONTag objects that may be reached
+by many paths. The guest caches proxies by reference, so one schema object
+becomes a distinct proxy per path: `===` fails, the JSONTag serializer cannot
+emit links, and each path repeats its host reads.
+
+Earliest meaningful cause: the design assumed path uniqueness for schema
+objects. Verification did not catch it because the query test schema is a
+tree. The correction belongs in the design's identity rule, followed by the
+implementation and a regression with a shared/cyclic schema.
+
+## Correction
+
+DES-20260924-TTZ7C-59 was corrected at 07c6d65 and d881fe1; implementation
+be90dd4 and test 8638f1f realize it. Schema objects are identified by stable
+private handles, query workers parse the schema's JSONTag text so ids survive
+(the maintainer's decision in SRC-20260926-TTZ7C-61), and guest views cache
+host reads per query. EVD-20260926-TTZ7C-64 records the regressions, 205
+passing tests and curriculum-store's schema returning in 34 ms (JSONTag) and
+271 ms (JSON).

.spiral/defects/DEF-20260926-TTZ7C-62.ttl

@@ -0,0 +1,14 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:DEF-20260926-TTZ7C-62
+    a sd:Defect ;
+    dcterms:identifier "DEF-20260926-TTZ7C-62" ;
+    sd:repositoryPath ".spiral/defects/DEF-20260926-TTZ7C-62.md" ;
+    sd:status sd:Active ;
+    sd:provenanceConfidence sd:Evidenced ;
+    sd:derivedFrom [ a sd:ArtifactReference ; sd:artifact project:SRC-20260926-TTZ7C-61 ; sd:gitCommit "bcf64e82a3a4290a1079d1029ec6bd86c67734bb" ] ;
+    sd:observes [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "bd61d9cf612008e96a518fa30fca9e33c40fa82e" ] ;
+    sd:observes [ a sd:ArtifactReference ; sd:artifact project:DES-20260924-TTZ7C-59 ; sd:gitCommit "bd61d9cf612008e96a518fa30fca9e33c40fa82e" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DEF-20260926-TTZ7C-62 ; sd:gitCommit "d34a97f400e5708d0a7aa23d4f601f3ab51ae7d6" ] .

.spiral/designs/DES-20260924-TTZ7C-59.md

@@ -15,10 +15,43 @@ and private synchronous callbacks. A disposable prototype scanned 10,000 records
 in about 362–430 ms through this view; the chosen boundary still warrants real
 integration measurements.
 
-References identify record numbers plus property paths (or schema paths), not
-host objects or paths on disk. Resolve them through the host parser for every
+References identify stable host identities, not host objects or paths on disk.
+A data object reuses its record number, and a data array its record number plus
+property path: od-jsontag stores every object as a record, so an array has one
+owning path. Schema objects and arrays may be shared or cyclic, so a path does
+not identify them. Number them once per parsed schema by a deterministic walk
+in property order; the same schema yields the same handles in every worker and
+query. Handles remain private to the bridge and add no query-visible property,
+symbol or value. Existing JSONTag `id` attributes are not reused because not
+every schema object has one. Resolve references through the host for every
 operation so bounded record caches remain effective. Preserve guest identity
-with weak references. Return scalars, tagged scalar copies and descriptions of
+with weak references keyed by these references, so one host object is one
+guest object and JSONTag serialization retains links.
+
+Structured cloning keeps shared objects but drops JSONTag metadata, including
+the schema's `id`s that JSONTag links name. The runtime therefore passes the
+loaded schema's original JSONTag text to each query worker, which parses it
+once at initialization. Query responses use the schema file's ids.
+
+Data cannot change during a query: updates are serialized with queries and
+each query has a fresh isolate. A guest proxy therefore caches its keys and
+property reads for the query's lifetime, so repeatedly traversing shared
+objects does not repeat host calls. Grants still apply on the first read. The
+cache uses built-ins captured before query code runs. A tagged scalar is copied
+once per view property and query, so repeated reads return the same object,
+as they do on the host. Child views are not retained by that cache, so scans
+do not pin every visited record.
+
+JSONTag links a shared object by its `id`, and generates and writes one when
+the object has none. Views are read-only and the isolate has no `crypto`. The
+serializer therefore first finds, by the same traversal JSONTag uses, the
+objects that will be linked but have no `id`. A view receives an output id
+derived from its stable identity: `~<record>` for records, as od-jsontag writes
+references, or `~schema-<handle>`. A query-built object receives
+`~query-<n>`, numbered per response. A derived id that is already a stored
+`id` gains a numeric suffix. View ids exist only while serializing, through
+the attribute read JSONTag performs; query code cannot observe them and the
+host is never asked to write. Return scalars, tagged scalar copies and descriptions of
 object/array views. Expose only ordinary property names and JSONTag's public
 metadata symbols. Do not forward parser-private symbols, prototypes, methods,
 raw buffers, source handles or isolated-vm References. Mutating the view throws;

.spiral/designs/DES-20260924-TTZ7C-59.ttl

@@ -5,4 +5,7 @@ project:DES-20260924-TTZ7C-59 a sd:Design ;
     dcterms:identifier "DES-20260924-TTZ7C-59" ;
     sd:repositoryPath ".spiral/designs/DES-20260924-TTZ7C-59.md" ;
     sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260924-TTZ7C-58 ; sd:gitCommit "fdef01a844e14f50c0af1aac6e89461714736f6d" ] ;
-    sd:status sd:Active .
+    sd:status sd:Active ;
+    sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:DEF-20260926-TTZ7C-62 ; sd:gitCommit "d34a97f400e5708d0a7aa23d4f601f3ab51ae7d6" ] ;
+    sd:constrainedBy [ a sd:ArtifactReference ; sd:artifact project:SRC-20260926-TTZ7C-61 ; sd:gitCommit "269a729823d135f48e4570e6e12af1e818bfcaf8" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DES-20260924-TTZ7C-59 ; sd:gitCommit "eb3a13c5aff090d5b6fd23528b79fadef03b5332" ] .

.spiral/evidence/EVD-20260926-TTZ7C-64.md

@@ -0,0 +1,99 @@
+# Schema identity across the query isolate
+
+Verifies the DEF-20260926-TTZ7C-62 correction at 8638f1f (implementation
+be90dd4, design d881fe1). Node 24.7.0, isolated-vm 6.2.0, Intel Core i7-1195G7.
+
+## Regressions
+
+Three tests in `test/query-isolate.mjs` cover the defect:
+
+- shared, linked and cyclic schema objects compare `===` across paths; the
+  guest sees the same own keys as the host, including JSONTag attributes;
+  a JSONTag response of `meta.schema` equals host `JSONTag.stringify`; a JSON
+  response of the cyclic schema fails promptly as circular, like host JSON;
+- a schema with 26 objects and 2^25 paths returns as JSONTag, equal to host
+  serialization, within the 1 s default timeout;
+- schema handles are equal for shared objects, across views and across
+  separately parsed copies of the schema, so identity outlives one query.
+
+Baseline commit 49f3573 recorded the first two failing on the old
+implementation (identity false; "Query execution timed out"). Restoring the
+pre-fix `src/` at the final tree makes all three fail; the fixed tree passes.
+
+## Full suite
+
+The EVD-20260924-TTZ7C-60 command, all 14 test files, passes 205 tests with
+no failures or skips (202 there, plus these three). Focused ESLint on the
+changed query files passes. `git diff --check` passes for every commit.
+
+## curriculum-store schema
+
+Actual runtime path, slow pool, curriculum-store's 92 KB `schema.jsontag`
+(281 objects, 734 shared references):
+
+| Query | Before | After identity and schema ids | After caching |
+|---|---|---|---|
+| `meta.schema`, JSONTag | 64 MiB exhausted, 2.4 s | 86 ms | 34 ms, 49,844 bytes |
+| `meta.schema`, JSON | 64 MiB exhausted, 2.6 s | 2,806 ms | 271 ms, 4,012,890 bytes |
+| `kerndoelen`, JSONTag | 28,227 bytes (expanded) | 6,957 bytes | 11 ms |
+| `referentiekader`, JSON | 442 ms | 267 ms | 33 ms |
+
+Response sizes equal host serialization of the same schema. The JSON response
+is large because JSON cannot express sharing; that is not a query-engine cost.
+Measurements used a disposable store with the copied schema file. With stable
+handles alone, JSONTag responses failed with "crypto is not defined" until
+query workers received the schema's JSONTag text.
+
+## Record queries
+
+`scripts/benchmark-query-runtime.mjs` medians (ms) compared with
+EVD-20260924-TTZ7C-60: 1,000 scan 50.7 (48.7), filter/sort/project 49.8
+(52.9), serialize 84.9 (81.3); 10,000 scan 483.6 (453.3), filter/sort/project
+467.6 (497.9), serialize 86.2 (81.0). Differences are within run-to-run
+variation; read caching neither helps nor harms single-pass scans.
+
+## Limits
+
+Command workers still receive the structured-cloned schema without JSONTag
+ids; this cycle changed only query workers. Guest JSONTag serialization of
+query-constructed shared objects without ids still needs `crypto`, which the
+isolate lacks. That is retained as a discovery, not corrected here. No
+performance claim is made beyond these warm-cache measurements.
+
+## Identity for all query data
+
+Verifies the extension at 92a0773 (design 832afc1).
+
+A runtime probe of stored data before the extension showed records already
+keep identity across paths and cycles, and a 26-record DAG with 2^25 paths and
+ids returns as JSONTag in about 110 ms, equal to host serialization. It also
+showed the gaps now corrected: shared objects without an `id` failed as JSONTag
+("crypto is not defined"; with a query-supplied crypto, "dataspace is
+immutable"), and tagged scalars were distinct copies per read.
+
+Two regressions cover the correction and failed at baseline 083542c:
+
+- shared stored objects without ids link as `~<record>` in JSONTag responses;
+  a record whose derived id matches a stored `id` (`~1`) receives `~1-1`;
+  unshared objects receive no id; repeated responses are identical; queries
+  cannot read output ids; query-built shared objects receive `~query-<n>`,
+  and mixed query-built/stored results link both;
+- tagged scalars are identical within a query, and a change to the copy does
+  not reach the next query.
+
+A command probe showed output ids stable across updates: a shared record kept
+`~1` through two modifying commands, and an object shared by a command received
+`~2` and stayed linked. The 2^25-path DAG without ids now returns in 7 ms.
+Shared arrays still return as separate arrays, as od-jsontag stores them
+inline; that is deferred by the maintainer.
+
+The full 14-file suite passes 207 tests. One earlier run failed the
+timing-based "hanging command times out unsafe" command test; it passed three
+isolated reruns and the next full run, and exercises no query code.
+
+Benchmark, three interleaved rounds on the same machine against 083542c
+(medians, ms): 10,000 scan 510 → 534, filter/sort/project 495 → 514,
+serialize 89 → 92; 1,000-record results are within variation. The machine ran
+slower than in the earlier measurements above. Tracking each view's identity
+costs about 4% on large scans. curriculum-store's schema remains 38 ms as
+JSONTag and 302 ms as JSON.

.spiral/evidence/EVD-20260926-TTZ7C-64.ttl

@@ -0,0 +1,13 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:EVD-20260926-TTZ7C-64 a sd:VerificationEvidence ;
+    dcterms:identifier "EVD-20260926-TTZ7C-64" ;
+    sd:repositoryPath ".spiral/evidence/EVD-20260926-TTZ7C-64.md" ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "8638f1f67f9a1b6509117be7c016ff61afd98c6d" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:DES-20260924-TTZ7C-59 ; sd:gitCommit "d881fe1e1e81450182b64d4e35ebf4d4bb648860" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:IMP-20260919-TTZ7C-33 ; sd:gitCommit "92a077373e5a983eab24a64feb475ecb33603850" ] ;
+    sd:verifies [ a sd:ArtifactReference ; sd:artifact project:DES-20260924-TTZ7C-59 ; sd:gitCommit "832afc19533e730013c43491a23d8cb41bba97f8" ] ;
+    sd:transforms [ a sd:ArtifactReference ; sd:artifact project:EVD-20260926-TTZ7C-64 ; sd:gitCommit "eb3a13c5aff090d5b6fd23528b79fadef03b5332" ] ;
+    sd:status sd:Active .