Spiral/cyc 20260819 09 zef 5 integrity detection

.github/workflows/spiral-integration.yml

@@ -27,6 +27,17 @@ jobs:
 
       - run: pip install -r .spiral-core/requirements.txt
 
+      - name: Test provenance validation
+        run: python test/spiral-provenance.py
+
+      - name: Validate every introduced artifact version and the candidate snapshot
+        env:
+          SPIRAL_BASE: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
+          SPIRAL_HEAD: ${{ github.event.pull_request.head.sha || github.sha }}
+        run: |
+          python scripts/validate-spiral-provenance.py --range "$SPIRAL_BASE..$SPIRAL_HEAD"
+          python scripts/validate-spiral-provenance.py --tree "$SPIRAL_HEAD"
+
       - if: github.event_name == 'pull_request'
         run: >-
           node .spiral-core/bin/spiral.mjs validate integration

.spiral/cycles/CYC-20260819-09ZEF-5.md

@@ -0,0 +1,222 @@
+---
+id: CYC-20260819-09ZEF-5
+---
+
+# Cycle: Integrity Detection
+
+Repository branch: `spiral/CYC-20260819-09ZEF-5-integrity-detection`
+Branch verified: verified on 2026-08-19 with `git branch --show-current`, branched from `master` after accepted CYC-018 was merged.
+
+## Analyze
+
+Project state / prior evaluation that makes this cycle relevant:
+
+SimplyStore has completed the Spiral process update cycle and returns to the durability production-readiness roadmap. Recent durability cycles established executable crash/replay behavior, retry/idempotency semantics, OD-JSONTag framing failure behavior, after-change characterization, and command/load timeout behavior. The human has especially called out concern about corrupted or altered OD-JSONTag files.
+
+Nearest important risk / uncertainty / desired movement:
+
+The remaining durability claim is still too weak against well-formed but altered durable data. Current evidence covers malformed or truncated OD-JSONTag record framing, not intentional or accidental content alteration that still parses. A developer evaluating SimplyStore needs to know whether altered durable artifacts are detected, ignored, recovered, or silently trusted.
+
+Relevant human direction / feedback:
+
+The human agreed that the next cycle should continue the durability roadmap toward integrity/tamper detection while preserving simplicity and avoiding silent API or on-disk format changes.
+
+Governing higher-level plan / direction:
+
+- `SRC-001` at `f2c3a597c12c2dbe6d7467582d5a3f4308acf02d` gives the ordered durability/extensibility direction.
+- `REQ-001` at `6a9977eb22b0a610a82da5c52864e3bb72b164c9` requests explicit, mechanically testable durability evidence.
+- `DES-001` at `85913e982e163c7b05a8e616c00e3f2014e775cc` defines D6 detectable corruption and places integrity-root work after crash/replay, adversarial storage, idempotency, and handler-contract evidence.
+
+Current position in that plan:
+
+Continue. The project has already covered crash/fault-injection, reconstruction/idempotency slices, adversarial malformed-framing cases, and handler/timeout characterization. The next coherent risk is integrity checking for altered durable data, with care not to overclaim broader repair or production safety.
+
+## Plan
+
+### Cycle goal
+
+Assess and implement the first simple integrity-detection behavior for durable OD-JSONTag data so altered committed storage fails explicitly instead of being silently used, while preserving SimplyStore's simplicity and avoiding unrelated runtime/API changes.
+
+### Why now / why this cycle boundary
+
+This is the next durability-roadmap uncertainty with high downstream leverage: integrity metadata choices affect on-disk compatibility, recovery behavior, developer-facing claims, and future extension/finalizer design. A focused cycle can decide and test the first narrow integrity behavior without turning SimplyStore into a broader storage framework.
+
+### Plan continuity decision
+
+Continue the original durability plan. The cycle pulls forward the integrity/tamper-detection concern because malformed framing has already been characterized and the human explicitly identified corrupted or altered OD-JSONTag files as a concern.
+
+### Current starting evidence
+
+- Malformed/truncated OD-JSONTag record framing already fails explicitly in covered cases.
+- The project context records that full syntax validation and well-formed tampering need future integrity metadata.
+- The current public API, JavaScript query API, and durable on-disk format must not silently change.
+
+### Evaluation basis
+
+- Current effective behavior for well-formed altered durable data is inspected and stated before runtime behavior changes.
+- The cycle records the Understanding/evidenced-gap checkpoint before consequential product modification.
+- Tests demonstrate the chosen integrity behavior on at least one committed durable-data alteration path.
+- Any on-disk format implication is explicit, bounded, and documented as either compatible, transitional, or intentionally not yet enabled.
+- Existing durability tests continue to pass.
+
+### Likely work
+
+- Inspect current durable file layout, OD-JSONTag load path, and existing adversarial storage tests.
+- Run a small probe or characterization test for well-formed altered committed data.
+- Present the Spiral checkpoint: My understanding / Current effective behavior / Evidenced gap / Material assumptions.
+- If confirmed, design the smallest integrity mechanism and tests that can detect altered committed storage without broad framework machinery.
+- Implement only the committed slice and add verification evidence.
+
+### Explicit non-goals
+
+- Automatic repair of corrupted data.
+- A Merkle tree, streaming integrity framework, database WAL, event-sourcing framework, message bus, or schema registry.
+- Broad REST API or JavaScript query API changes.
+- Silent durable format changes.
+- Solving VM2/query sandbox security.
+- Completing every possible corruption/tamper case in one cycle.
+
+### Pause / re-plan conditions
+
+- Current behavior already detects the targeted alteration class.
+- A safe first integrity mechanism requires a breaking on-disk format change that has not been explicitly accepted.
+- Investigation shows integrity checking depends on unresolved handler/commit-order semantics outside this cycle.
+- The smallest viable design conflicts with SimplyStore's simplicity or public compatibility commitments.
+
+## Act
+
+Important artifacts / semantic commits produced:
+
+- `EVD-20260819-09ZEF-6` records the baseline probe showing that same-length payload alteration can preserve OD-JSONTag framing and load as ordinary altered data.
+- `DES-20260819-09ZEF-7` defines an optional durable file integrity manifest as the first narrow integrity-detection mechanism.
+- `IMP-20260819-09ZEF-8` implements the optional integrity manifest, startup verification, command changeset digest recording, and focused tests.
+- `EVD-20260819-09ZEF-9` verifies the implementation with startup, full-server durability, ACID, handler, and touched-file lint checks.
+
+Material implementation decisions or deviations from the initial likely work:
+
+- The first step was kept to behavior characterization and evidence until the Spiral Understanding/evidenced-gap checkpoint was presented and confirmed.
+- The implemented integrity behavior is opt-in/manifest-driven so existing datasets without sidecar integrity metadata remain loadable by default.
+- The first integrity scope is deliberately limited to base data and committed changesets; command log/status and manifest self-protection remain out of scope.
+
+Out-of-scope discoveries retained for later:
+
+- Whole-repo ESLint still reports unrelated legacy errors in `src/index.id.mjs`, `src/index.offset.mjs`, `src/produce.mjs`, and `src/triplestore.mjs`.
+- The integrity manifest itself is not yet protected against coordinated data+manifest rewrite.
+- Command log/status integrity remains outside this first durable-data slice.
+
+## Evaluate
+
+Integrated result against cycle goal:
+
+Implemented and verified the first narrow integrity-detection behavior for durable OD-JSONTag data. Integrity-enabled stores now verify base data and committed changesets against an append-readable SHA-256 sidecar manifest before parsing. If a covered file's bytes change, startup fails explicitly instead of silently loading altered state. Existing stores without integrity metadata remain loadable by default.
+
+Evidence / acceptance result:
+
+Verification evidence is recorded in `EVD-20260819-09ZEF-9`.
+
+Automated checks:
+
+- `npm test` passed: 19/19 startup durability tests.
+- `npm run test:durability` passed: 16/16 full-server durability tests.
+- `npm run test:acid` passed: 5/5 ACID baseline tests.
+- `npm run test:handlers` passed: 3/3 handler characterization tests.
+- Touched-file ESLint passed.
+- Whole-repo ESLint still fails on unrelated legacy files.
+
+The human accepted the cycle on 2026-08-19.
+
+Metric or risk movement:
+
+The D6 detectable-corruption claim is stronger for manifest-covered base data and committed changesets. A same-length payload alteration that previously loaded silently is now detected when integrity metadata exists. This is still not a full tamper-resistance claim because coordinated data+manifest rewrite remains possible.
+
+What changed in our understanding:
+
+The smallest useful integrity mechanism can be compatible and opt-in: a sidecar manifest can prove altered-byte detection without immediately migrating all existing datasets or changing REST/query APIs.
+
+Surprises / model mismatches:
+
+No major design mismatch appeared. A full-server test was needed to verify the digest is written before the existing `done` commit boundary, because unit startup tests alone could not prove the command lifecycle ordering.
+
+Known compromises:
+
+- Integrity is not default-required for legacy datasets.
+- The manifest is append-readable and latest-wins, but is not chained, signed, or protected from coordinated rewrite.
+- Command log/status integrity remains outside this cycle.
+- Existing datasets need an explicit manifest entry before integrity-required startup succeeds.
+
+Unresolved issues within current goal:
+
+No known unresolved issue blocks evaluation of the agreed first narrow integrity slice.
+
+Candidate next-cycle inputs:
+
+- Decide whether to add a simple script to initialize/rebuild integrity manifests for existing datasets.
+- Consider extending integrity coverage to command log/status files.
+- Consider documenting the durability envelope in a developer-facing `DURABILITY.md` once enough slices have accumulated.
+- Retain the earlier reset/clean dataset script idea for dirty examples and local development stores.
+
+Human evaluation / feedback:
+
+Accepted by the human on 2026-08-19 after reviewing the implementation summary of the integrity check.
+
+Cycle accepted, still open, or deliberately re-planned:
+
+Accepted. Ready for prospective integration validation against the current `master`.
+
+## Process learning
+
+What context/constraint/evaluation helped:
+
+The explicit compatibility constraint was useful: making integrity manifest-driven avoided silently changing the durable format contract for existing datasets while still proving the targeted altered-data case.
+
+What bookkeeping was useless:
+
+None identified.
+
+What should the environment learn from this cycle:
+
+For SimplyStore durability work, pair direct startup/unit checks with at least one full-server lifecycle test when the claim depends on command commit ordering.
+
+## Compliance correction — 2026-09-16
+
+Reopened on the same branch at the maintainer's explicit request to correct the
+compliance audit findings (`DEF-20260916-TTZ7C-1`). The original acceptance above
+remains historical evidence for the runtime integrity slice. The cycle was reopened as Active for this bounded process correction; the
+correction was subsequently accepted as recorded below.
+
+Branch checked on reopening: `spiral/CYC-20260819-09ZEF-5-integrity-detection`.
+Scope: repair evidence references and governed implementation lineage, replace
+partial parsing with standards-based validation, add staged/range prevention,
+and reconcile context. No SimplyStore runtime changes are authorized by this
+correction. Governing durability direction remains unchanged.
+
+### Correction evaluation — 2026-09-16
+
+The four audit findings have concrete corrections in new commits:
+
+- `91f5d09` repairs the seven references while preserving historical test claims
+  and records the defect/reopened scope.
+- `f66ea74` defines standards-based reference validation and lineage repair.
+- `adccdd1` implements the validator, 16 regression tests, staged hook, CI range
+  checks, current provenance/lineage projections, and operator guidance.
+
+`EVD-20260916-TTZ7C-3` records verification. The current graph and introduced
+history pass. Checking the original snapshot still reports all seven missing
+artifact versions. No runtime source or dependency changed.
+
+At evaluation, acceptance was pending and the integration validator correctly
+blocked the Active cycle. A disposable hypothetical Accepted candidate passed
+the other integration constraints. The actual acceptance follows below.
+
+### Correction acceptance — 2026-09-16
+
+After reviewing the correction summary and the explanation of the integration
+gate, the maintainer replied "Agreed" to the explicit question asking whether
+they accepted the corrections. This accepts the completed compliance correction
+at `6f71005388b1e393eb1a6b30059e6e096c751bb2`, with verification in
+`EVD-20260916-TTZ7C-3`. The original runtime acceptance remains unchanged.
+
+Current cycle status: **Accepted and closed**. The cycle is ready for integration
+validation against the current target; acceptance does not waive that check.
+Historical invalid references remain preserved and reported by history audits.
+No runtime code or dependencies changed as part of this acceptance.

.spiral/cycles/CYC-20260819-09ZEF-5.ttl

@@ -0,0 +1,9 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:CYC-20260819-09ZEF-5
+    a sd:Cycle ;
+    dcterms:identifier "CYC-20260819-09ZEF-5" ;
+    sd:repositoryPath ".spiral/cycles/CYC-20260819-09ZEF-5.md" ;
+    sd:status sd:Accepted .

.spiral/defects/DEF-20260916-TTZ7C-1.md

@@ -0,0 +1,68 @@
+---
+id: DEF-20260916-TTZ7C-1
+---
+
+# Defect: Incomplete Spiral Provenance Validation
+
+## Source and authorization
+
+On 2026-09-16 the maintainer requested a compliance audit of the checked-out
+SimplyStore integrity branch, then responded "Lets try to correct this" and
+"Please continue". This is an explicit instruction to correct the reported
+process defects. The connected conversation is the primary source; this record
+retains the operative wording rather than claiming to retain a full transcript.
+
+## Understanding and evidenced gap
+
+Correct the current causal records and prevent recurrence without changing
+SimplyStore runtime behavior or rewriting Git history. At `fa329453bf640fab6083e39d57edfdbde222e282`:
+
+- Seven evidence references name implementation artifacts before their creation.
+- The integrity slice adds a pre-commit prerequisite and loader verification to
+  governed paths without recording their predecessor lineage.
+- The local partial-Turtle validator fails on vocabulary.ttl, and the core
+  snapshot/integration validator does not check historical target existence or
+  every introduced artifact version. Both official checks nevertheless passed.
+- Context calls the accepted integrity cycle active.
+
+The earliest actionable cause is insufficient validation, not a runtime defect.
+
+## Exact reference repairs
+
+| Evidence | Target | Invalid historical target | First retained artifact version |
+|---|---|---|---|
+| EVD-006 | IMP-002 | `da55e78ec53465e1b02aaac1bf69c25c409ed89d` | `676fa5c6038f92219e902ef126f284360c1763d6` |
+| EVD-007 | IMP-003 | `0fa4c441e8e8cb27562486c225bb02e8038d564e` | `c13d08a9e442e30ca4d66f28597939f55ef0a513` |
+| EVD-009 | IMP-004 | `29f016cdb3ec6c3cc03ff9acebac26691a095326` | `36bb7b1013c2de219d4a730556ae9ab98cea7113` |
+| EVD-010 | IMP-005 | `e2574dd0df70b1803c61f416c537a7f15376b92d` | `08f53c1d25468e97dbc517b3a6bcd7fb98bf4e5e` |
+| EVD-011 | IMP-006 | `7d514568349d6a14ac494155549579533af68bf4` | `2eeecded67ac99c7ab757113fc8f299af1deafc2` |
+| EVD-012 | IMP-007 | `570d35a4fa47db5edf4784c998ee441ce4f20821` | `28034073cf15170e367fe1bf9ca9846c299d23ea` |
+| EVD-014 | IMP-009 | `bc376bfc353fe2489ba61a99ca0dc47b38fc1214` | `313b5b71016067388c542120712082d6023d87d4` |
+
+For each row, `git diff --name-only <old> <replacement>` shows only `.spiral/`
+changes. The recorded product implementation and historical test claims are
+therefore retained. Corrected references do not retroactively validate the old
+artifact versions. Raw history audits must continue to report those violations.
+
+## Correction boundary
+
+Continue the existing integrity branch as a bounded correction of its review
+readiness. Reopen its cycle for process correction; preserve the original
+2026-08-19 acceptance as acceptance of the runtime slice, not advance approval
+of these corrections. No new product cycle or roadmap direction is selected.
+
+Record lineage prospectively for changed governed paths; do not assign every
+shared source file to one owner. New integrity helpers remain a new concern.
+Replace the partial parser with rdflib, add staged and introduced-range checks,
+and keep core integration validation as a separate check. No historical
+allowlist, amended commits, production changes, or new runtime dependencies.
+
+## Evaluation basis
+
+- Current graph references resolve to existing artifact versions at admissible commits.
+- Repaired evidence remains honest about when artifacts and tests existed.
+- Changed governed concerns preserve effective causes and predecessor lineage.
+- Regression tests reject malformed, dangling, non-ancestor and transient bad references.
+- CI checks both the introduced range and the prospective result.
+- Raw history checks still fail on known old violations.
+- Human evaluation of the correction remains pending until explicitly received.

.spiral/defects/DEF-20260916-TTZ7C-1.ttl

@@ -0,0 +1,15 @@
+@prefix sd: <https://muze.nl/ns/spiral-developer#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix project: <https://github.com/simplyedit/simplystore/spiral#> .
+
+project:DEF-20260916-TTZ7C-1
+    a sd:Defect ;
+    dcterms:identifier "DEF-20260916-TTZ7C-1" ;
+    sd:repositoryPath ".spiral/defects/DEF-20260916-TTZ7C-1.md" ;
+    sd:status sd:Active ;
+    sd:provenanceConfidence sd:Evidenced ;
+    sd:observes [
+        a sd:ArtifactReference ;
+        sd:artifact project:CYC-20260819-09ZEF-5 ;
+        sd:gitCommit "fa329453bf640fab6083e39d57edfdbde222e282"
+    ] .