diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-42.md b/.spiral/cycles/CYC-20260919-TTZ7C-42.md new file mode 100644 index 0000000..a09049c --- /dev/null +++ b/.spiral/cycles/CYC-20260919-TTZ7C-42.md @@ -0,0 +1,76 @@ +# Cycle: Updated Roadmap And Planning Context + +Status: Accepted. Human accepted on 2026-09-23 after cycle evaluation. +Branch: spiral/CYC-20260919-TTZ7C-42-roadmap. +Base: b482053bdf0d3b3fc8902cce3169408d31ed9f6e (master). + +Goal: update ROADMAP.md with the maintainer's clarified outcomes, preserve open +design decisions, and reconcile planning context so Spiral risk analysis selects +the actual next task rather than treating the roadmap order as a task queue. + +Plan continuity: revise the earlier priority order following the completed +durability/recovery and runtime correction cycles. Keep durability invariants +and ordinary recovery restrictions. Full rebuild and CLI usability are the first +listed outcomes; broader testing is long range. No product cycle is selected. + +Scope: retained interview source, roadmap, directly affected project context +and agent navigation, and documentation/provenance verification. + +Non-goals: product code, dependencies, data, format changes, selecting a sandbox, +fixing merge semantics, general schema research, UI implementation or deployment. + +Verification: compare the roadmap with each settled interview answer; preserve +revisions and deferred decisions; check local links, whitespace and causal +references. Runtime tests are unnecessary for documentation-only changes. + +The actual branch was checked at creation. Recheck before semantic commits. +Existing untracked example indexes and package.json~ are outside this cycle. +Evaluation and human cycle acceptance precede a separate integration decision. + +## Evaluation + +Documentation and human acceptance are complete. +ROADMAP.md records the six agreed groups, separates initial outcomes from later +options, and delegates actual next-task selection to Spiral risk analysis. +The retained source records the interview's consequential corrections. + +Review against the interview confirmed: + +- Full-history rebuild precedes CLI usability in the listed resilience scope. +- File-backed-only retrieval preserves the remote API while allowing internal + change; the maintainer's performance report is distinguished from new evidence. +- Versioning follows individual objects through deletion and reuse of the same ID. +- Workspace effects are prohibited; merge execution remains an investigation, + with optional instance-level approval and an explicit account list. +- Public query access requires configurable authentication, read-only grants, + restricted capabilities and enforced time/memory/output limits. +- UI audiences, login/landing/menu behavior, overridable exploration, schema + authoring through UI or files, command documentation, personal fixtures and + bring-your-own-AI conversation are retained. +- IDE, shared fixtures, authoring AI, MCP, broader resilience and automation + remain later options. + +Project context and AGENTS.md now lead future planning to the current roadmap. +The context recognizes the integrated runtime correction, the implemented +recovery baseline and the historical nature of older risk snapshots. + +Verification: local Markdown links, whitespace and code fences passed review; +staged provenance validation passed with 107 documents and 189 references. +The dedicated branch identity was verified before each semantic commit. +Runtime tests were not run because only documentation and its graph changed. +No next product task, merge execution design or sandbox choice was selected. + +## Human acceptance + +After reviewing the cycle status and evaluated outcome, the maintainer replied +on 2026-09-23: + +> This is accepted. + +This accepts the evaluated branch tip +`10a39df3e946944a168614fd59514f2aeb4aec05`, including the updated roadmap, +retained interview and aligned planning context. The cycle is closed and ready +to propose for integration. No next product cycle is selected. + +The branch was rechecked as `spiral/CYC-20260919-TTZ7C-42-roadmap` when +recording acceptance. Integration into master requires a separate decision. diff --git a/.spiral/cycles/CYC-20260919-TTZ7C-42.ttl b/.spiral/cycles/CYC-20260919-TTZ7C-42.ttl new file mode 100644 index 0000000..40da7b0 --- /dev/null +++ b/.spiral/cycles/CYC-20260919-TTZ7C-42.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:CYC-20260919-TTZ7C-42 + a sd:Cycle ; + dcterms:identifier "CYC-20260919-TTZ7C-42" ; + sd:repositoryPath ".spiral/cycles/CYC-20260919-TTZ7C-42.md" ; + sd:status sd:Accepted ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-43 ; + sd:gitCommit "79fcf670d4558c75b9933743e6f2ea5616221eb0" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:CYC-20260919-TTZ7C-42 ; + sd:gitCommit "10a39df3e946944a168614fd59514f2aeb4aec05" + ] . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index ed279f7..1532c75 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,7 +18,14 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Latest accepted cycle: CYC-20260919-TTZ7C-36, Reliable Runtime Contracts And Readable +Latest accepted cycle: CYC-20260919-TTZ7C-42, Updated Roadmap And Planning +Context, on spiral/CYC-20260919-TTZ7C-42-roadmap. The maintainer accepted the +evaluated branch tip 10a39df3e946944a168614fd59514f2aeb4aec05 on 2026-09-23. +The documentation cycle is closed; integration into master remains a separate +decision. No next product task is selected; Spiral risk analysis selects it +using ROADMAP.md and current evidence. + +Previously accepted and integrated cycle: CYC-20260919-TTZ7C-36, Reliable Runtime Contracts And Readable Workflows, on spiral/CYC-20260919-TTZ7C-36-runtime-contracts. The maintainer confirmed this cycle after independent review found three regressions in the accepted runtime refactor. It deliberately precedes remaining randomized/soak @@ -26,8 +33,9 @@ work to repair shutdown completion, storage-failure acceptance and duplicate-ID compatibility, and clarify inspection/recovery coordinators. Source SRC-37, understanding UND-38, request REQ-39 and design DES-40 use the full 20260919-TTZ7C namespace. The maintainer accepted the evaluated branch tip -061571e48c453e5f3d7086a33af0d94926b2f7ea on 2026-09-19. Integration is pending -a separate decision; no cycle is active. Implementation and acceptance are complete: +061571e48c453e5f3d7086a33af0d94926b2f7ea on 2026-09-19. It was integrated through +PR #23 into master at b482053bdf0d3b3fc8902cce3169408d31ed9f6e. +Implementation and acceptance are complete: 118 regression tests and focused lint pass; EVD-20260919-TTZ7C-41 records six new regression cases and the restored contracts. The production change adds 18 net lines across runtime, inspection and administrative recovery. @@ -41,7 +49,8 @@ exposed by the accepted durability cycle. Its source, understanding, request and design are `SRC-20260919-TTZ7C-28`, `UND-20260919-TTZ7C-29`, `REQ-20260919-TTZ7C-30` and `DES-20260919-TTZ7C-32`. The maintainer accepted the cycle on 2026-09-19. The accepted branch was integrated locally into `master` at -`0d23ed0354fb9f1285d9bc3b15c0a1089583d65b`; the correction cycle above is accepted and awaits integration. +`0d23ed0354fb9f1285d9bc3b15c0a1089583d65b`; the correction cycle above is also +accepted and integrated. Accepted-cycle outcome: `IMP-20260919-TTZ7C-33` moves opened-store state and lifecycle into one cohesive `StoreRuntime`, leaving @@ -73,6 +82,13 @@ Status: **Complete** Human-confirmed complete on: 2026-08-17 +Targeted frame refresh: the 2026-09-19 roadmap interview clarifies UI audiences, +file-backed-only storage direction, remote versus internal API compatibility, +and a conditional public-query target. SRC-20260919-TTZ7C-43 retains the answers. +These are future outcomes, not claims that deployment readiness has changed. +Other intake commitments remain in force; reopen specific topics when a concrete +cycle exposes missing requirements. + | Required topic | Disposition | Notes / source | |---|---|---| | Purpose, users/stakeholders, goals | Covered | Intended users and current direction sections; human input on production-readiness audience and durability priority | @@ -123,37 +139,78 @@ Legacy sequential artifact IDs remain valid and should not be renamed. CYC-018 i ## Intended Users -Primary audience: developers using or extending SimplyStore. - -Secondary audience: end users interacting with systems built on SimplyStore, including query/command UI users where relevant. +Primary audience: developers evaluating, using and extending SimplyStore. +For the redesigned UI, first serve developers testing queries and exploring data, +then administrators checking status and performing maintenance. Next support +schema/command work, grants/accounts and audit inspection by developers/admins. +Downstream systems must be able to extend or overhaul the UI simply. -Tertiary audience: operators running SimplyStore-backed systems. +End users of downstream systems remain relevant in their own project contexts. +The roadmap's configurable anonymous query access is a future target requiring +sandbox evaluation, not an assertion of current public deployment safety. -Known from existing README: SimplyStore is a Node.js/Express library for serving in-memory datasets through a derived API and JavaScript query endpoint. - -Current known downstream project: `slonl/curriculum-store`, described publicly as a SimplyStore server with curriculum data. Human context says this is part of the SLO OpenData curriculum system at `https://opendata.slo.nl/curriculum/`. Public lookup confirmed the repository and OpenData page exist on 2026-08-17; the exact runtime dependency relationship has not been inspected. +Known downstream project: `slonl/curriculum-store`. Its schema implementation +is the starting point for shared schema information. During the roadmap interview, +local inspection of `data/schema.jsontag`, `src/import.merge.mjs` and +`scripts/tojsontag.mjs` found types, properties, relationships, labels, constraints +and import-time structural checks. Generalizing them requires investigation of +other datasets; this is not evidence of comprehensive validation. ## Current Direction -SimplyStore should become more production ready and less experimental while keeping its focus on simplicity. Production readiness is currently aimed primarily at developers evaluating SimplyStore, secondarily at the `curriculum-store` environment, and later at proposed changes coming from that downstream use. - -Known from existing README/roadmap: the project is experimental, should not be treated as production-ready by default, and has a known direction to replace VM2 with a safer isolate-based query runtime. - -The first production-readiness priority is proving more of SimplyStore's ACID claims, specifically durability. The durability goal is for SimplyStore to survive crashes at any point in its update cycle and recover, or fail instead of silently using corrupted data. - -The current durability/extensibility source is `.spiral/sources/SRC-001.md`. The first ordered slice is baseline archaeology, followed by a crash/fault-injection harness, an independent reconstruction oracle, adversarial storage tests, retry/idempotency hardening, after-change handler contracts, handler failure evidence, optional integrity roots, integrity acceptance tests, property/randomized durability tests, destructive soak tests, a minimal post-commit extension seam, a demonstration derived store, and `DURABILITY.md`. - -**2026-09-19 refinement — revise immediate order:** `SRC-20260919-TTZ7C-18`, `UND-20260919-TTZ7C-19`, and `REQ-20260919-TTZ7C-20` bring durable file publication and administrative recovery ahead of general randomized/soak work. These refine the earlier failure-only allowance: explicit startup refusal must be accompanied by a tested recovery route. Accepted/completed acknowledgments need complete file/directory persistence barriers; storage-fault evidence precedes hardening, followed by recovery tooling, command-input completeness, consistent backup/restore, latency measurements, and a disposable filesystem power-loss exercise. The rest of the original roadmap remains. - -**Recovery contract (required, not yet implemented):** missing datasets do not prove that external effects never happened. Ordinary rerun requires administrator approval and no later accepted command with a dataset; execute eligible trailing missing/waiting commands in original order from verified preceding state. Automatic retries must not bypass this rule. Preserve original evidence, preview recovery on a copy, and validate before explicit promotion. Required command inputs must be logged; omissions are bugs. Separate full rebuild may replay selected history with changed code into a new store while instructing commands/hooks to suppress external side effects. Record that cooperative contract in this cycle; full rebuild tooling is follow-up work. - -Process continuity rule: before proposing the next durability/production-readiness cycle, re-read `.spiral/sources/SRC-001.md`, `.spiral/requests/REQ-001.md`, `.spiral/designs/DES-001.md`, and the 2026-09-19 refinement in `.spiral/sources/SRC-20260919-TTZ7C-18.md` / `.spiral/requests/REQ-20260919-TTZ7C-20.md`; identify the current position in that ordered plan; compare the latest evidence with it; and explicitly classify the proposal as continue, revise, or deliberate deviation. - -Current Spiral collaboration rule: treat planning, evaluation, and ambiguous human suggestions as discourse until a sufficiently explicit commitment exists. Human confirmation of a cycle goal is the commitment boundary for execution within that goal; later tentative comments should not silently become scope expansion. Before commitment, surface material ambiguities, contradictions, unsupported premises, or alternative framings when resolving them differently could materially change downstream work. - -Current Spiral risk-selection rule: when choosing the next risk to reduce, consider uncertainty, downstream leverage, late-discovery cost, and cheap falsifiability. Existing blocker/near-term/deferred/existential horizons remain useful disposition metadata, but they are not a scoring model or a substitute for asking what later work depends on an assumption. - -A later/secondary direction is adding more options for including SimplyStore as part of a larger system. This should happen through minimal lifecycle seams, not by implementing Kafka, queues, topics, consumer groups, webhooks, a message bus, or an event-sourcing framework in SimplyStore core before evidence requires it. +[ROADMAP.md](../ROADMAP.md), based on the maintainer interview retained in +[SRC-20260919-TTZ7C-43](sources/SRC-20260919-TTZ7C-43.md), is the current broad +product direction. Its order is presentational; **Spiral risk analysis selects +the actual next task**. No implementation cycle was selected in the interview. + +The listed outcomes are: + +1. Full-history rebuild into a separate dataset, then discoverable Unix-style + administrative CLI operations with useful help. +2. File-backed retrieval as the sole approach and evaluation of a replacement + query sandbox. Preserve the remote API; internal data access may change. +3. Recursive per-object history, including deletion and restoration under the + same ID, with command, author and message information. +4. UI redesign around developer query/exploration tasks, with maintenance and + grant-controlled administration, and simple downstream customization. +5. UI/file-based schema authoring, personal query/visualization fixtures, and + in-UI conversation using the user's own AI. +6. Temporary shared workspaces, conflict resolution and optional instance-level + merge approval by listed accounts. + +This is a **revision** of the old immediate sequence. Broader randomized/soak +and filesystem testing, availability improvements, a command IDE, published +fixtures, authoring AI, MCP and automation remain later options. Risk analysis +may bring forward an investigation that reduces important downstream uncertainty. +Keep the project's simplicity constraint central, including future change cost. + +SRC-001 / REQ-001 / DES-001 retain the original durability rationale and +invariants. SRC-20260919-TTZ7C-18 / REQ-20260919-TTZ7C-20 refined power-loss and +administrative recovery; that implementation is accepted. DURABILITY.md states +its bounded filesystem evidence. Do not read earlier requests for baseline tests +or recovery implementation as proof that this work is still missing. + +Ordinary recovery requires administrator authorization and no later accepted +command with a dataset before an earlier command may rerun. Missing files do +not prove effects did not happen. Preserve original evidence and deliberate +promotion. Full-history rebuilding with selected code and suppressed effects is +separate, still future work. Logged commands must contain all required inputs. + +Workspaces must prohibit external effects during workspace execution; accepted +merges may perform them. Actual merge execution versus publication of a validated +replay remains a design investigation, judged by simplicity now and later. +QuickJS/V8 selection, schema generalization and bring-your-own-AI connections +also remain open. The roadmap does not authorize file-format migrations. + +Before proposing a cycle, read the roadmap and applicable durability references, +reconcile current effective behavior and accepted evidence, and classify the +proposal as continue, revise or deliberate deviation. Prioritize uncertainty, +downstream leverage, late-discovery cost and cheap falsifiability. Horizon labels +are disposition metadata, not a scoring model or fixed task queue. + +Treat unresolved planning suggestions as discourse. Confirm a concrete goal and +an evidenced gap before consequential product changes. The roadmap establishes +direction, not blanket implementation or integration authorization. ## Project Goals And Important Outcomes @@ -163,16 +220,19 @@ A later/secondary direction is adding more options for including SimplyStore as | Developer evaluation confidence | Primary production-readiness audience | Developers should be able to evaluate SimplyStore's limited durability claims from invariants, executable tests, evidence artifacts, explicit failure behavior, and documented known gaps | Human input, `REQ-001`, `DES-001`, current durability tests / explicit and evidenced | | Curriculum-store support | Known downstream context | SimplyStore changes should consider `curriculum-store` as a real environment, without letting it silently define all project priorities | Human input and public lookup / explicit and evidenced | | Simplicity | Current human direction and project identity | Must remain a shaping constraint while production readiness improves; do not add database/message-bus/event-sourcing machinery without evidence | Human input, README, completed intake / explicit and evidenced | -| Durability proof for ACID claims | First production-readiness priority | Survive crashes at any point in the update cycle and recover, or fail instead of silently using corrupted data; first work is baseline evidence and crash/fault-injection testing | Human input, README roadmap, `SRC-001` / explicit and evidenced | +| Durability proof for ACID claims | Preserve the accepted foundation while extending the system | Retain acknowledged-state, ordering and recovery guarantees within the documented envelope; broader testing is long range | DURABILITY.md, accepted EVD-27/EVD-41, SRC-20260919-TTZ7C-43 | | Integration into larger systems | Secondary direction after durability | Add minimal lifecycle seams for derived stores and post-commit observers; do not add broad messaging/event machinery without evidence | Human input, `SRC-001` / explicit | | Usable self-describing API over simple datasets | Stated project purpose | Preserve existing public behavior unless a cycle explicitly justifies and evaluates a breaking change | README, completed intake | | JSONTag-based semantic data support | Central differentiator | Preserve compatibility expectations unless a cycle explicitly justifies and evaluates a breaking change | README, completed intake | | Safe query execution | JavaScript queries run against provided data | VM2 is known unsafe; target replacement pending | README | -| Dataset scale expectations | README states a test goal around 1GB in memory | Unknown; not a current intake driver | README | +| Dataset scale expectations | Larger datasets with reduced memory use | File-backed-only direction; exact scale/latency targets remain to be measured | SRC-20260919-TTZ7C-43; maintainer reports favorable tests, not reproduced here | ## Project Posture -Brownfield reusable Node.js library moving from experimental toward developer-evaluable production readiness. It is not yet an established production service; the current hardening focus is bounded durability evidence and minimal extensibility while preserving simplicity. +Brownfield reusable Node.js library moving from experimental toward +developer-evaluable production readiness. The roadmap builds on accepted bounded +durability evidence while preserving simplicity. Internet-facing queries are a +future evaluation target, not a current deployment-safety claim. ## Important Invariants And Commitments @@ -180,7 +240,7 @@ Brownfield reusable Node.js library moving from experimental toward developer-ev |---|---|---| | Simplicity remains central | Production readiness should not turn SimplyStore into a conventional database, event-sourcing framework, message bus, or broad data platform | Human input / explicit | | Durable on-disk format must not silently change | Any on-disk format change needs an explicit cycle that treats migration, compatibility, and failure behavior as part of the work | Human input / explicit | -| JavaScript query API must not silently change | Query API compatibility matters to developers and downstream users; breaking changes need explicit justification and evaluation | Human input / explicit | +| Remote API compatibility | Preserve the remote interface; internal data-access APIs may change where needed for file-backed retrieval/runtime independence. Do not infer authorization for unrelated public behavior changes. | SRC-20260919-TTZ7C-43 / explicit | | REST API must not silently change | REST behavior is part of the public integration surface; breaking changes need explicit justification and evaluation | Human input / explicit | | Existing public data behavior should be preserved by default | Behavior changes need a cycle-level reason and evidence, especially where `curriculum-store` or other downstream users may depend on it | Human input / explicit | | Durability claims require executable evidence | SimplyStore should survive crashes at any update point and recover, or fail instead of silently using corrupted data | Human input / explicit | @@ -204,11 +264,11 @@ This is not yet a claim that SimplyStore is production-safe for all workloads. I | Decision / commitment | Why it still matters | Reversibility / exit cost | Source/confidence | |---|---|---|---| | Node.js/Express library | Defines integration surface and runtime | Unknown | `package.json`, README / evidenced | -| In-memory data model | Shapes scale, persistence, query behavior, and failure modes | Unknown | README / evidenced | +| Existing in-memory/shared-memory model | Historical implementation; roadmap replaces it with file-backed retrieval only | Internal data access may change; preserve remote API and durability | SRC-20260919-TTZ7C-43 / explicit future direction | | JavaScript query interface | Core user-facing capability and security concern | Unknown | README / evidenced | | JSONTag support | Core semantic-data representation | Unknown | README / evidenced | | VM2 currently used for sandboxing | Known security issue and migration pressure | Intended to replace; exit cost unknown | README, `package.json` / evidenced | -| ACID/durability claims should be proven before broader production confidence | Current first priority | Must cover crashes at any point in update cycle; exact scenario ordering pending human input | Human input, README roadmap / explicit and evidenced | +| ACID/durability claims need bounded evidence | Still governs confidence as roadmap features change the runtime | Preserve the documented envelope; risk analysis selects further evidence work | SRC-001, DURABILITY.md, SRC-20260919-TTZ7C-43 | | Do not add machinery until an invariant or demonstrated use case requires it | Preserves simplicity while adding production evidence | High-level principle; local application must be justified per cycle | `SRC-001` / explicit | ## Core Concepts / Vocabulary @@ -264,9 +324,14 @@ This is not yet a claim that SimplyStore is production-safe for all workloads. I | JSONTag packages | Data format support | Core dependency | Parsing/serialization behavior | | JAQT | Query helper library | Unknown | Query examples and behavior | -## Known / Tolerated Problems And Risks +## Historical Risk Register — Revalidate Before Selection + +The rows below retain earlier investigation snapshots. Several were addressed +by later accepted cycles, particularly durability and recovery. They are not a +current task queue or proof of an outstanding defect. Use current code, +DURABILITY.md, accepted evidence and ROADMAP.md to reassess them before selection. -| Concern | Current disposition | Evidence/source | Notes | +| Concern | Disposition when recorded | Evidence/source | Historical notes | |---|---|---|---| | VM2 has known security issues | Investigate | README | README says to keep SimplyStore away from public access until replacement | | Durability claims are not yet sufficiently proven | Investigate | Human input, README roadmap, `SRC-001`, `EVD-001` | Active first Spiral cycle area; target is recovery after crashes at any update point or explicit failure rather than silent corrupted-data use | @@ -318,13 +383,10 @@ This is not yet a claim that SimplyStore is production-safe for all workloads. I ## Later Possibilities -- Continue guided brownfield intake as concrete durability, security, API, and downstream questions arise. -- Add a developer/operator script for resetting or cleaning local/example datasets after explicit recovery failures. -- Administrative inspection, authorized ordered recovery, and consistent backup/restore are now committed in CYC-20260919-TTZ7C-17. More general automatic repair remains a later decision. -- Characterize the VM2 replacement risk as a candidate early Spiral cycle. -- Extend process-level fault tests around command acceptance, changeset write, status append, and restart recovery. -- Define larger-system integration options after the durability priority is better bounded. -- Produce `DURABILITY.md` only when executable evidence supports the documented claims. +Use ROADMAP.md for current later options and scope. Broader testing and +availability, a command IDE, published fixtures, authoring AI, MCP and automation +remain longer range. Existing minimal post-commit/derived-store ideas may support +concrete integration needs without requiring a broad event framework. ## Durable Non-Goals diff --git a/.spiral/project.ttl b/.spiral/project.ttl index 3140952..25585ee 100644 --- a/.spiral/project.ttl +++ b/.spiral/project.ttl @@ -9,7 +9,17 @@ project:CTX-001 dcterms:identifier "CTX-001" ; sd:repositoryPath ".spiral/project-context.md" ; sd:status sd:Active ; - sd:productionReadinessFocus "Primarily developers evaluating, using, or extending SimplyStore; secondarily the curriculum-store environment; operators are a tertiary audience." ; + sd:productionReadinessFocus "Developers evaluating, using or extending SimplyStore; the UI prioritizes query/exploration developers, then maintenance administrators, then extended administration. Public-query suitability remains a future evaluation target." ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:SRC-20260919-TTZ7C-43 ; + sd:gitCommit "79fcf670d4558c75b9933743e6f2ea5616221eb0" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:CTX-001 ; + sd:gitCommit "b482053bdf0d3b3fc8902cce3169408d31ed9f6e" + ] ; sd:developerEvaluationReadinessBar "Developers should be able to evaluate bounded durability claims from invariants, executable tests, evidence artifacts, explicit failure behavior, and documented known gaps." ; sd:hasAudience project:AUD-DEVELOPERS, project:AUD-END-USERS, project:AUD-OPERATORS ; sd:hasDownstreamProject project:DOWNSTREAM-CURRICULUM-STORE ; @@ -37,14 +47,14 @@ project:AUD-DEVELOPERS project:AUD-END-USERS a sd:ProjectAudience ; dcterms:identifier "end-users" ; - sd:priority "secondary" ; - dcterms:description "End users interacting with systems built on SimplyStore, including query and command UI users where relevant." . + sd:priority "downstream-context" ; + dcterms:description "End users of systems built on SimplyStore; their UI needs remain specific to those downstream systems." . project:AUD-OPERATORS a sd:ProjectAudience ; dcterms:identifier "operators" ; - sd:priority "tertiary" ; - dcterms:description "Operators running SimplyStore-backed systems." . + sd:priority "secondary" ; + dcterms:description "Administrators checking status and performing standard maintenance, followed by extended schema, command, grants/accounts and audit work." . project:DOWNSTREAM-CURRICULUM-STORE a sd:DownstreamProject ; diff --git a/.spiral/sources/SRC-20260919-TTZ7C-43.md b/.spiral/sources/SRC-20260919-TTZ7C-43.md new file mode 100644 index 0000000..b72cc65 --- /dev/null +++ b/.spiral/sources/SRC-20260919-TTZ7C-43.md @@ -0,0 +1,132 @@ +--- +id: SRC-20260919-TTZ7C-43 +--- + +# Source: Roadmap Interview And Risk-Based Selection + +## Origin and scope + +Maintainer conversation on 2026-09-19, after acceptance of the runtime contract +corrections. The maintainer requested a keyword-by-keyword interview followed +by an updated SimplyStore roadmap. Relevant quotations are retained below; +the other statements summarize the clarified answers. No external conversation +locator is available. This authorizes a documentation update, not implementation +of the roadmap's features or selection of the next product cycle. + +## Resilience + +> rebuild from command history, then easier operations, using cli scripts, the other options are a long range option, not near term. + +Rebuild the full history initially, into a separate dataset, retaining the +earlier changed-code and cooperative external-effect-suppression direction. +Selecting a history endpoint is not needed initially. CLI improvements should +make commands easy to find and run, with decent Unix-style help. Broader failure +testing and availability improvements remain long range. + +## File-backed retrieval and compatibility + +> Two things: larger datasets with reduced memory use, also easier switch to another javascript runtime, since we don't need to support shared memory if we use direct file access. + +> the remote API is the important part, internal API can change where needed + +The maintainer chose file-backed retrieval only, without retaining an in-memory +mode. They report having run performance tests with no noticeable slowdown. +No benchmark artifact or configuration was supplied in this interview; this is +a maintainer report, not a newly reproduced benchmark or a mandate to use mmap. + +## Object history + +> No, I want to be able to write a javascript query to show previous versions of a specific object. And then its previous version, recursively. The goal is to allow a UI to show an audit trail. The version information should give access to its command, author and message. + +Deleted objects must remain findable. Restoration must preserve the deleted +version in the history. Explicit reuse of an ID identifies a new version of +the same object. This replaces the older roadmap's whole-dataset time-travel +proposal as the current requested versioning scope. + +## Workspaces and the merge clarification + +Temporary workspaces isolate changes from main, can be shared read-only or +read-write, and can be removed or merged. A merge is possible only if commands +can be replayed on current main. If conflicts exist, apply current main to the +workspace, resolve conflicts, then merge. + +The conversation first considered validating replay and publishing its result +without executing commands on main. The maintainer then clarified: + +> That answers your previous question. I agree, workspace commands should not allow external side effects. Then the merge can trigger those on merge. + +After actual execution on main was suggested, the maintainer said: + +> I think that might be the cleanest solution. My main goal is to keep SimplyStore simple. So when we get to this point we should investigate which solution carries the least added complexity, and the best option for future enhancements staying simple as well + +Thus neither merge mechanism is settled. Preserve the no-workspace-effects +requirement and investigate how accepted merges perform effects. Approval is +optional per SimplyStore instance, with a list of accounts allowed to accept +merge requests. A different-person approval rule was not requested. + +## Query sandbox + +Replace VM2 with a system sufficiently fast and isolated to consider an +internet-facing JavaScript query interface. QuickJS and V8 isolates are +candidates, not selected solutions. Anonymous versus authenticated access is +configurable. The maintainer confirmed read-only queries, no filesystem/network +access except controlled dataset reading, and enforced execution-time, memory +and result-size limits. Internet suitability still requires evaluation. + +## UI audiences, navigation and implementation + +The current UI was a placeholder. Redesign it after carefully analyzing what +users need to do easily. Use Metro, SimplyFlow, Helene and other Muze libraries +where they make sense; their precise roles are not fixed. + +Audience order: developers testing queries and exploring data; administrators +checking status and doing standard maintenance; developers/administrators +extending commands or schema, managing grants/accounts and inspecting history. +Systems extending SimplyStore must be able to extend or overhaul the UI simply. + +The landing page introduces the system and dataset, and links to querying and +browsable exploration. Other features are available through a grants-based menu. +When reading is protected, an unauthenticated first request shows only login. +Exploration supports finding objects, inspecting properties, following links +and history without writing code. Defaults should be readable but may remain +technical, and downstream systems should be able to override them easily. + +## Schema, command documentation and fixtures + +Use curriculum-store's implementation as the schema basis. Investigate what +other users/datasets need before deciding extensions. Authorized users can +create/edit schema through the UI; externally authored schema files must also +be usable. Whether and how to generalize validation remains design work. + +Initially document available commands and usage. A command creation/testing IDE +is a later exploration. Fixtures mean saved queries with selected output +visualizations, not regression-test inputs/expected outputs. Start with personal +fixtures; adding them to a workspace or main dataset is a later addition. + +## AI and automation + +Initially provide conversation in the UI for data exploration and answering +natural-language questions through the query interface. Users bring their own +AI. Provider/credential integration is not specified. Assistance with authoring +queries, visualizations, schema and commands comes later; investigate MCP later. +Keep scheduled commands and webhooks from the earlier roadmap as long-range work. + +## Ordering and authorization + +The assistant proposed this order, confirmed by the maintainer: + +1. Full-history rebuild, then CLI usability. +2. File-backed retrieval and sandbox replacement. +3. Object history and audit trails. +4. UI redesign: landing page, queries, exploration and maintenance. +5. Schema editing, personal fixtures and conversational AI. +6. Workspaces and merge approval. + +The maintainer's final instruction was: + +> You can put them in that order in the roadmap. The spiral developer risk analysis should be in charge of actual next task selection. + +The order is a presentation of direction, not a mandatory task queue. Preserve +the earlier durability invariants and recovery restrictions while revising +priorities. Early UI analysis was proposed as possible parallel discovery; no +implementation task or additional cycle was selected in this interview. diff --git a/.spiral/sources/SRC-20260919-TTZ7C-43.ttl b/.spiral/sources/SRC-20260919-TTZ7C-43.ttl new file mode 100644 index 0000000..6b62baa --- /dev/null +++ b/.spiral/sources/SRC-20260919-TTZ7C-43.ttl @@ -0,0 +1,11 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:SRC-20260919-TTZ7C-43 + a sd:Source ; + dcterms:identifier "SRC-20260919-TTZ7C-43" ; + sd:repositoryPath ".spiral/sources/SRC-20260919-TTZ7C-43.md" ; + sd:status sd:Active ; + sd:sourceAvailability sd:Retained ; + sd:provenanceConfidence sd:Explicit . diff --git a/AGENTS.md b/AGENTS.md index b8be8c5..650eddd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,16 @@ administrative recovery path, complete logged command inputs, and ordered rerun eligibility; it records separate new-store rebuild semantics without making full rebuild tooling part of the current power-loss cycle. -Before proposing a next durability/production-readiness cycle, re-read those durable references, identify the current position in the ordered plan, reconcile the latest evidence with that plan, and state whether the proposal continues, revises, or deliberately deviates from it. +The 2026-09-19 roadmap interview in +`.spiral/sources/SRC-20260919-TTZ7C-43.md` revises priorities and clarifies scope. +Read `ROADMAP.md` for current direction. Its order is presentational: Spiral +Developer risk analysis selects the actual next task. Preserve open decisions +such as workspace merge execution and sandbox selection as investigations. + +Before proposing a cycle, re-read the roadmap and applicable durable references, +reconcile current behavior and accepted evidence, and state whether the proposal +continues, revises or deliberately deviates from that direction. Earlier risk +snapshots are not proof that a defect remains unresolved. Do not create Spiral artifacts merely to populate folders. Add `.spiral/` artifacts and companion Turtle resources only when they are causally useful for an actual cycle, decision, implementation, evidence, acceptance, or lesson. diff --git a/ROADMAP.md b/ROADMAP.md index a1607d3..e712c1b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,46 +1,270 @@ # SimplyStore Roadmap -## Version Control and History +Updated from the maintainer interview on 2026-09-19. This records intended +outcomes and open investigations; it does not claim the features exist. +The [retained interview](.spiral/sources/SRC-20260919-TTZ7C-43.md) preserves +the decisions and their clarifications. -SimplyStore already keeps track of changes made to the data. Each command that alters the data, does so by adding a changeset. This is stored as a separate file, with only those objects that have actually changed. +## How to select the next task -But these files are all merged together upon startup. And after each command, the shared data is updated again. The history is not accessible in queries or commands. You can only access that by reading in the changesets. +The order below is the agreed presentation of direction, **not a fixed delivery +queue**. Spiral Developer risk analysis determines the actual next task and +cycle. Reconcile this roadmap with current code, accepted evidence, dependencies +and user needs. Consider uncertainty, downstream leverage, the cost of discovering +an incorrect assumption late, and the cheapest useful way to test it. -To add the option to access historical data in queries, SimplyStore must give you access to those changesets directly. The idea is that the main `data` dataspace, and `meta.index` will only point to the latest data. But each object in the dataspace, and the dataspace itself, will have a pointer to the previous version of that data. This pointer will give information about when it was created, which command caused it, and information in the command itself, like the author and commit message. +State whether a proposed cycle continues, revises or deliberately deviates from +this direction. An earlier investigation may be justified by a later feature's +risk; early UI task analysis need not wait for all backend work. Agree a concrete +cycle goal before product implementation. This roadmap selects no next cycle. -Each historical entry can have a similar property that will point to its previous version. This allows you to list all versions of an object, including the commands and thus the author and message that accompanied it. +Keep SimplyStore simple. Compare approaches by their immediate conceptual cost +and how simply they allow future enhancements. Prefer useful boundaries and +existing capabilities over speculative framework machinery. -The root `data` entry, will also have this property. Each previous version will be a complete set of data, at that point in time. To get a list of objects that have actually changed in that version, a new property or method will be added. +## Current foundation -The `meta.index.id` will only point to the latest version of objects. You will need to walk through its previous versions for a specific version if you want it. A helper function will be added to do this more easily. +The accepted durability/recovery work provides log-authoritative execution, +durable acceptance and completion, explicit startup inspection, administrative +recovery, and consistent backup/restore tooling. The accepted runtime correction +cycle repaired shutdown completion, acceptance failure checks and duplicate-ID +compatibility while improving workflow readability. -All in this lays the ground work for a user interface that will allow you to time-travel. The default query interface should add a list of versions, with information about the commands. This will be added to `meta`. Selecting a version will switch your query to the correct dataset and re-run it. +These are bounded claims. [DURABILITY.md](DURABILITY.md) describes the tested +Linux/ext4 power-loss envelope and its limitations; the +[administrator guide](docs/recovery.md) documents the existing tools. Preserve +these contracts as the architecture evolves. A new sandbox, storage approach or +UI must be evaluated against them where relevant. -## Private and Shared User Workspaces +## 1. Resilience: rebuilding and usable CLI operations -When version history is available, it becomes much easier to implement custom workspaces. A workspace is a 'branch' from the main dataset, which contains extra changes not available outside the workspace. Any workspace has its own URL for both queries and commands. Commands sent to the workspace will alter data inside that workspace only. +### Full-history rebuild -You can mark a workspace private, in which case only you can access it. You can share the workspace with specific other users, given their account names. In which case they will also have access to it. You can limit this to either read-only or full access. +Provide a deliberate way to rebuild a separate dataset from the complete +command history, including when command code has changed. The first version +processes the full history; choosing a stopping command is not required. -When you are happy with your changes, you can merge them back into the main dataset and close the workspace. The way to do this is to merge all changes from the main dataset into the workspace. If there are any conflicts, SimplyStore will provide a UI to resolve those. Once the workspace is up to date, the changesets can be applied to the main dataset, and the commands added to the main command log and status. This is similar to a git rebase. +- Use the authoritative command order and complete logged inputs. +- Communicate a rebuild mode that tells commands and hooks to suppress external + effects and update only the rebuilt data. +- Preserve the original store and keep validation and deliberate promotion + distinct from building the new dataset. +- Make the operation available through the administrative CLI. -A workspace URL will look like this: +Rebuilding is distinct from ordinary recovery. Missing output never proves +that an external effect did not happen. Ordinary recovery retains administrator +authorization and the rule that no later accepted command may have a dataset +when an earlier command is rerun. -``` -https://example.com/simplystore/workspace/my-workspace/ -``` +The existing suppression contract is cooperative. Evaluate the actual command +and hook behavior before claiming a rebuild is free of external effects. +Define treatment of failed, unsafe and pending history, required starting data, +and changed-code failures during the rebuild design; do not silently interpret +full history as replaying every attempt or only current done records. -To this url you can add `query/` for the query interface, and `command/` to send commands. +### Discoverable Unix-style CLI -On disk, each workspace will get its own `data/` folder. The command-log and command-status files will be moved to the data folder, so each workspace will have its own command history. +Make existing inspection, recovery, backup/restore and the new rebuild operations +easy to find and run, with useful help. Favor conventional command naming, +`--help`, clear usage and examples, meaningful exit status, and understandable +diagnostics. Choose a coherent entry point during design; no new CLI framework +is prescribed. -Each workspace will save its branch-off point, the last command in the main dataset before the workspace was created. Or after an update, the last command before the update was done. +The near-term need is usability of these operations. Broader failure testing +and availability improvements remain long-range work. -## Default Workbench UI +## 2. File-backed retrieval and sandbox replacement -The current query UI is fairly limited. You cannot call commands, or edit or add commands. With the necessary security checks in place, it should be possible to create a more complete user interface that allows access to the commands, allow you to add or edit them, and test run them in a temporary workspace. You can then just delete the workspace, and be safe that any errors in your commands cannot affect the main dataset. +### File-backed retrieval as the sole approach -## Automation +Read data from files to support larger datasets with lower memory use and remove +shared memory as a requirement for alternative JavaScript runtimes. Do not retain +a separately supported in-memory mode. -You should be able to setup automated code flows, e.g. once every day do X, set up webhooks (remote URL's to call when certain data changes or commands are called, and the other way around). This allows SimplyStore to become part of a larger system. +Preserve the remote API. Internal data-access APIs may change where necessary; +the roadmap does not require synchronous transparent object access or a specific +cache/mapping mechanism. File-format changes are not automatically authorized. +The maintainer reports performance tests with no noticeable slowdown. Retain +that as supporting experience, not a reproduced benchmark or a universal +performance guarantee. Verify representative query/command performance and +memory use, including data larger than RAM, when choosing the implementation. +Preserve consistent views, authoritative ordering and durability behavior. + +### Replace VM2; evaluate QuickJS and V8 isolates + +Replace the existing VM2 sandbox with an execution boundary whose isolation and +performance can support consideration of an internet-facing query interface. +QuickJS and V8 isolates are candidates to investigate, not selected dependencies +or a commitment to replace the whole server runtime. + +Required query behavior: + +- Anonymous or authenticated access, configurable per instance. +- Read-only dataset access under the applicable grants. +- No arbitrary filesystem or network access; dataset reads go through a + controlled boundary. +- Enforced execution-time, memory and result-size limits. + +Evaluate candidate isolation, termination, resource enforcement, compatibility +and performance against realistic and hostile queries. Choosing an engine alone +does not establish that public access is safe. File-backed retrieval should +help remove shared-memory coupling without exposing arbitrary files to queries. + +## 3. Object versioning and audit trails + +Allow a JavaScript query to find a specific object's previous version, then +follow previous versions recursively. Each version gives access to its command, +author and message. The purpose is a UI-readable audit trail. + +- Deleted objects remain discoverable by ID and retain their history. +- Deletion remains a visible step in the version chain. +- Explicitly using an existing or previously deleted ID creates a new version + of the same object, including when restoring it after deletion. +- Restoration preserves the intervening deleted version in its history. + +Choose the history-access API and representation during design. The earlier +whole-dataset time-travel UI proposal is replaced by this object-history scope; +it is not an implicit requirement for this milestone. Do not assume versioning +requires the earlier proposed linked dataspace/index layout. + +## 4. UI redesign: querying, exploration and maintenance + +### Start with tasks and audiences + +The current UI was a quick placeholder. Analyze what users should be able to do +and do easily before choosing the redesigned interaction and component structure. + +Audience order: + +1. Developers testing queries and exploring the dataset. +2. Administrators checking status and performing standard maintenance. +3. Developers/administrators managing schema, commands, grants and accounts, + and inspecting audit trails. + +Use Metro, SimplyFlow, Helene and other Muze libraries where they make sense. +Determine their roles from the tasks and fit. Keep the UI simple for downstream +systems to extend, replace individual displays, or overhaul entirely. + +### Landing page and navigation + +Provide a simple introduction to the system and dataset, with links to the +query interface and browsable exploration. Put other operations in a menu +according to the user's access grants. When read access is protected, an +unauthenticated first request shows only a login form. + +The landing page is not an elaborate operations dashboard. Give status, +maintenance, account/grant administration and audit inspection appropriate views +and navigation. Enforce authorization at the operation boundary as well as in +the visible menu. The exact maintenance controls need task analysis. + +### Querying and exploration + +Support writing and testing JavaScript queries and inspecting results. Provide +browsing without writing code: find objects through search or lists, inspect +properties, follow relationships and access object history. + +Default displays should be reasonably readable and can remain technical. +Make it easy for extending systems to override presentation for their own data. + +### Command documentation first + +Document available commands and how to use them. The initial scope does not +require an in-browser command runner or editor. A command creation/testing IDE +is a later investigation. + +## 5. Schema editing, personal fixtures and conversational AI + +### Schema information + +Use curriculum-store's schema implementation as the starting point. It includes +types, properties, relationships, labels and constraints, with schema-informed +structural checks in import code. Investigate requirements from other users and +datasets before selecting which extensions SimplyStore should support. + +Authorized users must be able to create and edit schema information through the +UI. Externally authored schema files must also be usable as supplied; the UI +must not become the only authoring route. + +Design the relationship between file-supplied and UI-edited schema, and determine +which validation and presentation semantics belong in the reusable system. +Do not assume that every curriculum-specific rule becomes a SimplyStore rule, +or that existing checks already provide comprehensive schema validation. + +### Personal query fixtures + +A fixture is a saved query paired with its chosen output visualization, ready +to reopen and run. It is not a regression-test input/expected-output fixture. +Start with personal fixtures. Publishing fixtures into a workspace or the main +dataset is a later addition. Visualization choices and extension points should +follow the UI task analysis rather than a prescribed charting framework. + +### Conversation over the query interface + +First provide an in-UI conversational interface that lets AI explore the data +and answer natural-language questions through the query interface. Users bring +their own AI; provider, account and credential integration require investigation. + +The assistant uses the current user's data-access grants and the read-only, +resource-limited query boundary. It does not gain extra privileges through its +provider connection. Query/result inspectability is a design consideration so +users can assess answers; the exact interaction remains to be designed. + +Later, explore AI assistance with authoring queries, visualizations, schema and +commands. Investigate MCP after the conversational UI; it is not the first +integration target. + +## 6. Temporary workspaces and merge approval + +Let people create a temporary workspace, change it without affecting main, and +share it with other accounts using read-only or read-write access. A workspace +can be deleted or merged into main. + +- Workspace commands must not cause external side effects. +- A merge is eligible only when its commands can be replayed on current main. +- When conflicts exist, apply current main to the workspace, resolve conflicts, + and then attempt the merge again. +- Merge approval is optional and configured per SimplyStore instance, with a + list of accounts permitted to accept merge requests. +- An accepted merge may trigger the intended external effects. + +The merge execution mechanism remains open. Investigate replaying commands on +main with effects enabled after validation/approval, alongside alternatives +that publish a validated replay result and arrange effects separately. Choose +the least added complexity, accounting for future enhancements as well as the +initial implementation. Neither alternative is a committed design. + +During design, resolve how changes to main invalidate validation or approval, +how conflicts are represented/resolved, how workspace effect restrictions are +upheld, and how partial failures involving external effects are handled. Do not +assume database-style rollback can undo arbitrary external actions. + +Workspace URLs, on-disk layout and history integration are design decisions. +The earlier roadmap's folder moves and rebase-like mechanism are not mandatory. +Workspace isolation, object history, rebuild behavior and sandbox restrictions +should share concepts where that simplifies their contracts. + +## Long-range options + +- Broader resilience evidence: randomized/adversarial campaigns, soak testing, + additional filesystems/storage environments and deployment-specific power loss. +- Availability improvements beyond the current failure/recovery behavior. +- A command IDE for creating and testing implementations. +- Fixtures published into workspaces or main. +- AI assistance with query, visualization, schema and command authoring; MCP. +- Automation, including scheduled commands and incoming/outgoing webhooks. + +Keep automation and larger-system integration small. Earlier ideas for minimal +post-commit/derived-store seams remain possible enabling work, not an obligation +to build a messaging platform. Risk analysis determines when these investigations +are justified; their presence here is not implementation authorization. + +## Continuity and evidence + +This roadmap revises priority and scope from the earlier roadmap and durability +sequence. It preserves the invariants in +[DES-001](.spiral/designs/DES-001.md) and the ordinary recovery restrictions in +[REQ-20260919-TTZ7C-20](.spiral/requests/REQ-20260919-TTZ7C-20.md). +The [current project context](.spiral/project-context.md) and executable evidence +remain necessary when forming a concrete cycle. Open investigations here must +not silently become implementation commitments.