Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
172 lines (131 loc) · 38.4 KB

ARCHITECTURE.md

File metadata and controls

172 lines (131 loc) · 38.4 KB
Raw

Architecture

All src/ paths are relative to packages/plugin/ (the published npm package). File locations live in STRUCTURE.md; this document explains how the pieces fit and — above all — the invariants that keep the Anthropic prompt cache stable. When in doubt about transform behavior, read "Transform pass mechanics" below before touching code.

Overview

Magic Context is an @opencode-ai/plugin (entry src/index.ts) that rewrites the message array and system prompt on every LLM call to keep a long session inside the context window without losing history. Core tenets:

  • Thin adapters, real logic separated. OpenCode-facing handlers live in src/plugin/; feature logic in src/hooks/magic-context/ (runtime), src/features/magic-context/ (services), src/tools/ (agent tools).
  • Durable SQLite state, never ephemeral — if storage is unavailable the plugin fails closed rather than silently letting the prompt grow past the provider limit. DB at ~/.local/share/cortexkit/magic-context/context.db, shared cross-harness (OpenCode + Pi); session-scoped tables carry a harness discriminator (including session_projects for session-to-project mappings backfilled on boot), project-scoped tables (memories, git commits) are shared.
  • Replay-everything for cache stability. Every persistent message mutation (reasoning clearing, structural-noise / placeholder / image / merged-assistant stripping, caveman compression, synthetic-todowrite, drop placeholders, and smart-drop/edit-marker compressions) is re-applied deterministically on EVERY transform pass — including defer passes — so the wire bytes stay byte-identical and the provider prompt cache survives.
  • Hidden subagents (historian, historian-editor, dreamer, sidekick) do the heavy LLM work out of band; the transform itself does no LLM calls.
  • Runtime SQLite backend (src/shared/sqlite.ts): bun:sqlite under Bun, node:sqlite (DatabaseSync) under Node (Pi) and Electron (Desktop). The non-Bun branch adds a savepoint-aware transaction() shim and readonly→readOnly mapping; otherwise identical. No native module, no prebuild.
  • Pi parity: packages/pi-plugin/ mirrors OpenCode semantics, importing shared core from @magic-context/core. Intentional divergences are tracked in packages/pi-plugin/PARITY.md. Provider prefixes are translated between canonical (OpenCode) and Pi configuration models via src/shared/harness-provider-map.ts at configuration read/write edges to keep shared model configurations portable.
  • Rust module / subc integration. A harness-agnostic Rust workspace in crates/ re-implements the cache-stability transform and autonomous historian to run under the subconscious daemon (subc), communicating via the subc command and subscribe wire protocols.

Layers

  • Bootstrap (src/index.ts): load config, register hidden agents + hooks + tools, start RPC server, dream-timer, auto-update checker; detect conflicting plugins (DCP / OMO / OpenCode auto-compaction) and disable the runtime if any is active.
  • Adapters (src/plugin/): hook wrappers, tool registry, RPC handlers, dream-timer lifecycle, per-session hook construction.
  • Runtime (src/hooks/magic-context/): the transform pipeline, postprocess phase, event/command handlers, system-prompt injection, compartment runners, decay rendering, strip-and-replay, nudges, m[0]/m[1] injection, and the shadow-transform sender (a flag-gated dev lane that mirrors transform passes to the Rust subc module).
  • Feature services (src/features/magic-context/): storage, scheduler, tagger, memory, dreamer, sidekick, git-commit + message FTS indexes, unified search, overflow detection, session-project mapping and backfill, migrations, project identity resolution (resolves git:<sha> or fallback dir:<md5-12> identifiers, caching directory fallbacks, and utilizing a cooldown period for transient git errors).
  • Tools (src/tools/): ctx_reduce, ctx_expand, ctx_note, ctx_memory, ctx_search. Gating: when memory.enabled is false, ctx_memory is not registered (on Pi, it registers but refuses queries for memory-off projects), and all memory-related guidance is stripped from the system prompt.
  • Config + shared (src/config/, src/shared/): Zod config (deep-merge raw JSONC before validation; invalid leaves fall back to defaults with warnings, never disable the plugin). Project-tier trust boundaries (src/config/project-security.ts) strip unsafe fields from untrusted repository configs before merge — preventing escalation via sqlite PRAGMAs, hidden-agent reprogramming (prompt, tools), embedding destinations, or developer-only shadow transform settings. Project-level compaction thresholds are raise-only to prevent cloned repos from forcing extra historian cost. Also includes logger, data paths, SQLite selector, harness id, RPC transport, conflict detector, tag-transcript primitive (shared with Pi), provider-id translation map (src/shared/harness-provider-map.ts), and exit-abort listener coordinator (src/shared/exit-abort-registry.ts).
  • TUI (src/tui/): sidebar + /ctx-status / /ctx-recomp dialogs, RPC-backed; shipped as raw TS via the ./tui export (not bundled into dist/index.js).
  • CLI (packages/cli/, separate @cortexkit/magic-context package): npx setup / doctor / migrate wizard, including local embedding runtime checks for missing or broken onnxruntime-node native bindings.
  • Rust Core & Store (crates/mc-core/, crates/mc-store/): Re-implement cache-stability transform/classification logic and durable SQLite schemas with row-version CAS in Rust.
  • Rust Tokenizer (crates/mc-tokenizer/): Tiktoken byte-BPE Claude token estimator.
  • Rust subc Module (crates/mc-module/): Integrate with the subc daemon, handle transform requests, route MCP facade requests (for ctx_memory, ctx_search, ctx_expand, and ctx_note), and orchestrate the autonomous historian. Serve prompt guidance with support for a "no_reduce" variant on reduce-less surfaces. Expose co-owned profile and feature epochs (PROFILE_EPOCH_CLAUDE_CODE_ANTHROPIC, TAGGER_FEATURE_EPOCH) during status checks to assert compatibility, folding profile-specific serializer epochs to coordinate hard transitions on format changes. Maintain a durable pass trace (mc_pass_trace table in mc-store) to audit session receive/complete/reject events separately from cache state. Use harness model codecs to translate between harness-specific JSON (OpenCode, Pi) and canonical CkWireMessage values, reject client-reported context limits below MIN_PLAUSIBLE_CONTEXT_LIMIT (1024) by falling back to 200,000, and use serializer profiles to selectively disable tail mutations for byte-splice consumers (e.g. Claude Code) to prevent phantom reclaims while absorbing covered system-role messages into the m0 baseline.

Transform pass mechanics

This is the heart of the system and the part most easily gotten wrong. A "transform pass" is one invocation of experimental.chat.messages.transform (src/hooks/magic-context/transform.ts), wrapped defensively in src/plugin/messages-transform.ts (transient SQLITE_BUSY → return messages unmodified so the prompt loop always proceeds). OpenCode fires it once per LLM round-trip (per step within a turn).

Pass lifecycle (in order)

  1. Resolve usage + scheduler decision (execute vs defer).
  2. Emergency overflow recovery if ≥95%.
  3. Compartment trigger check (off the in-memory args.messages tail — no opencode.db read steady-state); fire the historian async if eligible.
  4. Prepare compartment injection (decide m[0]/m[1] materialization).
  5. Tag messages; replay dropped-status, caveman, reasoning, placeholder, image strips.
  6. Compartment phase: inject the <session-history> (m[0]/m[1]) into message[0].
  7. Postprocess (transform-postprocess-phase.ts): the mutation gates — pending-op drain, heuristic cleanup, nudges, synthetic-todowrite, auto-search.

Pass taxonomy (every pass is exactly one)

  • SOFT+ (defer / cache_hit): nothing new. m[0] AND m[1] replay byte-identical; the entire system + m[0] + m[1] prefix stays cached. Only the conversation tail moves (where ctx_reduce/age drops land, themselves replayed deterministically). The steady state — most passes are this.
  • SOFT (cache-busting): m[1] re-renders (new compartments / memories / user-profile surface as deltas) while m[0] stays byte-identical. system + m[0] stays cached; the cache busts at the m[1] breakpoint. Driven by an execute pass, /ctx-flush, or a deferred-history drain.
  • HARD (m[0] fold): mustMaterialize fires → m[0] re-materializes, folding m[1] into the new decayed baseline and resetting m[1] to a placeholder. The whole prefix rebuilds — but "for free" because the provider cache key was already dead (see HARD triggers in "m[0]/m[1] cache layout"). Decay re-tiering happens ONLY on a HARD fold — a SOFT pass must never re-tier (that would change m[0] bytes).

The mutation gates (the part to get right)

Pending-op drain and heuristic cleanup are each gated by the same shape in transform-postprocess-phase.ts:

shouldApplyPendingOps / shouldRunHeuristics =
  (execute || materializationRequested || forceMaterialization || m0HardFoldThisPass)  // BUST clause: is this pass busting anyway?
  && (!compartmentRunning || emergencyBypassCompartmentGate)                            // VETO clause: is the historian mid-run?
  • BUST clause — only mutate (drop tools, run heuristics) on a pass that is already busting the prefix, so the mutation rides that one bust instead of causing its own. m0HardFoldThisPass is the fold-exec signal (an advisory mustMaterialize call earlier in postprocess).
  • VETO clause — compartmentRunning — block mutation while the historian is summarizing the tail, so we don't change the bytes it's reading mid-run. Bypassed by emergencyBypassCompartmentGate.
  • emergencyBypassCompartmentGate bypasses the veto when forceMaterialization (≥85%) OR m0HardFoldThisPass — i.e. a hard fold drains pending ops + runs heuristics even while the historian runs, because the prefix is busting regardless (see "drain into the known bust" invariant). This is safe per the disjoint-DB model below; Pi already does this (context-handler.ts).

Load-bearing invariants (memorize these)

  1. A HARD bust means the prefix is already gone → drain EVERYTHING into it. Never "defer" a hard bust. This pass IS the fold; there is no later fold to wait for. Deferring the drain only produces a second, avoidable bust ~one turn later. (The compartmentRunning veto must therefore yield to a hard fold — the fold-exec bypass.)
  2. A defer (SOFT+) pass must replay byte-identical. Any first-application of a strip/drop on a defer pass changes tail bytes and busts the whole prefix after it. Watermark-gated strips (placeholders, images, stale-ctx_reduce) use a frozen-id replay pattern: detect-and-freeze the affected ids only on cache-busting passes, replay the frozen set on every pass. There is exactly ONE drop placeholder string, [dropped §N§], a pure function of tag id — never re-derive bytes from mutated content (that caused repeated cache catastrophes).
  3. Deferred work rides the next bust cycle; it never forces its own. Historian publishes, compaction-marker moves, and queued drops accumulate while m[1] replays frozen, and materialize together on the next genuine bust (execute / hard fold / flush). A historian publish does NOT bust the cache — between busts every pass is cache_hit.
  4. Boundary execution defers mid-turn. execute decisions become defer while the latest assistant turn is mid-tool-use (CAS-flag deferred_execute_state), so we don't rewrite bytes while a turn is still accumulating tool calls. Bypasses: ≥85% force, explicit-bust, subagent. Drained re-peek-and-clear at end of postprocess.

Disjoint-DB safety model

Mutating while the historian runs is safe because the two databases are disjoint on the read/write side:

  • The historian reads raw OpenCode messages from opencode.db (read-only) for its chunk.
  • Drops + heuristics mutate context.db (tags / pending_ops) and the in-memory outgoing wire only.
  • The historian's in-flight snapshot is validated by computeRawRangeFingerprint, which hashes raw content only (ids, part types, content lengths) — never tag/drop state — so a concurrent drop can't invalidate it.
  • Its post-publish queueDropsForCompartmentalizedMessages is idempotent against already-dropped tags.

m[0]/m[1] cache layout

The compacted history renders into TWO synthetic user-role message slots at the head, so the large stable prefix survives steady-state work. inject-compartments.ts (renderM0 / renderM1 / materializeM0 / mustMaterialize), mirrored in inject-compartments-pi.ts. Both slots prepend with synthetic: true parts so they don't count toward OpenCode's title-generation gate.

  • m[0] — cumulative baseline (frozen, like system[0]). Holds <project-docs> (root ARCHITECTURE.md + STRUCTURE.md), baseline <user-profile>, and the decay-rendered compartment history as of the last materialization. Does NOT change on routine turns.
  • m[1] — volatile delta. Holds everything added since the last m[0] materialization: new user-profile additions, new memories (via the maxMemoryId watermark), <memory-updates> supersede deltas, and the newest compartments at full tier. Renders a minimal placeholder when empty (never fully empty — Anthropic cache-breakpoint structure).

mustMaterialize (HARD fold) triggers — organized around the bust taxonomy so the trigger list and the m[0]/m[1] contract can never silently disagree:

  • Provider-side cache eviction (the cache is already dead, so folding is free): model/provider change (cachedM0ModelKey), system-prompt-hash change (cachedM0SystemHash), idle > TTL (cacheExpired, self-consuming via lastResponseTime > cachedM0MaterializedAt).
  • Genuine m[0] content change (baseline bytes differ): first render, cached_m1_missing, project_memory_epoch change (dashboard / external mutation), pending m[0] mutations (max_mutation_id — structural compartment delete/merge/recomp), upgrade-state change.
  • Deliberately NOT triggers (these are m[1] deltas — triggering would bust m[0] on routine background work and defeat the design): new compartment sequence, project_user_profile_version, maxMemoryId, project-docs-hash change (docs edits fold in on the next natural hard bust, never on their own), and tool-set-hash change (process-global, false positives).
  • Pressure backstop refold: on a cache-busting pass, if no natural HARD bust has arrived but m[1] has grown large — gated by the m[1]/m[0] size ratio (with a small-m[0] floor) OR an absolute m[1] token cap (~20% of history budget) OR a large memory-mutation count.
  • applyMarkersToState updates ALL state.cachedM0* fields post-materialize (guards against an infinite re-materialize loop). /ctx-flush is SOFT (drives m[1] refresh + heuristics, not an m[0] fold).

Memory mutations route through m[1], not the epoch. In-session ctx_memory mutations do NOT bump project_memory_epoch: additive writes surface via the maxMemoryId watermark; non-additive (update/archive/merge) record a memory_mutation_log row rendered as a <memory-updates> delta. Both reconcile into m[0] on the next natural hard bust. The epoch is bumped only by dashboard mutations and /ctx-session-upgrade migration (an external editor can't otherwise signal a running session).

Historian compartment flow (produce → store → render)

The long-history pipeline. Tiered compartments + deterministic decay renderer (replaced the v1 flat-compartment + LLM-compressor model).

  1. Trigger (compartment-trigger.ts): threshold-relative pressure (context_limit × execute_threshold × 5%, clamped 5k–50k), commit clusters, and TC-chunked unsummarized-tail size (≥ triggerBudget × 3), while protecting the live tail. Runs off the in-memory tail (zero opencode.db reads steady-state); hands the resolved boundary snapshot to the runner so the historian sees exactly what the fire decision saw. On the Rust/subc module leg, the skip discriminant (trigger_false, no_models, etc.) is written durably to HistorianDurableState.last_no_fire (change-gated to avoid hot-path writes, and cleared on fire), and trigger_false details carry quantized measurements. The transform handler recognizes the mc-historian: session namespace as self-owned and returns an identity pass-through to prevent self-recursive loops.
  2. Produce (compartment-runner-incremental.ts): runs the historian subagent on the raw chunk above the last compartment boundary with a bounded prompt (no full state dump) — 4 rotating seed compartments + the last 6 persisted compartments + the project-memory block for fact dedup. Emits each compartment with 4 paraphrase tiers (p1 verbose → p4 anchor-only), an importance (decay-rate semantics), an episode_type, a <facts> block in the 5-category taxonomy, and an <events> block. On the Rust/subc side, the run uses a calibrated sampling temperature of 0.1 and a generous max_output_tokens of 32k with a 600s await budget. Truncation is tracked via ProducerOutput.length_capped and validation rejections write last_failure with validation errors and length-cap hints. Producer session IDs (mc-historian:) include a bound-session hash to isolate concurrent subagent lineages and prevent TerminalRunMismatch collisions. During assembly, enforce a min_chunk_tokens substance floor to avoid spawning a producer on near-empty content, but bypass this floor when the session is in emergency (≥95% usage) or on verbatim-tail profiles where folding is the sole reclaim path (fold_is_only_reclaim is true).
  3. Parse + validate: parseCompartmentOutput + validateHistorianOutput (sparse-but-strictly-increasing ordinals for consumer legs, non-overlapping ranges, correct unprocessed_from). Live coverage validation absorbs mid-span system role ordinals safely without rejecting the chunk.
  4. Discard-last boundary healing: if the historian consumed to the chunk edge with weak lookahead, the last (lookahead-free) compartment is not persisted; the next run re-reads it at the head with full lookahead. Guarded by progress (k≥2) and emergency-disabled. Skips unanchored fact, observation, and primer promotion on the discarded tail to prevent double-storing when the range is re-processed.
  5. Store: publish transaction appends compartments with tier columns. Promotable facts promote to project memory (exact-dedup); user_observations stored only when dreamer.user_memories.enabled (privacy gate). Events → compartment_events. Compartment-chunk embeddings generated on publish (memory-gated). Publish defers a compaction-marker move (see Subsystems) and signals a deferred history refresh — it does NOT force a bust. When reattaching, the historian subscriptions are always re-drained from the start (persisted producer_cursor is killed to prevent dropping output). First published compartment after an empty bootstrap forces a HARD fold (since a SOFT delta is impossible without an existing boundary).
  6. Render (decay): decay-render.ts (shared OpenCode + Pi) picks one tier per compartment via decay-curve.ts: half-life H = H50·2^((I−50)/D)/max(p,0.10) (H50=24, D=25), log-cost tier boundaries [0.201,0.729,1.322,2.587], budget pressure p once per pass. Older / lower-importance / higher-pressure compartments demote oldest-first; past the archive boundary they render P4/self-close or drop. Self-tunes as the context window changes — no LLM call. Legacy (pre-v2) rows render P3 (if they carry a U: line) else P4.
  7. Recomp / upgrade: /ctx-recomp rebuilds compartment structure from raw history (emits NO facts — preserves curated memories). /ctx-session-upgrade runs full recomp + a once-per-project 9→5-category memory migration (active only, permanent untouched, bumps the epoch).
  8. Wrapup: /ctx-wrapup manually compacts older raw conversation history into compartments in token-capped chunks while keeping the newest N raw messages intact. Guided by resolveWrapupProtectedTailBoundary, the orchestrator runs in sequential loops under a session-specific wrapup lease, and skips unanchored fact, observation, and primer promotion on the final chunk.

Protected-tail boundary

protected-tail-boundary.ts decides, per pass, which prefix of the raw tail is eligible for the historian and which suffix stays protected — from true-raw token sizes (not user-turn counts), so sparse-user-turn sessions can't deadlock the historian (#132). Boundary anchors at lastCompartmentEnd + 1; token target N capped at 0.40 × usable (ABS_CAP 96k); a live-prompt floor keeps it from crossing the newest meaningful user message on routine (<80%) passes. An emergency drain catch-up latch (usage-driven) bypasses the per-run token cap when pressure spikes ≥85%, draining continuously until usage falls back below the safe threshold. Open tool arcs (a tool invocation with no result in the window) only hold the boundary back when recent (≥ the size-walk start = the live window); a stale/interrupted open arc older than that is compactable — otherwise one dead running tool call at the eligible-head edge would freeze the historian indefinitely. The trigger/runner share a content-stable range fingerprint for cross-view staleness validation. For manual wrapup passes, resolveWrapupProtectedTailBoundary counts raw messages (including tools) rather than user turns to determine the keep watermark (default 20), protecting the newest tail segment while using the same tool-arc fencing and user-boundary snapping logic. On verbatim-tail profiles (where fold_is_only_reclaim is true, e.g. claude-code-anthropic), folding is the only reclaim path while the newest message is forwarded in full. Keep the newest message and its completed tool arc in the tail and out of the fold so that the live turn does not become a durable compartment boundary.

Memory, search & embeddings

  • Memories (memory/storage-memory.ts): project-scoped durable knowledge in the 5-category taxonomy (PROJECT_RULES / ARCHITECTURE / CONSTRAINTS / CONFIG_VALUES / NAMING), with FTS + vector side tables. ctx_memory exposes write/archive/update/merge/list; list is dreamer-only; primary agents may only mutate their own project's memories (workspace-shared categories aside). Writes are idempotent, and an embedding hash guard (saveEmbeddingIfHashMatches) ensures vectors are only saved if the memory content hasn't changed during the provider call.
  • Unified search (search.ts): one query embedding or text matching query dispatched across memories, raw message history (FTS via message-index.ts), indexed git commits, compartment-chunk embeddings, and session/smart notes. Hard-filters memories already visible in <session-history> and raw-message hits newer than the last compartment boundary (already in context). The raw-message index tracks failed incremental writes via a dirty floor (dirty_floor_ordinal), rewinding the async reconciler to cover gaps.
  • Embeddings: vectors stored as plain SQLite BLOBs, scanned in-memory via Float32Array cosine (sqlite-vec rejected — bun:sqlite can't load extensions + write-amplification). Provider resolved per-project; a substitution guard rejects a served model that doesn't match the requested one. Compartment-chunk embedding is on-demand via /ctx-embed (auto-drains the active session once per process; resilient retry with circuit-break; chunk-window config folds into chunk identity so it doesn't invalidate memory/commit vectors). When using the local embedding provider, if the onnxruntime-node native binding is missing or fails to load, it is permanently disabled for the process to prevent repeating import errors, and a warning is logged routing the user to the CLI doctor command.
  • Workspaces (workspaces / workspace_members): a project belongs to at most one workspace; member sessions read the union of members' memories (repo-attributed), gated per-category by share_categories. A cached_m0_workspace_fingerprint (sorted identity+epoch+categories hash) detects membership/policy changes for a single hard fold on change. Workspace sharing is fail-closed (read visibility only); primary ownership gates ensure tools cannot mutate foreign workspace memories.

Dreamer

Background maintenance (V2: per-task cron scheduling). A process-wide 15-min timer evaluates each task's cron (dreamer/cron.ts) against task_schedule_state (per-(project,task) lastRunAt/next_due_at/schedule, reconciled from config each pass), runs due tasks through their activity gate (task-gates.ts), and serializes them by conflict-domain lease (task-registry.ts): the memory-mutating tasks (map-memories, verify, verify-broad, curate, classify-memories, retrospective, promote-primers, refresh-primers) share one memory:<project> lease; others (maintain-docs, evaluate-smart-notes, review-user-memories) hold independent leases. Each lease uses a guarded write (runLeaseGuardedWrite via BEGIN IMMEDIATE) so read-then-write steps are atomic across processes. Each task runs in its own ephemeral child session via task-executor.ts; resolve model per-task (override → dreamer → session-model last resort, historian-only). Make manual runs (via /ctx-dream) wait up to 60 seconds (polling every 2 seconds) for a busy domain lease to free, while scheduled ticks exit immediately. lastRunAt advances only on success (so a failed run retries). The memory-maintenance reader tasks (map-memories, verify, verify-broad) and classify-memories are non-agentic: the host renders one prompt, a locked read-only (or zero-tool) agent emits ONE XML manifest, and the host parses + applies the DB writes (with fail-closed parsing: a missing root tag strictly rejects the output instead of applying a truncated prefix). Tasks:

  • map-memories — map each memory to its backing files (or a file-independent sentinel) so verify has a per-memory gate target; the locked read-only dreamer-memory-mapper agent reads code + path-seeds and emits a <map> manifest. Host-chunked, resumable; mostly a one-time backfill then a cheap trickle for new (unmapped) memories.
  • verify / verify-broad — the dreamer-memory-mapper agent re-checks mapped memories against code and emits a <verify> manifest (verified / update / archive), applied host-side via the cache-neutral mutation-log. verify gates per-memory on each memory's own verified_at (a mapped file changed since that memory was checked); partial progress sticks, so a timed-out run banks what it checked. verify-broad ignores the change gate and re-checks the whole mapped pool. File-independent + unmapped memories are excluded (curate + age decay own the file-independent ones). curate — whole-pool hygiene (consolidate / tighten / archive, agentic, gateless, weekly). Cross-category merges are structurally rejected.
  • classify-memories — score importance/scope/shareable as a zero-tool single-shot transform: the dreamer-classifier agent emits a <classify> manifest the host applies column-only via setMemoryClassification (cache-neutral). Stages on pool size (10/100): skip <10, full pool ≤100, new/changed (classified_at NULL) + stratified anchors >100. Shareability fail-closed via hasShareabilitySensitiveText.
  • retrospective — learn from user-friction: a cheap LLM gate over the project's new user messages (cross-session scan via dreamer/retrospective-raw-provider.ts, opencode.db read-only / Pi JSONL), then a ctx_search-only restricted child emits XML <learnings> that the host validates + routes (project memory / user-observation candidate) in dreamer/retrospective-learnings.ts. The scan is bounded per run (RETROSPECTIVE_MAX_SESSIONS_PER_RUN) and explicitly excludes hidden subagent sessions to prevent task prompts from triggering the friction gate.
  • Plus maintain-docs / promote-primers / refresh-primers / evaluate-smart-notes / review-user-memories (config-gated). Background tasks never force a prompt-cache materialization — their writes ride the next natural bust. The dreamer.inject_docs configuration flag (default true) controls the injection of project documents (ARCHITECTURE.md and STRUCTURE.md) into the m[0] <project-docs> block. If set to false, the block is omitted and its hash stored as "".

Other subsystems

  • Synthetic-todowrite: tool.execute.after captures todo state to last_todo_state (pure DB write). On a cache-busting pass, postprocess injects a synthetic tool_use/tool_result pair (call_id = mc_synthetic_todo_<sha256(state)[:16]}) into the latest assistant message, AFTER tagging so it's never dropped. Defer passes rebuild from the persisted state_json for byte-identity. To keep the prefix byte-stable across tail growth, a None-anchor synthetic todo pair is emitted immediately after m0/m1 and before the tail loop. On coverage folds where the synthetic-todo anchor is folded out of the tail, the pair's call_id and frozen bytes are kept while relocating its anchor_mid to the tail end (re-anchoring only rides coverage-advancing busts).
  • ctx_reduce nudges: Channel 1 appends a <system-reminder> to tool outputs in tool.execute.after (persisted to OpenCode's DB → replays for free). Channel 2 delivers a one-shot synthetic-user ceiling nudge at step boundaries via the in-process client (promptAsync with synthetic: true). Both gate on ctx_reduce actually being in the session's tool allow-list. Trigger math: severity over the working window, reclaimable ≥ usable/3, protected tags excluded.
  • Smart-drops (opt-in): Gated by the smart_drops config flag. Operates on cache-busting transform passes to reclaim additional context by targeting obsolete tool outputs rather than just the oldest:
    • Spent control-plane outputs: Drops older todowrite (keeping the newest 1), older ctx_reduce (keeping the newest 5), zero-value meta tools (like bash_status/bash_kill), and dismissed ctx_note actions.
    • Superseded edits: For older edit/write calls to a file that has since been edited again, it compresses the tool call to edit_marker mode. This mode preserves the filePath parameter verbatim and a region-hint prefix of the diff (length 40), while replacing the bulky output with [dropped §N§], keeping the model aware of which files and regions it previously modified. Both types of drops deduplicate with age-based reclaims and freeze their drop mode (full, truncated, or edit_marker) in the tags.drop_mode column for deterministic replay on later passes.
  • Tiered emergency drop (≥85%): target-headroom eviction down to fixedFloor + 0.30 × (ceiling − fixedFloor), tools oldest-first across tiers (T3 misc → T2 edit/search → T1 navigation), newest-20% recency reserve on T1/T2. floorTags (full active set, for floor accounting) vs tags (droppable candidates). last_emergency_input_sample is the idempotence latch. The newest-20 dropped tool calls keep a [dropped §N§] skeleton (the tool_use survives, output replaced) so provider tool-pairing holds; older drops are fully removed.
  • Compaction markers: inject an OpenCode-compatible compaction boundary so filterCompacted stops at the historian's last compartment, shrinking the transform-input array. The marker move is deferred from historian publish into the next materializing pass (one bust covers both the <session-history> rebuild and the boundary advance); CAS-guarded, restart-safe.
  • Content stripping (strip-content.ts, caveman.ts, sentinel.ts): stateless strip functions + deterministic in-place sentinel replacement + persisted watermarks. Provider-aware: empty-content sentinels only stay empty for providers that accept them (modelAcceptsEmptyContent); others get a [dropped] placeholder (e.g. Copilot/Bedrock break tool adjacency on empty parts — #135).
  • Message / git-commit indexes: Maintain an FTS5 raw-message index outside the search hot path (via async reconciliation + live message.updated events) and a HEAD-only non-merge git-commit corpus populated by the dream timer. For directories that are not git repositories or are empty (no commits yet), the sweep coordinator future-dates the last sweep time to park them on a 24-hour re-probe cooldown to avoid error log flooding.
  • Commit-detection utility: A unified detection helper (src/shared/commit-detection.ts) parses git commit hashes (7-12 hex chars) paired with commit-action verbs (commit, cherry-pick, merge, rebase) to detect commit boundaries consistently across the historian trigger, OpenCode note-nudge, and Pi note-nudge.
  • System-prompt injection (system-prompt-hash.ts): injects only the Magic Context guidance text + a frozen Today's date: line (per-session sticky, updated only on cache-busting passes). Guidance is gated by ctx_reduce availability (resolved from the tools map of the session's first user message). While the verdict is provisional (before the first user message has been processed/persisted), the guidance block still renders using a fail-open default, but the computed system prompt hash is not written to the database to prevent cache-busting flips when the first user message freezes the verdict. Adjunct blocks (<project-docs> / <user-profile>) are NOT here — they moved into m[0]/m[1] so the system prompt stays maximally cache-stable. Skipped entirely for OpenCode's internal title/summary/compaction agents and for hidden child sessions (detected by the magic-context- title prefix).
  • TUI ↔ server RPC: localhost server on an ephemeral port (published to session_meta); the TUI plugin reads all data via RPC (no direct SQLite, avoids lock contention).
  • Diagnostics & Redaction: The CLI doctor command and RPC endpoints sanitize file paths and credentials (src/shared/redaction.ts) to prevent leaking secrets in logs/dumps. Numeric redaction preserves non-secret scalars (like token counts or boolean flags) to keep diagnostic metrics readable while masking high-entropy strings.
  • Mid-turn release valve: OpenCode's mid-turn state (finish="tool-calls") normally blocks tail rewrites while tools run. A release valve (hasNewerRealUserMessage) ensures that if a turn stalls, a subsequent real user message breaks the mid-turn lock, whereas synthetic user messages (like Channel 2 nudges) preserve it.

Storage & migrations

storage-db.ts creates the schema and runs versioned migrations (migrations.ts, currently v1–v50). LATEST_SUPPORTED_VERSION is a schema fence — it MUST be bumped with every new migration (a unit test asserts it equals the highest migration), and a stale value makes the DB refuse to open after the migration applies. Schema helpers ensureColumn() + healAllNullColumns() (defined in storage-schema-helpers.ts to prevent cycles between storage-db and migrations) backfill upgraded DBs even if a migration row is lost. New session-scoped tables must be added to clearSession(). A bulletproof MAGIC_CONTEXT_TEST_DATA_DIR guard keeps the test suite off the live DB (running bun test once migrated a live DB and fail-closed running binaries). SQLite binds must use SPREAD positional args, never the array form (bun:sqlite binds a lone array positionally; node:sqlite reads it as named params and throws).

Session modes

Two effective modes; the heavier features (historian, nudges, adjunct injection) are gated, while tag/drop plumbing stays on everywhere. The primary reduce surface is additionally gated by the session's actual ctx_reduce tool availability, so agents that deny the tool also lose §N§ prefixes and reduce nudges.

Feature Primary sessions Subagents
Tag DB records ✓ ✓
§N§ prefix injection + ctx_reduce tool ✓ when ctx_reduce is available ✓ when ctx_reduce is available
Historian / compartments / decay / m[0]m[1] ✓ ✗
Channel 1 nudge ✓ when ctx_reduce is available ✓ when ctx_reduce is available
Channel 2 nudge ✓ when ctx_reduce is available ✓ when ctx_reduce is available
Synthetic-todowrite / auto-search ✓ ✗
Heuristic tool drops at execute ✓ once/turn ✓ every execute pass
85% force-materialize / 95% block ✓ ✗ (overflow path only)
Caveman text compression opt-in ✗

Subagents run heuristic drops on every execute pass (no once-per-turn guard) because a long subagent run is effectively one parent turn and would otherwise starve; they have no provider-cache reuse to protect.

Error handling

Fail closed when storage is unavailable (better to disable than silently overflow the prompt). Fail open in per-turn handlers (log and skip). Wrap the outer transform so transient SQLITE_BUSY/SQLITE_LOCKED never crash the prompt loop (#23). overflow-detection.ts parses provider context-overflow errors (Anthropic / OpenAI / Copilot) and persists the detected limit so later passes use the lower value. Subagent model fallback (model-suggestion-retry.ts) iterates the chain on retryable failures; abort/timeout/context-overflow short-circuit. Hidden agents carry a steps/maxSteps cap and are aborted via session.abort on timeout so a weak local model can't loop forever (#154).

Tag identity

Each tags row is one taggable source-content unit (message, file, or tool). message/file tags key on (session_id, message_id) (synthetic content id). tool tags key on a COMPOSITE (session_id, callID, tool_owner_message_id) — because OpenCode reuses a callID counter per assistant turn, so the same read:32 recurs across turns; including the owning assistant message id gives each invocation its own row (migration v10). Owner derivation: invocation parts own themselves; result parts pop a FIFO of unpaired invocations; a result whose invocation was compacted away falls back to the nearest prior persisted owner. The same composite keying mirrors in the drop queue and heuristic cleanup so dropped keys match what the tagger persisted. Per-tag token counts (token_count / input_token_count / reasoning_token_count) are computed once on tag insert and summed for sidebar / boundary / nudge math (off the hot path).