Skip to content

Collapse worker storage and search into HelixDB (beta) - #1

Draft
Gand4lf wants to merge 22 commits into
mainfrom
agent/helixdb-migration
Draft

Collapse worker storage and search into HelixDB (beta)#1
Gand4lf wants to merge 22 commits into
mainfrom
agent/helixdb-migration

Conversation

@Gand4lf

@Gand4lf Gand4lf commented Aug 11, 2026

Copy link
Copy Markdown

Summary

  • Replaces the SQLite + Chroma runtime split with one embedded, disk-backed HelixDB database.
  • Stores canonical entities, graph relationships, BM25/vector indexes, sync queues, and cursor state in one transactional system.
  • Consolidates multi-step repository and search-strategy paths into direct batched Helix queries.
  • Keeps SQLite and Chroma only inside the isolated one-time migration path, with source snapshots and post-import validation.
  • Pins Helix SDK 3.0.4 and embedded runtime 0.3.3.

This removes the dual-write, dual-search, and reconciliation machinery. There is now one storage model, one search path, and one source of truth.

Beta status

This is intentionally beta:

  • Helix embedded does not yet provide supported backup/checkpoint/restore.
  • The runtime requires a single active writer; a newer client fences the older client.
  • Validation was performed on macOS arm64. Windows and PostgreSQL-backed release matrices were not exercised.
  • Clean-room install, upgrade, and rollback remain release gates.

Validation

  • Typecheck passed.
  • Full suite: 2,457 passed, 0 failed, 22 environment-dependent skips.
  • Live Helix runtime: 84 assertions passed.
  • Live SQLite/Chroma migration passed.
  • Multi-process fencing and close/reopen durability passed.

ChenglinWei97 and others added 22 commits July 31, 2026 11:21
…hedotmack#3468)

Adds a ninth observation type to the code mode: `sensitive` — information
that isn't quite private, but that you wouldn't want leaking into further
content development in the wrong context (internal URLs, unreleased plans,
personal details, business metrics, client or partner names). These fire a
Telegram notification by default, alongside security_alert.

The notifier already matched obs.type against CLAUDE_MEM_TELEGRAM_TRIGGER_TYPES,
so wiring notifications needed only the new type in the default trigger set
plus its own emoji.

- plugin/modes/code.json: new `sensitive` type. Also fixes pre-existing
  staleness in type_guidance, which still said "6 options" and never listed
  the security types from thedotmack#2084 — that comment is the only type prose the
  observer agent sees, so a type missing from it is invisible to the model.
- SettingsDefaultsManager: default trigger types now security_alert,sensitive,
  plus a migration for the legacy persisted value. A fresh settings.json is
  seeded with every default and persisted values win on load, so installs
  created after thedotmack#2084 had "security_alert" frozen on disk and could never
  receive the new default — the feature would have been dead on arrival for
  them. Only the exact legacy value is rewritten; customized lists are left
  alone. (Greptile P1.)
- TelegramNotifier: sensitive observations post with their own emoji
- bmp-safe: astral fallback for the new emoji (issue thedotmack#2787 failure class)
- CorpusRoutes: allow the new type in knowledge-corpus filters
- openclaw feed + weekly-digests legend: recognize the new type

Parser validation, the observer prompt type enum, context legends, and the
viewer all derive from the mode file, so they pick this up with no change.

Regression coverage for the migration: legacy-default migration, write-back
with unrelated keys preserved, customized list preserved, superset list
preserved, empty opt-out preserved, idempotency across repeated loads.
…edotmack#3513)

The installer's provider prompt now leads with CMEM Pro and shows what each
option actually costs per 1,000 observations, so the choice is made on price
rather than on brand recognition. Anthropic moves last — it is the most
expensive per observation and it bills the user's own Claude plan.

Picking CMEM Pro opens cmem.ai/pro?from=installer, waits for the key the
signup flow hands back, and writes it into settings. No new provider class:
the worker's OpenRouter client is already a generic OpenAI-compatible client
whose base URL and model come from settings, so CMEM Pro is four settings
writes (provider=openrouter, base URL, model, key). 'cmem' is a prompt-only
sentinel and never reaches settings.json, which the worker would not
understand.

Cost strings live in the new src/npx-cli/cmem-pro-costs.ts and derive from a
single TOKENS_PER_OBSERVATION constant, so re-pricing is a one-line edit.
CMEM_PRO_ORIGIN overrides the origin for walking the flow against a dev
server.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…otmack#3515)

The provider prompt quoted constants derived from a deck. Model pricing changes
constantly and old models tend to get more expensive, so those figures were
wrong within weeks of shipping:

  deepseek/deepseek-v4-flash    hardcoded $0.091/M   live $0.1418/M    +56%
  anthropic/claude-haiku-4.5    hardcoded $0.297/M   live $1.0520/M   +254%
  google/gemini-2.5-flash-lite  hardcoded $0.113/M   live $0.1039/M     -8%

Rates now come from OpenRouter's public model catalogue at prompt time, blended
at claude-mem's real 98.7/1.3 input-output split. The lookup is bounded to 3s
and falls back to last-known figures, so an offline install still gets a working
prompt. A partial catalogue response counts as a fallback rather than reporting
itself live.

TOKENS_PER_OBSERVATION goes from an assumed 30,000 to a measured 55,960 —
rounded to 56,000 — taken from 201 real observations. The old value was low by
1.87x, and since every figure derives from it, the two errors compounded.

Net effect on what the prompt tells people:

  OpenRouter        $2.73/1k -> $7.94/1k
  Gemini            $3.40/1k -> $5.82/1k
  Anthropic         $8.91/1k -> $58.91/1k

The old numbers understated every option that bills the user, which meant the
prompt argued against CMEM Pro using CMEM Pro's own figures: at 75M tokens the
advertised Anthropic rate worked out to $22.28/mo against CMEM Pro's flat $30,
so a reader doing the arithmetic would have concluded Anthropic was cheaper. The
live rate puts that same comparison at $78.90.

Anthropic's suffix changes from "billed to your Claude plan" to "from your
Claude plan". Most people choosing that option are on a Claude subscription and
spend quota, not cash; the old wording implied a bill they do not receive.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts:
#	plugin/scripts/context-generator.cjs
#	plugin/scripts/mcp-server.cjs
#	plugin/scripts/server-service.cjs
#	plugin/scripts/transcript-watcher.cjs
#	plugin/scripts/worker-service.cjs
#	plugin/ui/viewer-bundle.js
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants