Skip to content

fix(memos-local-plugin): two-phase migration to prevent crash-loop on large databases - #1789

Closed
chiefmojo wants to merge 7 commits into
MemTensor:mainfrom
chiefmojo:fix/large-db-migration-crash
Closed

chiefmojo wants to merge 7 commits into
MemTensor:mainfrom
chiefmojo:fix/large-db-migration-crash

Conversation

@chiefmojo

Copy link
Copy Markdown
Contributor

Problem

Migration 007 (namespace-visibility) in v2.0.2–v2.0.5 runs UPDATE … SET share_scope='private' plus CREATE INDEX on the traces table inside a single db.tx(). On databases larger than ~500MB, this exceeds the host gateway's kill timeout. SQLite rolls back the entire transaction — including the schema_migrations INSERT — so migration 007 is never recorded and the bridge restarts into the same hang forever.

Small databases (~43MB) complete within the timeout, which is why this only manifests on larger installs. Tested against a 687MB database with ~98,000 traces.

Fix

Two-phase migration:

  • Phase 1 (inside transaction, milliseconds): ADD COLUMN only on all 12 namespace tables, plus DROP INDEX uq_skills_name. The schema_migrations record for v7 commits here.
  • Phase 2 (after migration loop, outside any transaction): Batched UPDATE in 2,000-row chunks (each its own implicit transaction) for share_scope backfill, then CREATE INDEX IF NOT EXISTS for all 18 owner/share indexes. ensureNamespaceColumns is called unconditionally on every boot so new tables in the namespace list get their columns.

Restart safety: If the bridge is killed during Phase 2, the v7 schema_migrations record survives (Phase 1 committed). Next boot skips Phase 1 entirely and resumes Phase 2 where it left off. The crash-loop is broken.

Verification

  • Confirmed the DB on the affected instance had exactly migrations 1–6 applied — migration 007 was never committed despite hundreds of boot attempts.
  • After the fix, Phase 1 committed in 2.4 seconds. Phase 2 backfill and index creation complete across bridge restarts.
  • Bridge boots cleanly with all 18 indexes created and pipeline.ready.

Related

Closes #1787

… large databases

Migration 007 (namespace-visibility) runs UPDATE ... SET share_scope
and CREATE INDEX on the traces table inside a single db.tx(). On
databases larger than ~500MB, this exceeds the host gateway kill
timeout, SQLite rolls back the entire transaction (including the
schema_migrations INSERT), and the bridge restarts into the same
hang forever.

This splits the migration into two phases:
  Phase 1 (inside transaction, ms): ADD COLUMN only on 12 namespace
    tables plus DROP INDEX uq_skills_name. The schema_migrations
    record commits here.
  Phase 2 (after migration loop, outside any transaction): Batched
    UPDATE in 2,000-row chunks (each its own implicit transaction)
    for share_scope backfill, then CREATE INDEX IF NOT EXISTS for
    all 18 owner/share indexes. Phase 2 also calls
    ensureNamespaceColumns unconditionally so new tables added to
    the namespace list get their columns on every boot.

Restart-safe: if the bridge is killed during Phase 2, the v7
schema_migrations record survives (Phase 1 committed). Next boot
skips Phase 1 entirely and resumes Phase 2 where it left off.

Closes MemTensor#1787
Memtensor-AI and others added 5 commits June 14, 2026 17:24
docs(memos-local-plugin): clarify install path and stale dir names (MemTensor#1540)

The README's 'Quick start' section told users to use install.sh instead
of npm install, but the warning was buried and users still tried
'npm install -g @memtensor/memos-local-plugin' first. The reporter in
MemTensor#1540 encountered this on a Hermes deployment.

This change:

- Promotes the 'do not run npm install -g' notice to a prominent
  IMPORTANT callout explaining why global install is wrong (no
  agent-home deploy, no config.yaml, no bridge/viewer) and that the
  tarball intentionally ships built artifacts only.
- Adds a Troubleshooting subsection covering the two specific symptoms
  in the bug report: the 'package not found' misread, and the stale
  web/ and site/ directory names (web/ is now viewer/, site/ was
  removed by commit 26e7e3d).
- Mentions install.ps1 for Windows alongside install.sh.
- CHANGELOG: record the docs fix and reference MemTensor#1540.

Documentation-only change; no code or runtime behavior touched.

Co-authored-by: MemOS AutoDev <autodev@memtensor.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>
…_() got an unexpected keyword a (MemTensor#1889)

fix: remove invalid chunker parameter from SystemParser test instantiation

- SystemParser.__init__() signature changed to (embedder, llm=None)
- Test was still passing chunker=None causing TypeError
- Fixes all 5 failing tests in test_system_parser.py

Fixes MemTensor#1888

Co-authored-by: MemOS AutoDev <autodev@memos.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>
…tributeError when given None (MemTensor#1884)

* test: add comprehensive tests for clean_json_response (issue MemTensor#1525)

- Add test suite in tests/mem_os/test_format_utils.py
- Cover None input ValueError with diagnostic message
- Cover markdown removal, whitespace stripping, edge cases
- Verify fix for AttributeError when LLM returns None

* style: format clean_json_response tests

---------

Co-authored-by: MemOS AutoDev <autodev@memos.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>
…date_cube_access — fails for ev (MemTensor#1903)

fix: validate current user not target in share_cube_with_user (MemTensor#1901)

share_cube_with_user(cube_id, target_user_id) called
_validate_cube_access(cube_id, target_user_id), but the validator
signature is (user_id, cube_id). The cube_id therefore landed in the
user_id slot and _validate_user_exists raised
"User '<cube_id>' does not exist or is inactive" for every well-formed
call, making the API unusable.

The in-code comment "Validate current user has access to this cube"
already documented the correct intent: the sharing user (self.user_id)
must have access to the cube being shared, not the target. Switch the
call to self._validate_cube_access(self.user_id, cube_id). The target
user's existence is independently checked on the next line via
validate_user(target_user_id), so that path is unchanged.

Add regression tests in tests/mem_os/test_memos_core.py that pin down:
- validate_user_cube_access is consulted with (self.user_id, cube_id),
- add_user_to_cube is called with (target_user_id, cube_id) on success,
- a missing target raises "Target user '<id>' does not exist".

Closes MemTensor#1901

Co-authored-by: MemOS AutoDev Bot <autodev@memtensor.local>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>
@Memtensor-AI
Memtensor-AI changed the base branch from dev-20260604-v2.0.19 to dev-v2.0.22 July 1, 2026 13:16
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Automated Test Results: FAILED

Cloud test-engine rerun against dev-v2.0.22 failed.

  • Run: tr-2cf1d73f-bc0 on cloud test-engine 10011

  • Verdict: fail_code_bug

  • Summary: Tests failed. Failed cases: test -- tests/unit/storage/migrator.test.ts tests/unit/storage/traces-count.test.ts

  • memos_local_plugin/unit: 7 passed, 1 failed, 0 skipped

Failed cases:

  • test -- tests/unit/storage/migrator.test.ts tests/unit/storage/traces-count.test.ts: should return accurate count > 500
    AssertionError: expected 500 to be 50 // Object.is equality

  • Expected

  • Received
  • 50
  • 500

❯ tests/unit/storage/traces-count.test.ts:76:27
74| // Verify list with no limit keeps the default page size.
75| const listed = repo.list({});
76| expect(listed.length).toBe(50);
| ^
77|
78| // Verify explicit high-limit requests can fetch beyond the old 50…

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

Do not merge until this is fixed and cloud tests pass.

@CarltonXiang
CarltonXiang deleted the branch MemTensor:main July 3, 2026 07:25
@syzsunshine219 syzsunshine219 reopened this Jul 3, 2026
@syzsunshine219
syzsunshine219 changed the base branch from dev-v2.0.22 to main July 3, 2026 08:23
@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes area:memory 记忆存储、检索、更新、召回逻辑 labels Jul 8, 2026
@Memtensor-AI
Memtensor-AI requested a review from bittergreen July 8, 2026 11:44
@Memtensor-AI Memtensor-AI added area:database graph_db + vector_db | 图数据库与向量数据库 and removed area:memory 记忆存储、检索、更新、召回逻辑 labels Jul 9, 2026
@Memtensor-AI Memtensor-AI assigned whipser030 and wustzdy and unassigned bittergreen Jul 9, 2026
@Memtensor-AI Memtensor-AI removed the area:database graph_db + vector_db | 图数据库与向量数据库 label Jul 9, 2026
@shinetata

Copy link
Copy Markdown
Collaborator

Thanks for the genuinely thorough investigation here, @chiefmojo — the bisect table, the 687MB repro, and the root-cause write-up on the migration-007 transaction timeout are all excellent and much appreciated.

I'm going to close this PR, because the crash-loop it targets (#1787) has already been fixed on main by #1871, and #1787 is now closed.

The merged fix took a simpler route to the same goal:

  • It removed the O(n) UPDATE traces SET share_scope='private' WHERE share_scope IS NULL entirely. That bulk rewrite — re-validating the JSON CHECK constraints on every row — was the actual cause of the CPU-bound hang you diagnosed.
  • share_scope is now added via ADD COLUMN ... DEFAULT 'private' (O(1) since SQLite 3.35), and the read path already treats a NULL scope as private through COALESCE(share_scope,'private'), so no backfill pass is needed.
  • On top of that, 001-initial.sql now folds in the namespace/visibility columns and their indexes, so on fresh installs migration 007 is effectively a no-op.

Given that, this PR can't be merged as-is:

  1. It's currently behind main and rewrites the exact function (ensureNamespaceVisibilityColumns) that Fix #1787: v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases > #1871 already replaced, so it would conflict heavily on rebase.
  2. It also changes tests/unit/storage/traces-count.test.ts (the 500 → 50/600 expectations), which is unrelated to the migration fix and isn't backed by a matching source change. On current main, list() still defaults to and caps at 500, so that assertion fails — which is exactly what the automated run flagged (expected 500 to be 50). This looks like a stray rebase artifact.
  3. The migrator.test.ts change inverts the design chosen in Fix #1787: v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases > #1871: it asserts phase-2 backfills NULL → 'private', whereas main intentionally leaves NULL in place and coalesces at read time.

If you think the one piece of net-new hardening here is still worth having — moving CREATE INDEX outside the migration transaction so index builds on very large tables can't hold the bootstrap tx — that would be a welcome, focused follow-up. A small PR rebased on the latest main that (a) only touches the index-creation path and (b) drops the traces-count.test.ts change would be easy to review and land.

One more thing that surfaced while digging into this: the #1787 regression test on main is currently red on its own — it inserts legacy role/embedding columns that no longer exist in the folded traces schema (SqliteError: table traces has no column named role). We'll fix that separately, but your rewrite of that test was actually pointing at the correct schema.

Thanks again for the deep debugging here — it genuinely helped confirm the root cause.

@shinetata shinetata closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB

10 participants