Skip to content

fix: [memos-local-plugin] Dreaming background processing starves Gateway event loop, causing WebSocket timeouts #1808

Closed

Description

@tianxin8206

Pre-submission checklist | 提交前检查

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
  • This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题

Bug Description | 问题描述

The memos-local-plugin's "dreaming" feature performs LLM inference calls synchronously on the Gateway's main Node.js event loop. When processing orphan or dirty episodes, this blocks the event loop for extended periods (observed eventLoopMax of 3.5s+), causing WebSocket handshake timeouts for OpenClaw TUI and Control UI connections.

How to Reproduce | 如何重现

Steps to Reproduce

  1. Enable memos-local-plugin with dreaming enabled
  2. Accumulate a large number of episodes (30K+ traces in our case)
  3. Have one or more episodes enter an orphan/dirty state (e.g., r_task = NULL)
  4. Restart the openclaw gateway
  5. Attempt to connect via TUI or Control UI during dreaming processing

Observed Behavior

  • OpenClaw Gateway event loop blocked for 3-5+ seconds during dreaming LLM calls
  • WebSocket upgrade requests timeout (default 3s budget for read probe)
  • TUI reports "cannot connect to gateway"
  • Control UI login page fails to load
  • Gateway logs show repeated Read probe: failed - timeout errors
  • Orphan episodes with failed LLM calls are retried indefinitely with no backoff

Environment | 环境信息

  • OpenClaw version: 2026.5.22
  • OS: openeuler (systemd user service)
  • memos.db size: 324MB, 30K+ traces
  • Model provider: DeepSeek (v4-flash for dreaming)

Additional Context | 其他信息

No response

Willingness to Implement | 实现意愿

  • I'm willing to implement this myself | 我愿意自己解决
  • I would like someone else to implement this | 我希望其他人来解决

Activity

  1. Memtensor-AI commented on May 27, 2026

    @Memtensor-AI
    Collaborator

    Sentinel Classification

    Field Value
    Module plugin
    Priority P2:normal
    Auto-fixable No

    Reason: Performance issues require profiling and benchmarking that the agent cannot perform.

    This issue has been classified and requires human attention.


    Classified by memos-sentinel

  2. tianxin8206 commented on May 27, 2026

    @tianxin8206
    Author

    Additionally, this bug also causes a sharp surge in LLM API calls. Meanwhile, the dreaming feature leads to excessive token usage. This plugin was originally designed to reduce token consumption, yet it ends up driving higher token costs in practice.

  3. deleted a comment from MaiHHConnect on May 28, 2026
  4. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 29, 2026
  5. self-assigned this
    on Jun 30, 2026
  6. Memtensor-AI commented on Jun 30, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: e6c7b853d623b11d
    Working branch: bugfix/autodev-1808
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  7. Memtensor-AI commented on Jun 30, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task e6c7b853d623b11d completed.

    Summary: Fix issue #1808 — memos-local-plugin no longer starves the OpenClaw Gateway event loop during plugin bootstrap.

    Root cause: core.init() in apps/memos-local-plugin/core/pipeline/memory-core.ts synchronously awaited the full reflect → reward → L2-induction chain for every orphan / dirty closed episode in SQLite. On the reporter's 30K-trace / 324MB database that pinned the Node.js event loop for 3-5s+, causing OpenClaw's 3s WebSocket read probe to time out so TUI and Control UI clients could not connect. A secondary symptom — failed LLM rescores being retried indefinitely with no backoff — kept the loop alive across restarts and the 10-minute periodic rescan.

    Fix delivered on bugfix/autodev-1808 (commit 26c96b0): (1) Split init() into a fast synchronous orphan / dirty classification and a background startupRecoveryPromise that runs the slow reflect/reward chain off the Gateway's critical path; (2) Added an optional MemoryCore.waitForStartupRecovery?() method so tests and one-shot batch tools can opt back into the historic "await everything" semantics — the promise is contracted never to reject; (3) shutdown() now awaits the background promise before tearing down storage so SQLite is not closed mid-flush; (4) Introduced per-episode failure tracking under meta.rewardDirty — after MAX_DIRTY_REWARD_ATTEMPTS=3 consecutive failed automatic rescores the episode enters an exponential backoff (1h → 24h cap) and the init + periodic scans skip it until the cooldown elapses; manual feedback / runManually still rescore unconditionally.

    Tests: tests/unit/pipeline/memory-core.test.ts now 32/32 green (3 existing orphan/dirty tests updated to await the new promise; 4 net new tests covering init latency contract, backoff filter, failure-counter logic, and shutdown synchronization). The pipeline + bridge + adapters slice passes 127/127. Full unit sweep is 1044/1047 — the 2 unrelated failures in tests/unit/storage/traces-count.test.ts and migrator.test.ts reproduce identically on the unmodified base branch (verified via stash roundtrip) and are out of scope for this bug. TypeScript tsc --noEmit is clean for both tsconfig.json and tsconfig.build.json. The Python backend, LLM client timeouts, and Makefile/CI are unchanged.

    Base branch: main
    Branch: bugfix/autodev-1808
    Commit: 26c96b0
    PR: #2002
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  8. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 30, 2026
  9. Memtensor-AI commented on Jul 1, 2026

    @Memtensor-AI
    Collaborator

    🤔 AutoDev task e6c7b853d623b11d needs clarification to proceed:
    The test failure list provided in your previous response does not match any existing tests in this repository. Could you please clarify:

    1. Issue scope: Is this issue about memos-local-plugin (TypeScript, under apps/memos-local-plugin/) or the main MemOS library (Python, under src/memos/)?

    2. Test failures: The mentioned tests (test_out_of_range_rejected_or_clamped_to_valid, test_resolution_order_matches_documented_precedence, etc.) do not exist in the current codebase. Are these:

      • Tests that need to be written as part of the fix?
      • Tests from a different branch or PR?
      • Tests from a different repository?
    3. vector_scan.max_age: This configuration parameter does not exist in the current codebase. Could you provide:

      • The file path where this config should be validated?
      • The design doc or spec for this feature?
    4. Original issue fix: [memos-local-plugin] Dreaming background processing starves Gateway event loop, causing WebSocket timeouts #1808: The task description mentions "dreaming background processing starve" but I cannot find references to issue fix: [memos-local-plugin] Dreaming background processing starves Gateway event loop, causing WebSocket timeouts #1808 in the git history. Could you clarify:

      • What is the actual symptom or bug report?
      • How to reproduce the issue?
      • Which component is affected?
        The suggested default is only used if a maintainer explicitly confirms it.
        Please reply with one of the following commands (only issue author or repo collaborators):
    • /autodev confirm yes — agree and continue with the suggested default assumption
    • /autodev confirm no <additional info> — provide more details
    • /autodev cancel — cancel this task
      If there is no reply within 24h, AutoDev will cancel this task using the same outcome as /autodev cancel. It will not continue with the suggested default automatically.
  10. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    ⏰ AutoDev task e6c7b853d623b11d received no /autodev confirm reply within 24h. AutoDev cancelled this task using the same outcome as /autodev cancel; it will not continue with the suggested default automatically.

  11. added a commit that references this issue on Jul 2, 2026
    4e16c68
  12. added and removed
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    on Jul 2, 2026
  13. added 3 commits that reference this issue on Sep 18, 2026
    b2823dc
    5cee4a5
    568336f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:pluginOpenClaw & Hermestypes:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions