Skip to content

bug: Thread leak in searcher._retrieve_paths causes container thread exhaustion #1273

Closed as not planned

Description

@brentkearney

Pre-submission checklist | 提交前检查

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
  • This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题

Bug Description | 问题描述

The search pipeline creates a new ContextThreadPoolExecutor on every request in four methods within searcher.py:

  • _retrieve_paths (line 357, max_workers=5)
  • _retrieve_from_long_term_and_user (line 638, max_workers=3)
  • _retrieve_from_tool_memory (line 791, max_workers=2)
  • _deduplicate_rawfile_results (line 1153, up to max_workers=10)

Each executor is used within a with block, which calls shutdown(wait=True) on exit. If any submitted task hangs — e.g., a slow Neo4j query, an unresponsive embedding API, or a network timeout — the executor never shuts down. The threads remain alive, and the next request creates another pool.

Over time, this causes unbounded thread accumulation. In our deployment we observed 8,744 threads in the memos-api container, at which point:

  • /search returns HTTP 200 with empty results (the can't start new thread error is caught silently)
  • /chat returns HTTP 503 (it calls search internally but doesn't handle the thread error gracefully)
  • Even docker exec fails — OpenBLAS cannot create pthreads

Note: Adding timeout to future.result() alone does not fix this. The timeout only skips waiting for the result — the thread itself keeps running, and shutdown(wait=True) still blocks until it finishes. The threads still accumulate.

How to Reproduce | 如何重现

  1. Deploy MemOS with Neo4j backend in Docker
  2. Send sustained /search traffic over hours/days
  3. If any downstream dependency (Neo4j, embedding API) experiences intermittent slowness, threads accumulate
  4. Monitor with: docker exec <container> cat /proc/1/status | grep Threads

Environment | 环境信息

  • Python version: 3.11
  • Operating System: Linux (Raspberry Pi / aarch64)
  • MemOS version: v2.0.8 (also present in v2.0.9 — searcher.py unchanged)
  • Backend: Neo4j
  • Deployment: Docker

Additional Context | 其他信息

Suggested Fix:

Use a shared, class-level ContextThreadPoolExecutor instead of creating a new one per request. The Searcher class already follows this pattern with _usage_executor (line 73). Adding a second shared pool for search operations would bound thread count regardless of request volume or downstream latency:

# In __init__:
self._search_executor = ContextThreadPoolExecutor(max_workers=10, thread_name_prefix="search")

# In each method, replace:
#   with ContextThreadPoolExecutor(max_workers=N) as executor:
# with:
#   executor = self._search_executor

All future.result() calls should also include a timeout (e.g., 30s) as a safety measure.

Workarounds:

  • Set pids_limit in docker-compose.yml to fail fast instead of consuming all system resources
  • Set OPENBLAS_NUM_THREADS=1 to reduce per-thread overhead from numpy

Willingness to Implement | 实现意愿

  • I'm willing to implement this myself | 我愿意自己解决
  • I would like someone else to implement this | 我希望其他人来解决

Activity

  1. added 3 commits that reference this issue on Mar 18, 2026
    668820a
    4407746
  2. CaralHsi commented on Mar 20, 2026

    @CaralHsi
    Collaborator

    Fixed in PR #1308 - Fixed thread-pool leakage in searcher by reusing a shared executor and adding safer timeouts/cancellation.
    已在 PR #1308 中修复 - 已修复 searcher 线程池泄漏问题:复用共享 executor,并增加更安全的超时/取消处理,避免线程无限增长。

  3. brentkearney commented on Mar 20, 2026

    @brentkearney
    ContributorAuthor

    Update: After continued testing, we identified two additional thread exhaustion vectors beyond searcher.py. PR #1308 addresses the original searcher issue but does not cover these.

    1. Scheduler thread pool default of 10,000

    In src/memos/api/config.py (line 840):

    "thread_pool_max_workers": int(
        os.getenv("MOS_SCHEDULER_THREAD_POOL_MAX_WORKERS", "10000")
    )

    The schema default in general_schemas.py is 50, but config.py overrides it with "10000" as the os.getenv fallback. Any deployment without this env var explicitly set will allocate a 10,000-worker thread pool.

    Fix: Change the fallback from "10000" to "50" to match the schema default.

    2. Unbounded executor in simple_struct.py

    In src/memos/mem_reader/simple_struct.py (line 673), a ContextThreadPoolExecutor() is created with no max_workers argument and a new pool per request:

    with ContextThreadPoolExecutor() as executor:
        futures = [
            executor.submit(processing_func, scene_data_info, info, mode=mode)
            for scene_data_info in list_scene_data_info
        ]

    In our deployment, this was the actual crash site — error logs show simple_struct.py:684 - _read_memory - Task failed with exception: can't start new thread.

    Fix: Set an explicit max_workers bound, or use a shared instance like the searcher.py fix.

  4. self-assigned this
    on Mar 23, 2026
  5. github-actions commented on Apr 26, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  6. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Apr 26, 2026
  7. MaiHHConnect commented on Apr 27, 2026

    @MaiHHConnect
  8. removed
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Apr 27, 2026
  9. github-actions commented on May 27, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  10. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on May 27, 2026
  11. 20 remaining items

  12. removed
    area:databasegraph_db + vector_db | 图数据库与向量数据库
    on Jul 13, 2026
  13. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jul 21, 2026
  14. added
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    ai:codingAI agent is coding | AI 正在编码
    and removed
    status:readyReady for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发
    on Jul 21, 2026
  15. Memtensor-AI commented on Jul 21, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: b1848bec14f3201e
    Working branch: feature/autodev-1273-20260721042041846
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  16. Memtensor-AI commented on Jul 21, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task b1848bec14f3201e completed.

    Summary: Fixes #1273 (thread leak in Searcher retrieval paths). Searcher previously opened a fresh ContextThreadPoolExecutor per /search request in four methods (_retrieve_paths, _retrieve_from_long_term_and_user, _retrieve_from_tool_memory, _deduplicate_rawfile_results); any hung sub-task blocked shutdown(wait=True) and leaked threads, growing container thread count without bound (reported 8,744 threads leading to empty /search results, /chat 503s, and OpenBLAS pthread_create failures).

    The fix pre-allocates four class-level shared ContextThreadPoolExecutor instances in Searcher.__init__ (mirroring the original per-site max_workers 5/3/2/10, with distinct thread_name_prefix for observability), rewrites the four call sites to submit to the shared executor without a with block, and adds a SEARCH_FUTURE_RESULT_TIMEOUT = 30.0 module constant applied to every future.result() / as_completed() with warning-level exception handling. Total worker thread ceiling per Searcher drops from unbounded to 20 regardless of QPS. Four separate pools (not one shared) prevent the deadlock that would arise from nested submits between _retrieve_paths and its sub-paths.

    Tests: 4 new regression tests lock in shared executor identity across calls, absence of naked .result(), and survival under a 5x-timeout slow sub-task. All 4 pre-existing behavioural tests still pass; wider tests/memories/textual/ suite is 62/62 green. Ruff check + format both clean. Two pre-existing tests/memories/activation/test_kv.py failures (transformers DynamicCache API drift) are unrelated and reproduced on the stashed tree.

    Files touched: src/memos/memories/textual/tree_text_memory/retrieve/searcher.py, tests/memories/textual/test_tree_searcher.py. No API schema, config, or dependency changes.

    Base branch: main
    Branch: feature/autodev-1273-20260721042041846
    Commit: cf0abb7
    PR: #2134
    Assigned to: @bittergreen
    Reviewers: @bittergreen

  17. added
    ai:testingAI agent is running tests | AI 正在运行测试
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Jul 21, 2026
  18. github-actions commented on Aug 29, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  19. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Aug 29, 2026
  20. github-actions commented on Sep 6, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务ai:testingAI agent is running tests | AI 正在运行测试area:memory记忆存储、检索、更新、召回逻辑status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions