Repository navigation
Description
Pre-submission checklist | 提交前检查
- I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
- I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
- This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题
Bug Description | 问题描述
The search pipeline creates a new ContextThreadPoolExecutor on every request in four methods within searcher.py:
_retrieve_paths(line 357,max_workers=5)_retrieve_from_long_term_and_user(line 638,max_workers=3)_retrieve_from_tool_memory(line 791,max_workers=2)_deduplicate_rawfile_results(line 1153, up tomax_workers=10)
Each executor is used within a with block, which calls shutdown(wait=True) on exit. If any submitted task hangs — e.g., a slow Neo4j query, an unresponsive embedding API, or a network timeout — the executor never shuts down. The threads remain alive, and the next request creates another pool.
Over time, this causes unbounded thread accumulation. In our deployment we observed 8,744 threads in the memos-api container, at which point:
/searchreturns HTTP 200 with empty results (thecan't start new threaderror is caught silently)/chatreturns HTTP 503 (it calls search internally but doesn't handle the thread error gracefully)- Even
docker execfails — OpenBLAS cannot create pthreads
Note: Adding timeout to future.result() alone does not fix this. The timeout only skips waiting for the result — the thread itself keeps running, and shutdown(wait=True) still blocks until it finishes. The threads still accumulate.
How to Reproduce | 如何重现
- Deploy MemOS with Neo4j backend in Docker
- Send sustained
/searchtraffic over hours/days - If any downstream dependency (Neo4j, embedding API) experiences intermittent slowness, threads accumulate
- Monitor with:
docker exec <container> cat /proc/1/status | grep Threads
Environment | 环境信息
- Python version: 3.11
- Operating System: Linux (Raspberry Pi / aarch64)
- MemOS version: v2.0.8 (also present in v2.0.9 —
searcher.pyunchanged) - Backend: Neo4j
- Deployment: Docker
Additional Context | 其他信息
Suggested Fix:
Use a shared, class-level ContextThreadPoolExecutor instead of creating a new one per request. The Searcher class already follows this pattern with _usage_executor (line 73). Adding a second shared pool for search operations would bound thread count regardless of request volume or downstream latency:
# In __init__:
self._search_executor = ContextThreadPoolExecutor(max_workers=10, thread_name_prefix="search")
# In each method, replace:
# with ContextThreadPoolExecutor(max_workers=N) as executor:
# with:
# executor = self._search_executorAll future.result() calls should also include a timeout (e.g., 30s) as a safety measure.
Workarounds:
- Set
pids_limitindocker-compose.ymlto fail fast instead of consuming all system resources - Set
OPENBLAS_NUM_THREADS=1to reduce per-thread overhead from numpy
Willingness to Implement | 实现意愿
- I'm willing to implement this myself | 我愿意自己解决
- I would like someone else to implement this | 我希望其他人来解决
Activity
fix: add timeout to thread pool tasks in searcher._retrieve_paths
fix: add pids_limit and OPENBLAS_NUM_THREADS to docker-compose
fix: prevent thread pool leaks in searcher (MemTensor#1273)
CaralHsi commentedon Mar 20, 2026
Fixed in PR #1308 - Fixed thread-pool leakage in searcher by reusing a shared executor and adding safer timeouts/cancellation.
已在 PR #1308 中修复 - 已修复 searcher 线程池泄漏问题:复用共享 executor,并增加更安全的超时/取消处理,避免线程无限增长。
brentkearney commentedon Mar 20, 2026
Update: After continued testing, we identified two additional thread exhaustion vectors beyond
searcher.py. PR #1308 addresses the original searcher issue but does not cover these.1. Scheduler thread pool default of 10,000
In
src/memos/api/config.py(line 840):The schema default in
general_schemas.pyis 50, butconfig.pyoverrides it with"10000"as theos.getenvfallback. Any deployment without this env var explicitly set will allocate a 10,000-worker thread pool.Fix: Change the fallback from
"10000"to"50"to match the schema default.2. Unbounded executor in
simple_struct.pyIn
src/memos/mem_reader/simple_struct.py(line 673), aContextThreadPoolExecutor()is created with nomax_workersargument and a new pool per request:In our deployment, this was the actual crash site — error logs show
simple_struct.py:684 - _read_memory - Task failed with exception: can't start new thread.Fix: Set an explicit
max_workersbound, or use a shared instance like thesearcher.pyfix.fix: use shared thread pool in searcher to prevent thread leak
github-actions commentedon Apr 26, 2026
This issue has been automatically marked as stale due to inactivity.
MaiHHConnect commentedon Apr 27, 2026
github-actions commentedon May 27, 2026
This issue has been automatically marked as stale due to inactivity.
20 remaining items
Memtensor-AI commentedon Jul 21, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
b1848bec14f3201eWorking branch:
feature/autodev-1273-20260721042041846Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(searcher): bound thread pool to stop leak in Searcher retrieval p…
Memtensor-AI commentedon Jul 21, 2026
✅ AutoDev task
b1848bec14f3201ecompleted.Summary: Fixes #1273 (thread leak in Searcher retrieval paths).
Searcherpreviously opened a freshContextThreadPoolExecutorper/searchrequest in four methods (_retrieve_paths,_retrieve_from_long_term_and_user,_retrieve_from_tool_memory,_deduplicate_rawfile_results); any hung sub-task blockedshutdown(wait=True)and leaked threads, growing container thread count without bound (reported 8,744 threads leading to empty/searchresults,/chat503s, and OpenBLASpthread_createfailures).The fix pre-allocates four class-level shared
ContextThreadPoolExecutorinstances inSearcher.__init__(mirroring the original per-sitemax_workers5/3/2/10, with distinctthread_name_prefixfor observability), rewrites the four call sites to submit to the shared executor without awithblock, and adds aSEARCH_FUTURE_RESULT_TIMEOUT = 30.0module constant applied to everyfuture.result()/as_completed()with warning-level exception handling. Total worker thread ceiling perSearcherdrops from unbounded to 20 regardless of QPS. Four separate pools (not one shared) prevent the deadlock that would arise from nested submits between_retrieve_pathsand its sub-paths.Tests: 4 new regression tests lock in shared executor identity across calls, absence of naked
.result(), and survival under a 5x-timeout slow sub-task. All 4 pre-existing behavioural tests still pass; widertests/memories/textual/suite is 62/62 green. Ruff check + format both clean. Two pre-existingtests/memories/activation/test_kv.pyfailures (transformersDynamicCacheAPI drift) are unrelated and reproduced on the stashed tree.Files touched:
src/memos/memories/textual/tree_text_memory/retrieve/searcher.py,tests/memories/textual/test_tree_searcher.py. No API schema, config, or dependency changes.Base branch:
mainBranch:
feature/autodev-1273-20260721042041846Commit:
cf0abb7PR: #2134
Assigned to: @bittergreen
Reviewers: @bittergreen
test(searcher): drop private ThreadPoolExecutor attrs from MemTensor#…
github-actions commentedon Aug 29, 2026
This issue has been automatically marked as stale due to inactivity.
github-actions commentedon Sep 6, 2026
This issue has been automatically closed due to inactivity.