Skip to content

Scheduler initialized but not processing tasks (uvicorn vs direct call difference) #1395

Closed

Description

@WuSho1995

Environment

  • MemOS Version: 2.0.11 (also reproducible on 2.0.10)
  • Deployment: Docker (memtensor/memos-base:v1.0)
  • Storage: Neo4j 5.21 + Qdrant v1.16.3
  • Queue: Local (not Redis, DEFAULT_USE_REDIS_QUEUE=false)
  • OS: Debian 12, Docker 28.5.2

Issue Description

The MemOS scheduler initializes successfully — OptimizedScheduler._running=True and MessageConsumerThread is alive — but no tasks are ever processed. When memories are added via /product/add, the scheduler status always shows all zeros:

{
  "scheduler_summary": {
    "waiting": 0, "in_progress": 0, "pending": 0,
    "completed": 0, "failed": 0, "total": 0
  }
}

Environment Variables

MOS_ENABLE_SCHEDULER=true
API_SCHEDULER_ON=true
DEFAULT_USE_REDIS_QUEUE=false

Reproduction Steps

  1. Start MemOS with MOS_ENABLE_SCHEDULER=true and API_SCHEDULER_ON=true
  2. Add a memory: curl -X POST http://localhost:8000/product/add -H "Content-Type: application/json" -d '{"user_id":"jarvis","messages":[{"role":"user","content":"test"}],"async_mode":"async"}'
  3. Check scheduler status: curl http://localhost:8000/product/scheduler/allstatus
  4. Result: all task counts are 0

Key Finding

When init_server() is called directly in a Python shell inside the container:

  • Scheduler starts correctly
  • Consumer thread is alive and working
  • ✅ Logs show: Scheduler initialized, Scheduler started, Message consumer thread started

When MemOS starts via uvicorn (uvicorn memos.api.server_api:app):

  • No scheduler-related startup logs appear
  • No Scheduler initialized or Scheduler started or Message consumer thread started
  • But the scheduler object exists and _running=True
  • ❌ No tasks are processed

The startup logs show only:

INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000

Missing all scheduler initialization messages.

Suspected Root Cause

The server_router.py module calls components = handlers.init_server() at module import time. When running under uvicorn, this initialization appears to complete (the API works), but the scheduler consumer thread may not actually start processing messages. There may be a timing or threading issue specific to the uvicorn event loop.

Additional Notes

  • The validate_partial_initialization warning from AuthConfig appears because rabbitmq/openai/graph_db are not configured
  • The add handler creates memories in Neo4j successfully — only scheduler processing is broken
  • Tested on both v2.0.10 and v2.0.11 with identical behavior

Question

Could you confirm if the scheduler is tested in a Docker deployment with local queue (no Redis)? Is there any additional configuration required beyond MOS_ENABLE_SCHEDULER=true and API_SCHEDULER_ON=true?

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 29, 2026
  2. self-assigned this
    on Jun 30, 2026
  3. Memtensor-AI commented on Jun 30, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: fcda6b589ec78706
    Working branch: bugfix/autodev-1395
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. added a commit that references this issue on Jun 30, 2026
    977bfe2
  5. Memtensor-AI commented on Jun 30, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task fcda6b589ec78706 completed.

    Summary: Fixes issue #1395: under Docker + uvicorn with the local in-memory message queue (DEFAULT_USE_REDIS_QUEUE=false), GET /product/scheduler/allstatus always returned {waiting:0, in_progress:0, pending:0, ...} even though the consumer thread was alive and dispatching MEM_READ tasks. Root cause: handle_scheduler_allstatus only aggregated per-stream entries whose key starts with "scheduler:" — these only exist for the Redis-backed queue monitor. The local queue monitor returns just flat top-level totals (running, remaining, pending), so the prefix filter collapsed the summary to zero regardless of actual queue depth or dispatcher state.

    Fix: in src/memos/api/handlers/scheduler_handler.py::handle_scheduler_allstatus, detect whether per-stream entries are present; if not and mem_scheduler.use_redis_queue is False, fall back to the flat totals so in_progress, waiting, and pending reflect live local-queue state. Redis-queue aggregation is untouched. The API response model is unchanged, so no OpenAPI regeneration is required; completed/failed/cancelled remain zero for local queues (documented behavior — persistent task status tracking is Redis-only).

    Tests: added 3 regression tests in tests/api/test_scheduler_handler_allstatus.py covering local-non-zero, local-empty, and Redis per-stream paths. Initial run of the local-non-zero test produced the expected RED (assert 0 == 2); after the fix all three pass. Wider regression sweep across tests/api/test_server_router.py + tests/mem_scheduler/ + the new file: 79 passed / 1 skipped. Four pre-existing failures in tests/api/test_mcp_serve.py are unrelated to this change (confirmed by stashing the diff and reproducing on the base branch HEAD). Ruff lint + format clean on the touched files. The secondary symptom in the issue (missing scheduler INFO logs on Docker stdout) is caused by memos/log.py pinning the console handler to WARNING when DEBUG=False; this is documented in the design doc as out of scope to avoid a broader operational change.

    Base branch: main
    Branch: bugfix/autodev-1395
    Commit: 977bfe2
    PR: #1982
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  6. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 30, 2026
  7. added a commit that references this issue on Jul 2, 2026
    e3f4e51
  8. added and removed
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    on Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:coreMOS 编排层 / 框架底座 / 跨模块问题types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions