Skip to content

fix(scheduler): initialize RabbitMQ IO thread before shutdown - #2532

Merged
syzsunshine219 merged 4 commits into
MemTensor:dev-v2.0.36from
linsheng-li:fix/rabbitmq-shutdown-thread-init
Oct 29, 2026
Merged

syzsunshine219 merged 4 commits into
MemTensor:dev-v2.0.36from
linsheng-li:fix/rabbitmq-shutdown-thread-init

Conversation

@linsheng-li

Copy link
Copy Markdown
Contributor

Description

Stopping a scheduler that has never initialized RabbitMQ logs AttributeError: 'OptimizedScheduler' object has no attribute '_io_loop_thread' during API shutdown. This includes local-queue deployments and initialization skipped in CI. No restart is required to trigger it.

The constructor initializes _rabbitmq_io_loop_thread, but both initialize_rabbitmq() and rabbitmq_close() use _io_loop_thread. Initialize the same field in the constructor. The production change is one line; connection cleanup, thread joins, shutdown ordering, and exception handling are unchanged. No new dependencies or configuration are required.

Related issue: N/A; the reproduction and regression tests are included here.

Type of change

  • Bug fix (non-breaking)

How Has This Been Tested?

  • Unit tests: pytest tests/mem_scheduler/test_rabbitmq_shutdown.py -q — 5 passed. Without the fix, 3 failed, 2 passed. Covers uninitialized/repeated close, CI initialization skip, connection cleanup with a real IO thread, native optimized-scheduler lifecycle cleanup, and propagation of unexpected join errors.
  • Test steps: isolated API shutdown with RabbitMQ uninitialized, followed by restart and a second shutdown — both exits were clean; graph/vector/business snapshots were unchanged. No generation or embedding requests were made.
  • make format — all Ruff checks passed; 631 files unchanged.
  • Pre-commit checks on both changed files — passed.

Additional directory run: pytest tests/mem_scheduler/ -q — 70 passed, 1 skipped, 1 failed. The existing test_dynamic_cache_layers_access requires PyTorch, which is absent in the test image; the same failure was reproduced against the unmodified source. Pytest also warns that the image lacks the asyncio plugin. The full repository suite was not run.

Checklist

  • Self-reviewed the one-line fix and focused tests.
  • Added tests demonstrating the failure and preserving existing cleanup behavior.
  • No documentation/API/schema changes are needed for this internal field-name correction.

@Memtensor-AI Memtensor-AI added area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Oct 21, 2026
@Memtensor-AI
Memtensor-AI requested a review from wustzdy October 21, 2026 15:05
@Memtensor-AI

Memtensor-AI commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2532
Task: 0c46326d7dbb4808
Base: dev-v2.0.36
Head: fix/rabbitmq-shutdown-thread-init

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Generated by cloud-assistant via Open Code Review.

@endxxxx
endxxxx changed the base branch from main to dev-v2.0.36 October 24, 2026 13:57
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

❌ Automated Test Results: FAILED

Auto-fix retry 1/2 triggered.

Failed tests:

  • test_close_without_rabbitmq_initialization_is_safe_and_repeatable
  • test_early_initialization_skip_still_closes
  • test_real_io_thread_and_connection_cleanup_still_execute
  • test_actual_optimized_scheduler_lifecycle_joins_threads_and_shuts_executor
  • test_unexpected_join_error_is_not_hidden
Error details
Tests failed. Failed cases: test_close_without_rabbitmq_initialization_is_safe_and_repeatable, test_early_initialization_skip_still_closes, test_real_io_thread_and_connection_cleanup_still_execute, test_actual_optimized_scheduler_lifecycle_joins_threads_and_shuts_executor, test_unexpected_join_error_is_not_hidden

Branch: fix/rabbitmq-shutdown-thread-init

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: All five tests fail at import/instantiation time with ImportError: Missing required module - 'pika', raised by the dependency-check decorator in src/memos/dependency.py before any test logic executes.
Branch: fix/rabbitmq-shutdown-thread-init

@Memtensor-AI
Memtensor-AI requested a review from wustzdy October 29, 2026 06:21
@syzsunshine219
syzsunshine219 merged commit e85b9db into MemTensor:dev-v2.0.36 Oct 29, 2026
16 checks passed
@endxxxx endxxxx mentioned this pull request Nov 9, 2026
4 tasks done
syzsunshine219 added a commit that referenced this pull request Nov 9, 2026
## Description

Merge `dev-v2.0.36` into `main` for MemOS v2.0.36. The final
release-branch commit sets both the package version and
`memos.__version__` to `2.0.36`.

This release includes the PostgreSQL graph storage updates (#2531),
knowledge memory and recall changes (#2479, #2436), and fixes for
PolarDB recall (#2551), RabbitMQ shutdown (#2532), API key validation
(#2491), and rate limiting (#2490).

## How Has This Been Tested?

- [x] Unit tests — 24 passed in
`tests/graph_dbs/test_postgres_400_compat.py` and
`tests/mem_scheduler/test_rabbitmq_shutdown.py`.
- [x] Verified that both version fields are `2.0.36`, ran `make format`
and `git diff --check`, and simulated the merge with `main`.

## Checklist

- [x] Reviewed the release diff and the two-file version commit.
- [x] Verified the release branch merges cleanly with the current `main`
locally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants