Skip to content

PYTHON-6226 AsyncMongoClient stays stuck in ReplicaSetNoPrimary after a monitor task cancels itself - #3180

Merged
blink1073 merged 7 commits into
mongodb:mainfrom
blink1073:PYTHON-6226
Nov 9, 2026
Merged

blink1073 merged 7 commits into
mongodb:mainfrom
blink1073:PYTHON-6226

Conversation

@blink1073

Copy link
Copy Markdown
Member

PYTHON-6226

Changes in this PR

A replica set member can remove itself from the topology with its own hello response, for example during a node restart. The monitor's executor then cancelled its own task, so the monitor never ran again and the client stayed in ReplicaSetNoPrimary until restart.

  • A monitor closing its own executor no longer cancels itself.
  • Closing the executor from another task, or from a thread without a running event loop, keeps its previous behavior.

Test Plan

The self-close unit test and the integration test fail before the fix and pass after it.

  • Executor unit tests cover closing from the monitor's own task and from a thread without an event loop.
  • An integration test fakes a two-node replica set, removes each node from the topology for a few seconds, and requires the client to recover (skips on the sync driver, which is not affected).
  • Also run: the full periodic executor suite, the sync mirrors of the new tests, just typing, and just lint-manual.

Checklist

Checklist for Author

  • Did you update the changelog (if necessary)?
  • Is there test coverage?
  • Is any followup work tracked in a JIRA ticket? If so, add link(s). Not yet: the ticket's suggested extra hardening has no ticket.

Checklist for Reviewer

  • Does the title of the PR reference a JIRA Ticket?
  • Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
  • Is all relevant documentation (README or docstring) updated?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test can hang on Python versions where Server.wait_closed() waits for accepted connections to disconnect, so client.close() must be awaited before waiting for the fake servers to close.

1 open issue
What changed in this PR

Fixes async monitor self-cancellation that could leave AsyncMongoClient stuck in ReplicaSetNoPrimary after a replica set member removes itself from the topology.

Changes:

  • Prevents an executor from cancelling its own task while preserving cancellation from other callers.
  • Adds executor lifecycle tests and a two-node replica set recovery regression test.
  • Documents the fix in the changelog.
File Description
test/​test_periodic_executor.py Sync mirror of executor lifecycle tests.
test/​test_discovery_and_monitoring.py Sync mirror with the async-only recovery test skipped.
test/​asynchronous/​test_periodic_executor.py Tests self-close, restart, and off-loop closure.
test/​asynchronous/​test_discovery_and_monitoring.py Adds simulated replica set recovery coverage.
pymongo/​periodic_executor.py Guards against executor self-cancellation.
doc/​changelog.rst Documents the recovery fix and links PYTHON-6226.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cancellation during monitor cleanup can leave the application pool unclosed after its server is removed, preventing later cleanup from retrying the close.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread pymongo/asynchronous/topology.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new test cancels a real task from another thread, causing a RuntimeError and assertion failure when asyncio debug mode is enabled.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Avoid cancelling a real task from another thread in the test

test/​asynchronous/​test_periodic_executor.py:220

With asyncio debug mode enabled (for example, PYTHONASYNCIODEBUG=1), this worker cancels a real task on another thread's running event loop, raising RuntimeError and failing this assertion. Since the PR intentionally preserves existing off-loop cancellation behavior, use a recording task mock to test the current_task() fallback, then cancel and join the real executor on the event-loop thread. Regenerate the synchronous test mirror after updating the async source.

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to weigh an unpublished finding that a thread join can block the async test’s event loop for up to two seconds.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change addressing monitor recovery and safe server shutdown.

No open issues 🎉

Review effort: Lite

@blink1073
blink1073 marked this pull request as ready for review November 9, 2026 12:26
@blink1073
blink1073 requested a review from a team as a code owner November 9, 2026 12:26
@blink1073
blink1073 requested a review from sleepyStick November 9, 2026 12:26
Comment on lines +691 to +692
if _IS_SYNC:
self.skipTest("PYTHON-6226: monitor task self-cancellation is async-only")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor but should these be an @unittest.skipUnless(not _IS_SYNC, ...) decorator? (i think we do that for other tests?)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@blink1073
blink1073 requested a review from sleepyStick November 9, 2026 19:45
@blink1073
blink1073 merged commit 3319cf4 into mongodb:main Nov 9, 2026
21 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants