Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
It is not actually mod_wsgi that is doing this. Likely you are using a configuration of mod_wsgi for testing that is using Python sub interpreters and CPython as a result does it when a Python sub interpreter is destroyed. If you ignore this and don't try to shutdown the thread yourself when the interpreter is being cleaned up, then pymongo could crash with any Python application that started using Python sub interpreters and it isn't going to be a mod_wsgi specific issue. Things are a bit more forgiving when background threads run in the main interpreter context, so you likely have never seen an issue there for normal use of pymongo. For sub interpreters though, CPython has been a bit more strict about what happens with background threads in sub interpreters for at least a couple of Python versions now. So just be aware that not addressing the actual issue may come back and bite you in the future if people start actually using pymongo in Python sub interpreters using the Python APIs which allow sub interpreter access from Python code (vs from a C extension). |
|
If you don't already, I would actually suggest you start incorporating tests which use concurrent.interpreters (https://docs.python.org/3/library/concurrent.interpreters.html) to create sub interpreters yourself from Python code and exercise pymongo in a sub interpreter and then cleanup up the sub interpreter. This will likely reveal the same issue. |
|
Great, thanks for the pointers @GrahamDumpleton! |
The Evergreen tasks depend on toolchain builds of mod_wsgi.so for each Python version. The new Mod WSGI jobs install mod_wsgi with pip on an ubuntu runner and test the latest stable CPython against a replica set in both daemon and embedded mode. mod_wsgi is pinned to 4.9.4 because releases 5 and newer crash Apache children in embedded mode under load.
The jobs test the lowest supported MongoDB with the oldest CPython and minimum dependencies, and the latest MongoDB with the newest CPython. mod_wsgi moves into a dependency group pinned to 4.9.4, because releases 5 and newer crash Apache children in embedded mode under load.
mod_wsgi 6 destroys interpreters as child processes exit and deletes the thread states of still-running threads, crashing live pymongo monitor threads. Set WSGIDestroyInterpreter Off in the test Apache config to keep the mod_wsgi 4 lifecycle. The smoke script raises the nofile limit so mongod is not starved by the connection storm, and the test client now dumps the Apache error log on any failure.
45c7020 to
bb36972
Compare
Pin the action to a commit hash so the PR's CodeQL check no longer flags the new setup-mongodb steps for using an unpinned third-party action ref.
PeriodicExecutor now falls back to a non-daemon thread when daemon threads are disallowed (subinterpreters). The monitor shutdown handler is also registered with threading._register_atexit so executors are stopped and joined before interpreter teardown. Adds regression tests for PeriodicExecutor in a real subinterpreter and for concurrent MongoClients across subinterpreters (Python 3.14+).
Runs live MongoClients inside interpreters managed by the standard InterpreterPoolExecutor, mirroring the existing multi-threaded checks. The pool's interpreters do not allow daemon threads, so this exercises the non-daemon monitor thread fallback and its shutdown handling.
blink1073
changed the title
The worker interpreters may not have the repo root on sys.path (the stdlib test package shadows the repo's), so the pickled worker function failed to unpickle. Submit the builtin exec with a code string that inserts the main interpreter's sys.path before importing pymongo.
|
Moved to mongodb#3059 |
- Pin uv to an immutable release with checksum verification in the smoke test instead of piping the mutable install script into a shell. - Install the minimum mod_wsgi version (4.9.4) in the min deps job; the lock file has the latest version and TEST_MIN_DEPS does not cover the mod_wsgi group. - Only register the monitor shutdown handler with threading._register_atexit in interpreters that disallow daemon threads, preserving normal main interpreter shutdown ordering. - Pin actions/checkout to a commit hash to satisfy the semgrep mutable-action-tag check. - Clarify the test matrix documentation: MongoDB 6.0 is hardcoded because ubuntu-22.04 runners cannot install older versions. - Add timeout-minutes and apt-get update to the new jobs, guard interpreter cleanup in the tests, and fix the rationale for calling the test scripts directly in the CONTRIBUTING instructions.
Add mod_wsgi to GROUP_MAP so the group is included in the --resolution=lowest-direct resolution, testing the minimum version from pyproject.toml rather than hardcoding it in the workflow.
The root phase already runs before the unprivileged re-exec and has curl, so install the pinned uv to /usr/local/bin there instead of installing it again as the smoke user.
Runs live AsyncMongoClients inside interpreters managed by InterpreterPoolExecutor. The async client runs its background tasks on the interpreter's own event loop rather than in threads, covering the other shutdown path.
Invert the is_running guard in test_subinterpreters: closing an idle interpreter runs threading._shutdown, which stops and joins pymongo's monitor threads, so the successful path must close them. Note in the changelog that both clients are covered by the subinterpreter tests.
This branch has not been deployed
3 participants
PYTHON-6114
PYTHON-5418
Changes in this PR
Mod WSGI Min DepsandMod WSGI Latest, each covering the standalone and embedded modes.CPYTHONSandALL_VERSIONSingenerate_config_utils.py.mod_wsgidependency group instead of consuming toolchain builds.WSGIDestroyInterpreter Offin the test Apache config; mod_wsgi 6 deletes the thread states of still-running threads when it destroys interpreters as child processes exit, which crashes live pymongo monitor threads.just smoke-mod-wsgirecipe that runs the same steps locally in an ubuntu container.test/mod_wsgi_test/README.rstandCONTRIBUTING.mdfor the new workflow.Subinterpreter support (PYTHON-5418)
PeriodicExecutornow falls back to a non-daemon thread when daemon threads are disallowed (subinterpreters), and the monitor shutdown handler is registered withthreading._register_atexitso executors are stopped and joined before interpreter teardown.PeriodicExecutorin a real subinterpreter, concurrentMongoClients across subinterpreters, andMongoClientinside the standardInterpreterPoolExecutor(Python 3.14+).Test Plan
just smoke-mod-wsgipasses: both modes, two rounds each, against a single-node replica set in an ubuntu:24.04 container with mod_wsgi 6.0.6.workflow_dispatchrun oftest-python.ymlis green for both Mod WSGI jobs (min-deps round on MongoDB 6.0 / Python 3.10, latest round on latest MongoDB / Python 3.14).just lint,just lint-manual,just typing, anduv lock --checkall pass.Checklist
Checklist for Author
Checklist for Reviewer