Skip to content

PYTHON-6114 & PYTHON-5418 Test mod_wsgi in GitHub Actions and add Subinterpreter Support - #3059

Draft
blink1073 wants to merge 21 commits into
mongodb:mainfrom
blink1073:mod-wsgi-github-actions
Draft

blink1073 wants to merge 21 commits into
mongodb:mainfrom
blink1073:mod-wsgi-github-actions

Conversation

@blink1073

Copy link
Copy Markdown
Member

PYTHON-6114
PYTHON-5418

Changes in this PR

  • Run the mod_wsgi tests on pull requests in two GitHub Actions jobs, Mod WSGI Min Deps and Mod WSGI Latest, each covering the standalone and embedded modes.
  • Test the minimum dependencies with the oldest supported CPython against MongoDB 6.0, and the newest dependencies with the newest supported CPython against the latest MongoDB; the versions come from CPYTHONS and ALL_VERSIONS in generate_config_utils.py.
  • Install mod_wsgi from a mod_wsgi dependency group instead of consuming toolchain builds.
  • Remove the mod_wsgi tasks and build variant from the Evergreen config.
  • Set WSGIDestroyInterpreter Off in the test Apache config; mod_wsgi 6 deletes the thread states of still-running threads when it destroys interpreters as child processes exit, which crashes live pymongo monitor threads.
  • Add a just smoke-mod-wsgi recipe that runs the same steps locally in an ubuntu container.
  • Update test/mod_wsgi_test/README.rst and CONTRIBUTING.md for the new workflow.

Subinterpreter support (PYTHON-5418)

  • PeriodicExecutor now falls back to a non-daemon thread when daemon threads are disallowed (subinterpreters), and the monitor shutdown handler is registered with threading._register_atexit so executors are stopped and joined before interpreter teardown.
  • New tests: PeriodicExecutor in a real subinterpreter, concurrent MongoClients across subinterpreters, and MongoClient inside the standard InterpreterPoolExecutor (Python 3.14+).

Test Plan

  • just smoke-mod-wsgi passes: both modes, two rounds each, against a single-node replica set in an ubuntu:24.04 container with mod_wsgi 6.0.6.
  • A workflow_dispatch run of test-python.yml is green for both Mod WSGI jobs (min-deps round on MongoDB 6.0 / Python 3.10, latest round on latest MongoDB / Python 3.14).
  • just lint, just lint-manual, just typing, and uv lock --check all pass.

Checklist

Checklist for Author

  • Did you update the changelog (if necessary)?
  • Is there test coverage?
  • Is any followup work tracked in a JIRA ticket? If so, add link(s). (Upstream mod_wsgi issue for the child-exit crash pending; tracked in the PYTHON-6114 pitfalls.)

Checklist for Reviewer

  • Does the title of the PR reference a JIRA Ticket?
  • Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
  • Is all relevant documentation (README or docstring) updated?

The Evergreen tasks depend on toolchain builds of mod_wsgi.so for each
Python version. The new Mod WSGI jobs install mod_wsgi with pip on an
ubuntu runner and test the latest stable CPython against a replica set
in both daemon and embedded mode. mod_wsgi is pinned to 4.9.4 because
releases 5 and newer crash Apache children in embedded mode under load.
The jobs test the lowest supported MongoDB with the oldest CPython and
minimum dependencies, and the latest MongoDB with the newest CPython.
mod_wsgi moves into a dependency group pinned to 4.9.4, because releases
5 and newer crash Apache children in embedded mode under load.
mod_wsgi 6 destroys interpreters as child processes exit and deletes
the thread states of still-running threads, crashing live pymongo
monitor threads. Set WSGIDestroyInterpreter Off in the test Apache
config to keep the mod_wsgi 4 lifecycle. The smoke script raises the
nofile limit so mongod is not starved by the connection storm, and the
test client now dumps the Apache error log on any failure.
Pin the action to a commit hash so the PR's CodeQL check no longer
flags the new setup-mongodb steps for using an unpinned third-party
action ref.
PeriodicExecutor now falls back to a non-daemon thread when daemon threads are disallowed (subinterpreters). The monitor shutdown handler is also registered with threading._register_atexit so executors are stopped and joined before interpreter teardown.

Adds regression tests for PeriodicExecutor in a real subinterpreter and for concurrent MongoClients across subinterpreters (Python 3.14+).
Runs live MongoClients inside interpreters managed by the standard
InterpreterPoolExecutor, mirroring the existing multi-threaded checks.
The pool's interpreters do not allow daemon threads, so this exercises
the non-daemon monitor thread fallback and its shutdown handling.
The worker interpreters may not have the repo root on sys.path (the
stdlib test package shadows the repo's), so the pickled worker function
failed to unpickle.  Submit the builtin exec with a code string that
inserts the main interpreter's sys.path before importing pymongo.
@semgrep-code-mongodb

Copy link
Copy Markdown

Semgrep found 1 github-actions-mutable-action-tag finding:

GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

🛟 Help? Slack #semgrep-help or go/semgrep-help.

Resolution Options:

  • Fix the code
  • Reply /fp $reason (if security gap doesn’t exist)
  • Reply /ar $reason (if gap is valid but intentional; add mitigations/monitoring)
  • Reply /other $reason (e.g., test-only)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The shutdown ordering and Apache module-loading issues can break applications and prevent the new tests from starting.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity · 1 Low severity

Open (4)
What changed in this PR

Moves mod_wsgi testing from Evergreen to GitHub Actions and adds subinterpreter-safe monitor shutdown behavior.

Changes:

  • Adds min/latest mod_wsgi GitHub Actions jobs and local smoke testing.
  • Adds subinterpreter regression tests and executor shutdown handling.
  • Removes obsolete Evergreen tasks and updates documentation.
File Description
uv.lock Locks mod_wsgi dependencies.
test/​test_threads.py Adds subinterpreter integration tests.
test/​test_periodic_executor.py Adds executor shutdown regression coverage.
test/​mod_wsgi_test/​README.rst Updates test instructions.
test/​mod_wsgi_test/​apache24ubuntu.conf Updates Apache lifecycle configuration.
test/​mod_wsgi_test/​apache22ubuntu1204.conf Removes obsolete Apache configuration.
test/​mod_wsgi_test/​apache22amazon.conf Removes obsolete Apache configuration.
test/​asynchronous/​test_periodic_executor.py Mirrors periodic executor tests.
pyproject.toml Adds the mod_wsgi dependency group.
pymongo/​synchronous/​monitor.py Registers early shutdown handling.
pymongo/​periodic_executor.py Supports non-daemon executor threads.
pymongo/​asynchronous/​monitor.py Provides canonical shutdown registration changes.
justfile Adds the mod_wsgi smoke recipe.
CONTRIBUTING.md Updates contributor instructions.
.github/​workflows/​test-python.yml Adds mod_wsgi CI jobs.
.evergreen/​scripts/​mod_wsgi_tester.py Modernizes setup and teardown.
.evergreen/​scripts/​mod_wsgi_smoke_test.sh Adds containerized smoke testing.
.evergreen/​scripts/​mod_wsgi_matrix.py Generates endpoint test versions.
.evergreen/​scripts/​generate_config.py Removes Evergreen mod_wsgi generation.
.evergreen/​generated_configs/​variants.yml Removes the generated variant.
.evergreen/​generated_configs/​tasks.yml Removes generated tasks.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .evergreen/scripts/mod_wsgi_smoke_test.sh Outdated
Comment thread .github/workflows/test-python.yml
Comment thread pymongo/asynchronous/monitor.py Outdated
Comment thread test/mod_wsgi_test/README.rst Outdated
@codecov

codecov Bot commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 37.50000% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pymongo/asynchronous/monitor.py 16.66% 5 Missing ⚠️
pymongo/synchronous/monitor.py 50.00% 3 Missing ⚠️
pymongo/periodic_executor.py 50.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

- Pin uv to an immutable release with checksum verification in the smoke
  test instead of piping the mutable install script into a shell.
- Install the minimum mod_wsgi version (4.9.4) in the min deps job; the
  lock file has the latest version and TEST_MIN_DEPS does not cover the
  mod_wsgi group.
- Only register the monitor shutdown handler with
  threading._register_atexit in interpreters that disallow daemon
  threads, preserving normal main interpreter shutdown ordering.
- Pin actions/checkout to a commit hash to satisfy the semgrep
  mutable-action-tag check.
- Clarify the test matrix documentation: MongoDB 6.0 is hardcoded
  because ubuntu-22.04 runners cannot install older versions.
- Add timeout-minutes and apt-get update to the new jobs, guard
  interpreter cleanup in the tests, and fix the rationale for calling
  the test scripts directly in the CONTRIBUTING instructions.
Add mod_wsgi to GROUP_MAP so the group is included in the
--resolution=lowest-direct resolution, testing the minimum version from
pyproject.toml rather than hardcoding it in the workflow.
The root phase already runs before the unprivileged re-exec and has
curl, so install the pinned uv to /usr/local/bin there instead of
installing it again as the smoke user.
@blink1073
blink1073 requested a balanced review from Copilot September 21, 2026 21:36
@blink1073 blink1073 changed the title PYTHON-6114 & PYTHON-5418 Move the mod_wsgi tests from Evergreen to GitHub Actions PYTHON-6114 & PYTHON-5418 Test mod_wsgi in GitHub Actions and add Subinterpreter Support Sep 21, 2026
Runs live AsyncMongoClients inside interpreters managed by
InterpreterPoolExecutor. The async client runs its background tasks on
the interpreter's own event loop rather than in threads, covering the
other shutdown path.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The minimum-dependency job remains locked to newer packages, and subinterpreter cleanup uses an inverted running-state check.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (4)

Comment thread test/test_threads.py Outdated
Invert the is_running guard in test_subinterpreters: closing an idle
interpreter runs threading._shutdown, which stops and joins pymongo's
monitor threads, so the successful path must close them. Note in the
changelog that both clients are covered by the subinterpreter tests.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Apache loads mod_wsgi too late, and the minimum-dependency job currently tests neither the minimum Python nor the minimum mod_wsgi version.

Review effort: Balanced
Findings: None

Resolved since last review (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Test CPython 3.9 as the oldest supported version

.evergreen/​scripts/​mod_wsgi_matrix.py:12

CPYTHONS[0] is 3.10, but this project still declares and tests CPython 3.9 (requires-python >=3.9). Consequently this matrix does not test the “oldest supported CPython” promised by the PR and README. Use the existing minimum-support list for this entry.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants