Skip to content

fix: defer boot_agent until after eventlet monkey_patch to prevent recursion error - #903

Open
CagriYonca wants to merge 1 commit into
mainfrom
fix-eventlet
Open

CagriYonca wants to merge 1 commit into
mainfrom
fix-eventlet

Conversation

@CagriYonca

@CagriYonca CagriYonca commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem & Root Cause

When running under Gunicorn with the eventlet worker class, Instana's early auto-instrumentation eagerly imports third-party libraries in the master process before eventlet.monkey_patch executes.

Modules such as urllib3 and pymongo capture module-level references to the unpatched ssl.SSLContext. When Eventlet subsequently patches the ssl module, ssl.SSLContext is replaced with its green subclass. Initializing SSL connections later causes super lookups inside property setters to resolve the global green subclass instead of the original class, resulting in an infinite recursion and a RecursionError.

Solution

This PR defers boot_agent execution when eventlet is present in the environment:

  • Wraps eventlet.monkey_patch using wrapt.
  • Defers boot_agent and all eager instrumentation imports until after eventlet.monkey_patch completes in the forked worker process.
  • Suppresses sensor boot in the Gunicorn arbiter process, ensuring instrumentation runs only in worker processes.

Changes

  • src/instana/__init__.py: Added _defer_boot_until_eventlet_patch to defer agent boot until worker post-patch.
  • tests/frameworks/test_eventlet_autotrace.py: Added unit tests covering deferred boot registration, worker execution, and arbiter suppression.

@CagriYonca
CagriYonca marked this pull request as ready for review September 16, 2026 12:51
@CagriYonca
CagriYonca requested a review from a team as a code owner September 16, 2026 12:51

@pvital pvital left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @CagriYonca.
I have a few considerations.

Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py
Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py
Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py Outdated

@GSVarsha GSVarsha left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few requests.

@CagriYonca
CagriYonca force-pushed the fix-eventlet branch 5 times, most recently from 4a4d649 to f9bc0fd Compare September 29, 2026 12:12
…cursion error

Signed-off-by: Cagri Yonca <cagri@ibm.com>
@CagriYonca

Copy link
Copy Markdown
Contributor Author

Update after comments:

  • Prevent double boot_agent() execution: Handled module-level unconditional call with a return_early flag and added an _eventlet_booted guard per process.

  • Fix Gunicorn 26+ crash on startup: Replaced eager wrap_function_wrapper("gunicorn.workers.geventlet", ...) with wrapt.when_imported(...) to prevent ModuleNotFoundError when the removed eventlet worker is not present.

  • Remove fragile /proc inspection: Replaced /proc/{ppid}/cmdline checking (which failed under Docker PID 1, console scripts, and python -m gunicorn) by hooking directly into EventletWorker.patch(), running exclusively post-fork in workers.

  • Make deferred boot opt-in: Restricted deferred initialization behind INSTANA_EVENTLET_DEFERRED_BOOT=true so non-eventlet environments (Celery, gthread, standalone scripts) don't silently lose tracing.

  • Fix SQLAlchemy test collection: Moved module-level create_engine(...) into a fixture and explicitly pinned the dialect to postgresql+psycopg2:// to avoid Pytest collection failures on Python 3.11+ / SQLAlchemy 2.1+.

@CagriYonca
CagriYonca requested a review from GSVarsha September 29, 2026 13:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants