Skip to content

[Bug] Intermittent CPython 3.13 process-exit hang in CoreRuntimeHandle/Tokio shutdown (Linux x86_64, temporalio 1.35.0) #2107

Closed

Description

@AppSolves

What are you trying to do?

Run short-lived Python processes using the Temporal Python SDK in integration tests, including namespace registration and workflow/worker observations, and let the parent process wait for normal exit and closed stdout/stderr pipes.

Describe the bug

With the official PyPI temporalio==1.35.0 wheel, a child occasionally finishes its asyncio work, prints its last phase marker and reaches its atexit marker, yet never exits. The parent eventually terminates the child with its unchanged watchdog (roughly 29–30 seconds). This is intermittent: other runs of the same tests exit normally.

We independently observed the same native stack categories described in #2046, which was closed as not planned by its reporter. We commented there with our initial findings, but are filing this separately because the defect still reproduces for us, and we have additional independent evidence.

Environment

  • OS: GitHub-hosted Ubuntu 24.04, Linux x86_64
  • SDK: official temporalio==1.35.0, not our experimental wheel
  • CPython 3.13.17 in frozen diagnostic runs; normal CI also reproduced post-atexit survivors with 3.13.18
  • Temporal Server 1.32.0 in the relevant integration tests
  • Short-lived subprocesses invoked by integration/browser test harnesses

Native evidence

In real hung CI children, GDB classified:

  • Main thread: CoreRuntimeHandle::drop / Tokio runtime shutdown and synchronization
  • Another native thread: Python GIL acquisition, including PyThread_hang_thread, in a Temporal/Tokio context

One 3.13.18 CI child reached ASYNCIO_RETURN and ATEXIT around 0.8 s but exited only after the watchdog sent SIGTERM about 29 s later. Other naturally completed children exited normally.

These observations strongly match #2046's described finalization deadlock. We have not independently established the full pyo3_async_runtimes::set_result callback/owner/wait-for cycle in our optimized Linux x86_64 SDK build. Not every unrelated test timeout has been classified as this bug.

Attempted candidate fix (isolated only)

We built a separate, hash-verified test-only Python wheel from SDK 1.35.0 sources with the one-file finalization patch proposed in #2047. No application dependency or normal CI used this custom build.

  • Bounded ordinary runtime create/drop tests exited normally for both official and patched wheels; worker threads disappeared after each normal drop.
  • We observed two stable process-global Unix socket descriptors after the first runtime initialization, with no additional growth over three cycles in either wheel; exact initialization ownership is not conclusively established.
  • A deliberately injected exception from sys.is_finalizing() revealed a concern in the candidate patch: three consecutive runtime drops retained 2, 4, 6 native threads in the patched wheel, versus 0, 0, 0 in the official wheel.
  • This injection does not establish that such an exception occurs in production, but it makes an unconditional retain-on-check-error path unsuitable for us without a narrowly justified lifetime policy.
  • Unmodified PostgreSQL/Temporal registration integration controls passed once per arm, but we did not reproduce the native wait cycle in a paired baseline-versus-patched experiment. We therefore do not claim the candidate fixes the intermittent hang.

We have not rolled out a patched SDK or modified existing test timeouts.

Expected behavior

After normal completion of the asyncio work and cleanup, short-lived subprocesses should exit and close their pipes without hanging in native runtime shutdown.

Questions for maintainers

  1. Is this recognized as an SDK interpreter-finalization / native-callback lifecycle bug, and is an upstream correction planned?
  2. Was there a correctness concern behind the closure of #2047, or is another approach preferred?
  3. What is the recommended abi3-compatible strategy for finalization-safe Core runtime destruction without retaining threads if the Python finalization check fails outside interpreter shutdown?
  4. Would a minimal, sanitized reproducer and a focused regression-tested PR be welcome?

We can provide further sanitized stack categories, reproducibility details and build/test evidence. We would prefer an upstream fix over maintaining a private SDK variant.

Related: #2046, #2047, #300.

Activity

  1. AppSolves commented on Nov 16, 2026

    @AppSolves
    Author

    Hi @VegetarianOrc and @Sushisource, a polite but urgent follow-up from the original reporter. Could we please get a maintainer triage update on this issue?

    The problem is still affecting our release/integration CI, and we've since confirmed the same post-atexit process-exit hang with the official temporalio==1.36.0 wheel, not just 1.35.0. In recent Linux x86_64 / CPython 3.13.18 runs, a short-lived Temporal observer/registration subprocess completed its operations and emitted ASYNCIO_RETURN and ATEXIT (in about 0.5–1.5 seconds), but remained alive with 3 native threads, 9 FDs and a FUTEX wait until the unchanged parent watchdog sent SIGTERM at about 29 seconds. Other identical-path children exited normally. We have multiple separately recorded CI occurrences.

    We are not claiming to have proven the exact native wait-for cycle for these latest processes, nor that every Temporal test timeout shares this cause. However, the recurring, specifically instrumented post-ATEXIT hangs are making an otherwise green integration suite intermittently fail and consuming significant engineering/review time. We have intentionally not masked the issue with timeout increases, disabled checks, os._exit, or a private SDK patch.

    I noticed the related discussions in #300 and the proposed approaches in #2047 and #2062, which are both closed without a merged fix. Could you clarify:

    1. Is this recognized as an upstream SDK shutdown/finalization issue, and is there an owner or a plan to investigate it?
    2. Is there a supported mitigation for short-lived Python SDK processes while an upstream fix is being developed, without bypassing normal shutdown?
    3. What would best unblock upstream work: a smaller sanitized reproducer, our phase/native-thread evidence, or a targeted regression test/PR? Is there an approximate release target, if one is known?

    Even a brief status and a recommended next step would help us plan responsibly. We're happy to contribute further sanitized evidence and testing, and we appreciate the maintainers' work on this SDK. Thank you!

  2. VegetarianOrc commented on Nov 17, 2026

    @VegetarianOrc
    Contributor

    Hey @AppSolves,

    Thanks for taking the time to report the issue. Here's the current answers to your questions:

    1. The issue does appear to be real and we're prioritizing investigation into getting a reliable reproduction and fix.
    2. Unfortunately, we don't have a suggested mitigation right now.
    3. A reliable reproduction would be much appreciated. There isn't a known release target with a fix yet.
  3. AppSolves commented on Nov 18, 2026

    @AppSolves
    Author

    Thanks for the update and for prioritizing this.

    Important update (18 Nov 2026): the same post-atexit hang has now been confirmed on GitHub-hosted Ubuntu 26.04.1 as well. Ubuntu 26.04 does NOT reliably prevent this issue. This new observation supersedes any tentative implication from our earlier small Ubuntu comparison.

    In a Main CI run on Ubuntu 26.04.1 LTS (GitHub runner image ubuntu-26.04, version 20261111.196.1), CPython 3.13.18, and the official PyPI temporalio==1.36.1 x86-64 wheel, one short-lived Temporal observer child (PID 17681) successfully completed CONNECT, START_WORKFLOW, DESCRIBE, ASYNCIO_RETURN, and ATEXIT within approximately 616 ms, but did not exit normally. At approximately 28 seconds it still had 3 native threads, 9 FDs and one FUTEX wait; the unchanged parent watchdog sent SIGTERM at approximately 29 seconds, after which both pipes closed. The test failed with OBSERVATION_UNAVAILABLE:TIMEOUT. This was the simulated-confirmation E2E test in CI run 44203628015, job 134463835512 (18 Nov UTC; Main SHA 13bb1ac74492c9f2d5aef6339b04fb5cd7d40f3f). The preceding real Temporal booking and witness steps passed; the subsequent Voice-send step did not run. We classify this as the documented post-atexit process-exit hang, not merely a namespace/setup timeout. CI job evidence (our repository is private; the key diagnostic facts are summarized here).

    Separately, we ran a bounded local follow-up looking for a smaller reproducer. On Linux x86-64 in Debian 13 (trixie) containers under Docker Desktop/WSL2, with CPython 3.13.18 and Temporal Server 1.32.0, we ran six isolated child processes with each of the official SDK wheels 1.36.0 and 1.36.1 (12 total). Each child connected, started a synthetic workflow and called describe(). All 12 returned from asyncio.run(), reached ATEXIT, exited with code 0 and closed both output pipes (0/12 reproduced). No application code or worker was involved in these minimal children.

    For historical context only, an earlier small GitHub-runner comparison showed Ubuntu 24.04 / CPython 3.13.17: 1/3 confirmed post-atexit hangs; Ubuntu 26.04 / CPython 3.13.17: 0/3; and Ubuntu 26.04 / CPython 3.13.18: 0/3. These negative samples do not imply an Ubuntu-26 fix, as the newly confirmed Ubuntu 26.04.1 / CPython 3.13.18 / SDK 1.36.1 recurrence demonstrates. Those runs also differed in glibc, OpenSSL, compiler and Python builds, so we cannot isolate an OS-specific cause.

    We still do not have a reliable minimal standalone reproducer. The native wait-for cycle/owner has not been conclusively proven, and we have not patched the SDK or relaxed test timeouts. If there is a particular reduced lifecycle or environment comparison that would be most helpful, we can collect more narrowly scoped, sanitized evidence.

  4. VegetarianOrc commented on Nov 19, 2026

    @VegetarianOrc
    Contributor

    Hi @AppSolves, we've recently merged #2120 that aims to alleviate this. Is it possible for you to run your reproduction steps against the current main branch to help gain confidence in the approach?

  5. AppSolves commented on Nov 19, 2026

    @AppSolves
    Author

    Hi @VegetarianOrc, thanks for asking us to test #2120. We ran a bounded comparison against current upstream main, commit 333910697b8f7080660a30e14745b68265f55f42, which contains the #2120 merge 7dcabf077e9e33840f3df080d8a7f113caa5ed9b. The baseline was the official PyPI 1.36.1 wheel (release tag 968c3c9dcf508ceca3f4e34b9ea910425351f6c5). Both arms used separate disposable services with the same Debian 13 x86-64 / glibc 2.41 environment, CPython 3.13.18, Temporal Server 1.32.0 and PostgreSQL 17.11.

    Check PyPI 1.36.1 Upstream main
    Our unchanged registration/replay regression, 3 independent runs of 2 child processes each 6/6 normal 6/6 normal
    Our unchanged observer witness, 5 cases 5/5 pass 5/5 pass
    Real Temporal/PostgreSQL booking regressions for SQL approval/history replay, worker restart and one effect with concurrent workers 3/3 pass 3/3 pass
    #2120 shutdown reproducer with pending log callbacks, 1 runtime and 3 runtimes Both hit the original 30 s subprocess timeout Both exit 0, all expected callbacks complete and both pipes reach EOF
    #2120 reproducer with RPC result delivered while the completion callback remains active Hit the original 30 s subprocess timeout Exit 0, one completed callback, empty stderr, both pipes reach EOF

    All six registration children per arm completed the RPCs, returned from asyncio.run(), reached ATEXIT and exited naturally with code 0 and pipe closure. The five separately instrumented observer bootstrap children per arm also showed stdout/stderr EOF and natural exit 0 after ATEXIT. The intermittent application hang did not reproduce on either arm in this local sample. The forced-callback comparisons above are the stronger evidence because they fail on the released wheel under the same conditions.

    A three-cycle explicit Runtime create/drop check returned to its original native thread count every time on both arms. FDs returned from 9 to the same plateau of 5 after every drop (3 before the first Runtime), with no per-cycle growth observed. In one targeted Linux fork control per arm, both rejected use of the inherited Runtime and completed an actual RPC with a fresh child Runtime. The baseline then exceeded a 35 s outer deadline waiting for the child to exit. With main, the child exited 0 and the parent reused its Runtime successfully. This was one fork scenario, with no broader claim about concurrent forks.

    Build provenance: unmodified official sources, SDK Core c675c32469af707e1c979a7b808c70b4a0608b82, Cargo.lock SHA-256 308f5168f34ed09e36444b72374adc10246dbf0e62cc677f76a6dc1dd75abd5d. Built with Rust/Cargo 1.99.0, Maturin 1.9.6, build CPython 3.13.5 and protoc 3.21.12 using maturin build --release --locked --compatibility linux. The local cp311-abi3-linux_x86_64 wheel SHA-256 is fd59c5d3b828fe0d79b6b8b22bbf1011a84a621cba1a2ab20824dbb0813aa16e. The PyPI baseline wheel SHA-256 is 5ce0ddfaf35316271eee632e1579890be6f3c0f1de7fddfd1f3086f3f83d6b78, and its native library matched the copy used in the tests.

    This gives us confidence in the shutdown change for the exercised cases. Our intermittent CI failure remains unverified against a released wheel containing the change. We kept our application dependency and CI unchanged. Thanks for the fix and the targeted reproductions.

  6. VegetarianOrc commented on Nov 19, 2026

    @VegetarianOrc
    Contributor

    Thanks for the confirmation! I'll close this issue. The patch will be released in the next version which is planned for next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions