Repository navigation
Description
What are you trying to do?
Run short-lived Python processes using the Temporal Python SDK in integration tests, including namespace registration and workflow/worker observations, and let the parent process wait for normal exit and closed stdout/stderr pipes.
Describe the bug
With the official PyPI temporalio==1.35.0 wheel, a child occasionally finishes its asyncio work, prints its last phase marker and reaches its atexit marker, yet never exits. The parent eventually terminates the child with its unchanged watchdog (roughly 29–30 seconds). This is intermittent: other runs of the same tests exit normally.
We independently observed the same native stack categories described in #2046, which was closed as not planned by its reporter. We commented there with our initial findings, but are filing this separately because the defect still reproduces for us, and we have additional independent evidence.
Environment
- OS: GitHub-hosted Ubuntu 24.04, Linux x86_64
- SDK: official
temporalio==1.35.0, not our experimental wheel - CPython 3.13.17 in frozen diagnostic runs; normal CI also reproduced post-
atexitsurvivors with 3.13.18 - Temporal Server 1.32.0 in the relevant integration tests
- Short-lived subprocesses invoked by integration/browser test harnesses
Native evidence
In real hung CI children, GDB classified:
- Main thread:
CoreRuntimeHandle::drop/ Tokio runtime shutdown and synchronization - Another native thread: Python GIL acquisition, including
PyThread_hang_thread, in a Temporal/Tokio context
One 3.13.18 CI child reached ASYNCIO_RETURN and ATEXIT around 0.8 s but exited only after the watchdog sent SIGTERM about 29 s later. Other naturally completed children exited normally.
These observations strongly match #2046's described finalization deadlock. We have not independently established the full pyo3_async_runtimes::set_result callback/owner/wait-for cycle in our optimized Linux x86_64 SDK build. Not every unrelated test timeout has been classified as this bug.
Attempted candidate fix (isolated only)
We built a separate, hash-verified test-only Python wheel from SDK 1.35.0 sources with the one-file finalization patch proposed in #2047. No application dependency or normal CI used this custom build.
- Bounded ordinary runtime create/drop tests exited normally for both official and patched wheels; worker threads disappeared after each normal drop.
- We observed two stable process-global Unix socket descriptors after the first runtime initialization, with no additional growth over three cycles in either wheel; exact initialization ownership is not conclusively established.
- A deliberately injected exception from
sys.is_finalizing()revealed a concern in the candidate patch: three consecutive runtime drops retained 2, 4, 6 native threads in the patched wheel, versus 0, 0, 0 in the official wheel. - This injection does not establish that such an exception occurs in production, but it makes an unconditional retain-on-check-error path unsuitable for us without a narrowly justified lifetime policy.
- Unmodified PostgreSQL/Temporal registration integration controls passed once per arm, but we did not reproduce the native wait cycle in a paired baseline-versus-patched experiment. We therefore do not claim the candidate fixes the intermittent hang.
We have not rolled out a patched SDK or modified existing test timeouts.
Expected behavior
After normal completion of the asyncio work and cleanup, short-lived subprocesses should exit and close their pipes without hanging in native runtime shutdown.
Questions for maintainers
- Is this recognized as an SDK interpreter-finalization / native-callback lifecycle bug, and is an upstream correction planned?
- Was there a correctness concern behind the closure of #2047, or is another approach preferred?
- What is the recommended abi3-compatible strategy for finalization-safe Core runtime destruction without retaining threads if the Python finalization check fails outside interpreter shutdown?
- Would a minimal, sanitized reproducer and a focused regression-tested PR be welcome?
We can provide further sanitized stack categories, reproducibility details and build/test evidence. We would prefer an upstream fix over maintaining a private SDK variant.
Activity
AppSolves commentedon Nov 16, 2026
Hi @VegetarianOrc and @Sushisource, a polite but urgent follow-up from the original reporter. Could we please get a maintainer triage update on this issue?
The problem is still affecting our release/integration CI, and we've since confirmed the same post-
atexitprocess-exit hang with the officialtemporalio==1.36.0wheel, not just 1.35.0. In recent Linux x86_64 / CPython 3.13.18 runs, a short-lived Temporal observer/registration subprocess completed its operations and emittedASYNCIO_RETURNandATEXIT(in about 0.5–1.5 seconds), but remained alive with 3 native threads, 9 FDs and a FUTEX wait until the unchanged parent watchdog sent SIGTERM at about 29 seconds. Other identical-path children exited normally. We have multiple separately recorded CI occurrences.We are not claiming to have proven the exact native wait-for cycle for these latest processes, nor that every Temporal test timeout shares this cause. However, the recurring, specifically instrumented post-
ATEXIThangs are making an otherwise green integration suite intermittently fail and consuming significant engineering/review time. We have intentionally not masked the issue with timeout increases, disabled checks,os._exit, or a private SDK patch.I noticed the related discussions in #300 and the proposed approaches in #2047 and #2062, which are both closed without a merged fix. Could you clarify:
Even a brief status and a recommended next step would help us plan responsibly. We're happy to contribute further sanitized evidence and testing, and we appreciate the maintainers' work on this SDK. Thank you!
VegetarianOrc commentedon Nov 17, 2026
Hey @AppSolves,
Thanks for taking the time to report the issue. Here's the current answers to your questions:
AppSolves commentedon Nov 18, 2026
Thanks for the update and for prioritizing this.
Important update (18 Nov 2026): the same post-
atexithang has now been confirmed on GitHub-hosted Ubuntu 26.04.1 as well. Ubuntu 26.04 does NOT reliably prevent this issue. This new observation supersedes any tentative implication from our earlier small Ubuntu comparison.In a Main CI run on Ubuntu 26.04.1 LTS (GitHub runner image
ubuntu-26.04, version20261111.196.1), CPython 3.13.18, and the official PyPItemporalio==1.36.1x86-64 wheel, one short-lived Temporal observer child (PID 17681) successfully completedCONNECT,START_WORKFLOW,DESCRIBE,ASYNCIO_RETURN, andATEXITwithin approximately 616 ms, but did not exit normally. At approximately 28 seconds it still had 3 native threads, 9 FDs and one FUTEX wait; the unchanged parent watchdog sentSIGTERMat approximately 29 seconds, after which both pipes closed. The test failed withOBSERVATION_UNAVAILABLE:TIMEOUT. This was the simulated-confirmation E2E test in CI run44203628015, job134463835512(18 Nov UTC; Main SHA13bb1ac74492c9f2d5aef6339b04fb5cd7d40f3f). The preceding real Temporal booking and witness steps passed; the subsequent Voice-send step did not run. We classify this as the documented post-atexitprocess-exit hang, not merely a namespace/setup timeout. CI job evidence (our repository is private; the key diagnostic facts are summarized here).Separately, we ran a bounded local follow-up looking for a smaller reproducer. On Linux x86-64 in Debian 13 (trixie) containers under Docker Desktop/WSL2, with CPython 3.13.18 and Temporal Server 1.32.0, we ran six isolated child processes with each of the official SDK wheels 1.36.0 and 1.36.1 (12 total). Each child connected, started a synthetic workflow and called
describe(). All 12 returned fromasyncio.run(), reachedATEXIT, exited with code 0 and closed both output pipes (0/12 reproduced). No application code or worker was involved in these minimal children.For historical context only, an earlier small GitHub-runner comparison showed Ubuntu 24.04 / CPython 3.13.17: 1/3 confirmed post-
atexithangs; Ubuntu 26.04 / CPython 3.13.17: 0/3; and Ubuntu 26.04 / CPython 3.13.18: 0/3. These negative samples do not imply an Ubuntu-26 fix, as the newly confirmed Ubuntu 26.04.1 / CPython 3.13.18 / SDK 1.36.1 recurrence demonstrates. Those runs also differed in glibc, OpenSSL, compiler and Python builds, so we cannot isolate an OS-specific cause.We still do not have a reliable minimal standalone reproducer. The native wait-for cycle/owner has not been conclusively proven, and we have not patched the SDK or relaxed test timeouts. If there is a particular reduced lifecycle or environment comparison that would be most helpful, we can collect more narrowly scoped, sanitized evidence.
VegetarianOrc commentedon Nov 19, 2026
Hi @AppSolves, we've recently merged #2120 that aims to alleviate this. Is it possible for you to run your reproduction steps against the current main branch to help gain confidence in the approach?
AppSolves commentedon Nov 19, 2026
Hi @VegetarianOrc, thanks for asking us to test #2120. We ran a bounded comparison against current upstream
main, commit333910697b8f7080660a30e14745b68265f55f42, which contains the #2120 merge7dcabf077e9e33840f3df080d8a7f113caa5ed9b. The baseline was the official PyPI1.36.1wheel (release tag968c3c9dcf508ceca3f4e34b9ea910425351f6c5). Both arms used separate disposable services with the same Debian 13 x86-64 / glibc 2.41 environment, CPython 3.13.18, Temporal Server 1.32.0 and PostgreSQL 17.11.All six registration children per arm completed the RPCs, returned from
asyncio.run(), reachedATEXITand exited naturally with code 0 and pipe closure. The five separately instrumented observer bootstrap children per arm also showed stdout/stderr EOF and natural exit 0 afterATEXIT. The intermittent application hang did not reproduce on either arm in this local sample. The forced-callback comparisons above are the stronger evidence because they fail on the released wheel under the same conditions.A three-cycle explicit Runtime create/drop check returned to its original native thread count every time on both arms. FDs returned from 9 to the same plateau of 5 after every drop (3 before the first Runtime), with no per-cycle growth observed. In one targeted Linux fork control per arm, both rejected use of the inherited Runtime and completed an actual RPC with a fresh child Runtime. The baseline then exceeded a 35 s outer deadline waiting for the child to exit. With main, the child exited 0 and the parent reused its Runtime successfully. This was one fork scenario, with no broader claim about concurrent forks.
Build provenance: unmodified official sources, SDK Core
c675c32469af707e1c979a7b808c70b4a0608b82, Cargo.lock SHA-256308f5168f34ed09e36444b72374adc10246dbf0e62cc677f76a6dc1dd75abd5d. Built with Rust/Cargo 1.99.0, Maturin 1.9.6, build CPython 3.13.5 and protoc 3.21.12 usingmaturin build --release --locked --compatibility linux. The localcp311-abi3-linux_x86_64wheel SHA-256 isfd59c5d3b828fe0d79b6b8b22bbf1011a84a621cba1a2ab20824dbb0813aa16e. The PyPI baseline wheel SHA-256 is5ce0ddfaf35316271eee632e1579890be6f3c0f1de7fddfd1f3086f3f83d6b78, and its native library matched the copy used in the tests.This gives us confidence in the shutdown change for the exercised cases. Our intermittent CI failure remains unverified against a released wheel containing the change. We kept our application dependency and CI unchanged. Thanks for the fix and the targeted reproductions.
VegetarianOrc commentedon Nov 19, 2026
Thanks for the confirmation! I'll close this issue. The patch will be released in the next version which is planned for next week.