test: Make the "Trapped" signal handler signal-safe - #365
Merged
leongdl merged 1 commit intoSep 3, 2026
Merged
Conversation
`test_notify_ends_process` fails intermittently. Seen on
`Python (macos-latest, 3.13)` while every other leg of the same matrix passed on
the same commit, including all six Windows legs and macOS 3.14:
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stdout>'>
Process pid 2878 exited with code: 1
assert 'Trapped' in [...]
The four `app_20s_run*.py` support scripts print and flush stdout in their main
loop, and their signal handler does the same. When the signal lands while the
interpreter is inside the loop's `sys.stdout.flush()`, the handler re-enters the
same `BufferedWriter`, CPython raises, "Trapped" never reaches stdout, and the
child exits non-zero -- so the test's `assert "Trapped" in all_messages` fails.
The handler now writes with `os.write(1, ...)`, which is unbuffered and cannot
re-enter. Ordering is unaffected: every `print` in these scripts is flushed on
the next line, so nothing is sitting in the buffer for the handler's output to
appear ahead of.
Reproduced deterministically rather than by waiting for the flake. Giving the
child a pipe nobody drains parks its main loop inside a blocked
`sys.stdout.flush()`, after which any SIGTERM lands inside the flush. Against the
current scripts that yields exit 1, the reentrant `RuntimeError`, and no
"Trapped" -- matching CI. Against the patched scripts, "Trapped" is present and
there is no `RuntimeError`.
All four copies are fixed, not just the one CI happened to hit: `app_20s_run.py`
and `app_20s_run_ignore_signal.py` under both `sessions_v0` and `sessions_v1`
were byte-identical in this handler.
An earlier attempt at a reproduction found a *different* race -- SIGTERM arriving
before `signal.signal()` had run, giving rc=-15 with no handler at all. That one
is not reachable from this test, which waits for "Log from test 0" before calling
`notify()`, so it is deliberately not addressed here.
test_subprocess.py, test_runner_base.py and test_wrap_cancelation.py give an
identical result before and after (6 failed, 189 passed); those 6 are
pre-existing gracetime failures on a machine with no bare `python` on PATH.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
wyongzhi
approved these changes
Sep 3, 2026
AlexTranAmz
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: no GitHub issue; observed as an intermittent
Code Qualityfailure, most recently on run 33807699249 (Python (macos-latest, 3.13)).What was the problem/requirement? (What/Why)
test_notify_ends_processfails intermittently. On the run above it failed onmacos-latest / 3.13while every other leg of the same matrix passed on the same commit — all six Windows legs and macOS 3.14 included:The four
app_20s_run*.pysupport scripts print and flush stdout in their main loop:and their signal handler does the same thing:
A signal handler runs on the main thread at an arbitrary bytecode boundary. When SIGTERM lands while the interpreter is inside the loop's
sys.stdout.flush(), the handler re-enters the sameBufferedWriter; CPython refuses,"Trapped"never reaches stdout, and the child exits non-zero. The test'sassert "Trapped" in all_messagesthen fails.Nothing in the package's own code is involved — this is the test's fixture racing itself.
What was the solution? (How)
The handler writes with
os.write(1, ...), which is a direct unbufferedwrite(2)and cannot re-enter the buffer:Ordering is unaffected: every
printin these scripts is flushed on the very next line, so there is never buffered content for the handler's output to appear ahead of. The comment in the code records that, so the next reader does not "tidy" it back into aprint.All four copies are fixed, not just the one CI happened to hit.
app_20s_run.pyandapp_20s_run_ignore_signal.pyunder bothsessions_v0andsessions_v1were byte-identical in this handler (same md5), so the same flake was latent in every test that uses them —test_subprocess.py,test_runner_base.py,test_wrap_cancelation.py,test_windows_process_killer.py.What is the impact of this change?
Test fixtures only. No
src/change, no behaviour change, no public interface change. It removes an intermittent CI failure whose only remedy today is a re-run.How was this change tested?
Yes, unit tests have been run.
Reproduced deterministically first, rather than waiting for the flake. Giving the child a pipe that nobody drains parks its main loop inside a blocked
sys.stdout.flush(), after which any SIGTERM is guaranteed to land inside the flush — no timing luck needed:TrappedpresentRuntimeErrorThe first row matches CI's log verbatim, including the exit code. The second is the same probe against the fix, which is the check that the fix is what closed it rather than the probe having stopped working.
Worth recording, because it changed what this PR does: a first attempt at a reproduction found a different race — SIGTERM arriving before
signal.signal()had run, givingrc=-15with no handler installed at all, in 13 of 250 trials. That is real but not reachable from this test, which waits for"Log from test 0"before callingnotify(), so the handler is always installed by then. It is deliberately not addressed here, and the probe was rewritten to target the mechanism CI actually reported.Regression:
test_subprocess.py,test_runner_base.pyandtest_wrap_cancelation.pygive an identical result before and after — 6 failed, 189 passed, 12 skipped, 8 xfailed. Those 6 are the same pre-existingtest_run_gracetime_when_process_ends_but_grandchild_uses_stdoutparametrisations, which fail on this machine because it has no barepythononPATH.test_notify_ends_processpasses both before and after locally, which is expected of a flake and is why the deterministic probe above is the load-bearing evidence rather than a green local run.Was this change documented?
Yes — the handler carries a comment naming the
RuntimeError, whywrite(2)cannot hit it, and why ordering is safe.Is this a breaking change?
No. Test support files only.
Does this change impact security?
No. It replaces a buffered write to fd 1 with an unbuffered write to fd 1, in a test fixture.
Cross-port to openjd-rs
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.