Skip to content

fix: Re-dispatch SIGTERM to the handler add_signal_handler displaces - #226

Open
selimacerbas wants to merge 1 commit into
restatedev:mainfrom
selimacerbas:fix/sigterm-redispatch-displaced-handler
Open

fix: Re-dispatch SIGTERM to the handler add_signal_handler displaces#226
selimacerbas wants to merge 1 commit into
restatedev:mainfrom
selimacerbas:fix/sigterm-redispatch-displaced-handler

Conversation

@selimacerbas

Copy link
Copy Markdown
Contributor

What happens

asgi_app installs its SIGTERM handler on the first request it serves:

loop.add_signal_handler(signal.SIGTERM, _on_sigterm)

Underneath, in both uvloop and stdlib asyncio, add_signal_handler installs a dummy handler through signal.signal. That silently replaces whatever the hosting ASGI server registered. uvicorn registers Server.handle_exit in Server.capture_signals(), so once the SDK has served one request, uvicorn no longer has a SIGTERM handler.

_on_sigterm notifies the active receive channels and returns. Nothing tells the host server to shut down, so a uvicorn-hosted endpoint stops responding to SIGTERM at all.

What I measured

Endpoint served by uvicorn in a container with a 45 second stop grace period:

  • docker kill --signal=SIGTERM leaves the process running. Still serving at T+72s.
  • /proc/1/status shows SIGTERM caught, not blocked and not ignored, so the signal does arrive.
  • signal.getsignal(SIGTERM) returns asyncio's _sighandler_noop rather than uvicorn's handle_exit.
  • The listening socket stays open. uvicorn never starts its drain.
  • Docker force-kills at the end of the grace period. Exit 137 on every stop and every deploy.

The change

Remember the handler that add_signal_handler displaces, then call it from _on_sigterm once the channels have been notified. The re-dispatch runs on the event loop rather than in signal context, which is why frame is None.

If there was no previous handler, meaning SIG_DFL or SIG_IGN, nothing is called and the standalone case behaves as before.

I kept add_signal_handler instead of switching to signal.signal, since _on_sigterm touches asyncio objects and wants the loop-deferred dispatch that add_signal_handler gives it.

Tests

Two tests in tests/server.py:

  • a host handler installed through signal.signal gets re-dispatched after a real raise_signal(SIGTERM)
  • raising SIGTERM with no previous handler installed does not error

Notes

I ran into this on Restate workers that were being force-killed on every deploy. We worked around it downstream by re-asserting uvicorn's handler from a poll loop. That holds, but the displacement seemed like something the SDK should deal with rather than each host.

loop.add_signal_handler installs a dummy handler through signal.signal, so
installing ours silently replaces whatever the hosting ASGI server registered.
uvicorn registers Server.handle_exit in Server.capture_signals(), and _on_sigterm
never stops the server, so a uvicorn-hosted endpoint stopped reacting to SIGTERM
once it had served one request. The host then gets force-killed when its
supervisor's stop grace period expires.

Remember the displaced handler and call it from _on_sigterm after the channels
have been notified. Re-dispatch happens on the event loop rather than in signal
context, hence frame=None. SIG_DFL and SIG_IGN are not callable, so the
standalone case is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant