fix(app): exit 128+signum on SIGTERM so a kill is not a clean exit - #121
Merged
Merged
Conversation
Main used signal.NotifyContext and then returned normally, so a SIGTERM kill was indistinguishable from a voluntary quit in every signal an operator has: launchd reports `last exit code = 0`, and a wait-based supervisor sees status 0. That is what made the restart storm in #112 take hours to find -- hundreds of kills that all looked like clean exits. Use an explicit signal.Notify channel so the signal is known, log "shutting down: received <signal>", and exit 128+signum (143 for SIGTERM, 130 for SIGINT). init/pi-web.service gains SuccessExitStatus=130 143 so systemd still treats those as a clean stop; launchd's KeepAlive=true restarts regardless of exit status, so macOS behaviour is unchanged. Two consequences of the rework: - Main now waits for the shutdown goroutine instead of returning as soon as ListenAndServe reports ErrServerClosed, which previously let the process exit while srv.Shutdown() and manager.Close() were still running. - Exiting via os.Exit skips deferred functions, so the state-file unlink is now a named closure called from both the defer and the signal path. Without that, every SIGTERM would strand pi-web-state.json and the next start would report "another pi-web instance appears to be running". Closes #119
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mainusedsignal.NotifyContext(..., os.Interrupt, syscall.SIGTERM)and then returned normally, so aSIGTERMkill was indistinguishable from a voluntary quit in every signal an operator has:last exit code = 0— identical to "user stopped it"wait-based supervisor sees status0and logs a voluntary exitThat ambiguity is what made #112 take hours to diagnose: hundreds of
kickstart -kkills that all read as clean exits, sending the reporter through launchd environment,stdinEOF,ThrottleInterval, plist churn, and bind conflicts before the actual cause. The storm is fixed (#113), but the diagnostic gap would cost the next person the same time for any other kill source.This switches to an explicit
signal.Notifychannel so the signal is known, logsshutting down: received <signal>, and exits128+signum(143 forSIGTERM, 130 forSIGINT).Supervisor impact:
init/pi-web.servicegainsSuccessExitStatus=130 143, sosystemctl stopstill records a clean stop andRestart=on-failuredoes not fire.install.shcopies the unit verbatim and reinstalls it when it changes, so existing installs pick this up on the next upgrade. Until they do, an externalkill -TERM(notsystemctl stop, whose intent systemd tracks) would count as a failure and trigger a restart on those hosts.KeepAlive=trueincom.pi-web.plistrestarts regardless of exit status, so macOS behavior is unchanged; only the recorded exit code becomes informative.stopServerine2e/lib/server.tssendsSIGTERMand does not inspect the exit code.Two consequences of the rework, both deliberate:
Mainnow waits for the shutdown goroutine rather than returning as soon asListenAndServereportsErrServerClosed. Previously the process could exit whilesrv.Shutdown()andmanager.Close()were still running. (httpServer.Shutdownhas exactly one call site, so the wait cannot hang.)os.Exitskips deferred functions, so the state-file unlink is now a named closure called from both thedeferand the signal path. Without that, everySIGTERMwould strandpi-web-state.jsonand the next start would reportanother pi-web instance appears to be running.docs/sequence-flows/server-startup.mdis updated to match.Related issue
Closes #119
(Split out of #112, secondary finding 3, reported by @laulpogan.)
Type of change
fix— bug fixLive vs. Export
Testing
make checkpasses (test + build + vet)vitest) cover the changego test ./...) cover the changeNo automated test: the behavior is process-level (signal → exit status), and the test suite has no harness that spawns and signals the built binary. Verified by hand against the built binary (macOS, isolated
PI_CODING_AGENT_DIR, port 31999):SIGTERM143shutting down: received terminatedSIGINT130shutting down: received interruptReproduce with:
The state-file check matters: an earlier build of this branch, before the
releaseStateFileclosure was shared with the signal path, leftpi-web-state.jsonbehind on everySIGTERM— confirmed by running that build. The systemdSuccessExitStatuschange is the remaining unexercised part and still wants a look from someone on a systemd host, since CI cannot cover it.