`launchctl kickstart -k` in the pi extension turns one health-check blip into a self-sustaining restart storm

[laulpogan]

pi-web: v0.0.1-beta.36 (@ygncode/pi-web)
pi: 0.84.4 (/opt/homebrew/bin/pi)
Host: macOS 15.8, arm64, kernel 24.6.0, supervised by ~/Library/LaunchAgents/com.pi-web.plist (launchd gui/501)
Tailscale: CLI 1.98.3, tailscaled 1.98.9 (macOS app; version skew prints a warning on stderr but does not affect these findings)

Symptom

pi-web under launchd restarts continuously — hundreds of times — while the *same binary run by hand in a terminal is perfectly stable.

$ launchctl print gui/$(id -u)/com.pi-web | egrep 'runs = |last exit|exit timeout'
        runs = 582
        last exit code = 0
        exit timeout = 5

(The plist written by install.sh sets KeepAlive=true and no ExitTimeOut, so the exit timeout = 5 above is launchd's default, not a configuration on my side.)

Observed on my machine: 62 exits in 14 minutes (~6/min) during one burst, then quiet for 90 minutes, then bursts again. 502s from tailscale serve during a burst, so the phone PWA intermittently cannot reach the server at all.

The last exit code = 0 sends you hunting in entirely the wrong direction. I spent hours ruling out stdin EOF, launchd environment, ThrottleInterval, plist churn, a second instance, a bind conflict, and GUI-session teardown — all while the actual agent was a SIGTERM that pi-web handles and exits 0 for (see finding 3, which is why this is so misleading).

Root cause

.pi/extensions/pi-web.ts opportunistically starts the server on every extension load — i.e. once per pi process, including the pi --mode rpc workers that pi-web itself spawns:

// pi-web.ts:759 (default export)
void detectHostPort(pi)
  .then((detected) => {
    if (!detected) return;
    return ensurePiWebRunning(pi, detected.host, detected.port);
  })

ensurePiWebRunning (pi-web.ts:314) health-checks once and, on any failure, calls startPiWeb:

// pi-web.ts:319
if (await healthCheck(host, port)) return true;
try { await startPiWeb(pi); } catch { return false; }

startPiWeb (pi-web.ts:221) starts the service with:

# pi-web.ts:236
launchctl bootstrap "gui/$(id -u)" "$plist" 2>/dev/null || launchctl load "$plist" 2>/dev/null || true
launchctl kickstart -k "gui/$(id -u)/com.pi-web" 2>/dev/null || launchctl start com.pi-web

kickstart -k kills the currently running instance. It does not mean "start if needed" — it means "stop it now, then start". pi-web handles SIGTERM and shuts down cleanly, so launchd records a successful exit (last exit code = 0), which looks like a voluntary quit and not like being killed.

Why it becomes a storm rather than one restart

The kill briefly takes the port down, so every other pi process that starts during that window also fails its health check and issues its own kickstart -k. Each restart creates more failures. The amplifier closes completely because pi-web spawns pi --mode rpc workers, which load this extension — the server is being killed by its own children.

That also explains the two most confusing properties of the bug:

• It is bursty. It only runs while pi processes are starting. With ~200 sessions and a load average of 40–59, my machine fails the 1 s check routinely, so bursts sustain themselves for tens of minutes; overnight, with nothing starting, there are zero restarts.
• A manually started pi-web is immune. kickstart -k addresses the launchd job, not a hand-run process. So "it's broken under launchd but fine in a terminal" is not a launchd/environment problem at all — it is the signature of this code path. If someone else hits this bug, that comparison will actively mislead them for hours.

Evidence

Wrapped the launchd child in a supervisor that traps TERM/INT/HUP, forwards, and records the cause. Then ran both commands against the live job:

baseline (launchd owns :31415)      runs=1   child=6326

A) patched start path (see Proposed fix)
                                    runs=1   child=6326   tailnet 200   ← job untouched

B) launchctl kickstart -k gui/501/com.pi-web      (what beta.36 does)
   +2s                              runs=2   child=10470                 ← instance replaced
   supervisor log:
     DEAD  child pid=6326 status=0 after 115s (supervisor was signalled and forwarded it)

status=0 for a SIGTERM death — the exact value that made this look like a clean voluntary exit for hours.

Minimal reproduction of the atomic action (watch the counter, then kill):

launchctl print gui/$(id -u)/com.pi-web | grep 'runs = '
launchctl kickstart -k gui/$(id -u)/com.pi-web
launchctl print gui/$(id -u)/com.pi-web | grep 'runs = '   # +1, child pid replaced within ~2s

Proposed fix for the storm

An automated, load-time "is it running?" check must never be able to kill a live server. Two changes:

1. In startPiWeb's darwin branch, treat something bound to the port as running, and drop the kill from the automated path:

launchctl bootstrap "gui/$(id -u)" "$plist" 2>/dev/null || launchctl load "$plist" 2>/dev/null || true
if nc -z '<host>' '<port>' 2>/dev/null; then exit 0; fi   # bound != not running
launchctl start com.pi-web 2>/dev/null || true

2. In ensurePiWebRunning, confirm before touching a service other processes are using — one fetch is not proof of death:

if (await healthCheck(host, port)) return true;
await new Promise((r) => setTimeout(r, 1500));
if (await healthCheck(host, port)) return true;

/pi-web restart (pi-web.ts:299) should keep kickstart -k — an explicit user request to restart should kill. Only the implicit path needs the kill removed. I can open a PR; the patch above is what I run locally (with the timeout bump below) and the storm has not recurred since.

Secondary finding 1 — the 1 s health-check timeout is the false-negative source

// pi-web.ts:206
const res = await fetch(`http://${host}:${port}`, { signal: AbortSignal.timeout(1000) });

A healthy server answers in ~3 ms when the machine is idle (measured: 40 samples, max 0.003 s, mean 0.001 s). Under load, or while a restart is in flight, it exceeds 1 s — and because 401/403 already count as "up", the only way to fail is a timeout or connection refusal, i.e. precisely the transient states a busy machine produces. A false negative here is very expensive (it restarts a service other people are using), so 4 s plus the confirmation retry above is the cheap fix. The restart loop then amplifies itself, which is why the check is not the whole bug — but it is the ignition.

Secondary finding 2 — serveRuleConflict false alarm makes pi-web refuse its own Serve rule

tailscaleServeRuleState (internal/app/tailscale.go:120-142, with findJSONKey at :144 and collectJSONStrings at :165) looks for the port as an object key anywhere in the JSON, then compares the strings under that key to http://127.0.0.1:<port>:

rule, ok := findJSONKey(status, port)
if !ok { return serveRuleMissing, nil }
strings := collectJSONStrings(rule)
for _, s := range strings { if s == target { return serveRuleSame, nil } }
return serveRuleConflict, nil

In the real tailscale serve status --json, the only key equal to the port is under TCP, and it holds no strings at all. The matching proxy lives under Web, whose key is host.example.ts.net:31415 — never equal to "31415", so it is never reached:

{ "TCP": { "31415": { "HTTPS": true } },                    // ← exact-key match; 0 strings
  "Web": { "host.example.ts.net:31415":                     // ← key contains the port, never matches
             { "Handlers": { "/": { "Proxy": "http://127.0.0.1:31415" } } } } }

Enumerated against my live output: the only exact match is $.TCP.31415 → {"HTTPS": true} → collectJSONStrings returns [] → serveRuleConflict is the only possible outcome once a rule exists on that port — including a rule pi-web created itself, since configureTailscaleServe runs tailscale serve --bg --https=<port> <target> whenever the rule state is serveRuleMissing.

Consequence, on every start (configureTailscaleServe is called unconditionally unless --host is set, internal/app/app.go:130):

Tailscale Serve unavailable: tailscale HTTPS port 31415 is already configured for another service;
not overwriting it. To replace it, run: tailscale serve --bg --https=31415 http://127.0.0.1:31415

1135 occurrences in my /tmp/pi-web.error.log (one per start). So a rule that is already correct is reported as someone else's, tailscaleUrl stays empty in pi-web-state.json, and authMiddleware.AllowHost(tsURL) (app.go:137) is never called. I could not demonstrate user-visible breakage from the missing AllowHost — token auth over the tailnet works fine today — so I am flagging that as something to check, not as a proven failure. The visible cost is the noise plus pi-web never adopting/verifying its own Serve rule.

Fix: match on the value rather than the port-as-key, e.g. walk the JSON for any string equal to target, or read Web["<dns-name>:<port>"].Handlers[*].Proxy explicitly. Treating TCP[port] == {"HTTPS": true} with a matching Web proxy as serveRuleSame would also do it.

Secondary finding 3 — exiting 0 on SIGTERM destroys the diagnostic

internal/app/app.go:189 uses signal.NotifyContext(..., os.Interrupt, syscall.SIGTERM) and the process then returns from Main normally, so a SIGTERM kill is indistinguishable from a clean shutdown in every external signal available to an operator:

• launchd reports last exit code = 0 (identical to "user stopped it").
• A wait-based supervisor sees status 0 and logs a voluntary exit.

That ambiguity is what made this take so long to find, and it will cost the next person the same time. Suggested: log one line on shutdown cause ("shutting down: received SIGTERM") and/or exit 143 (128+15) so last exit code carries the information. SIGTERM from systemctl stop/launchctl stop would still be a documented, expected non-zero — worth a note in the docs if you prefer to keep 0, but then a supervisor-visible cause string is the alternative.

What the design got right

internal/app/state_file.go:21 (writeStateFile) plus state_file_unix.go's non-blocking flock on pi-web-state.json mean a second instance exits 1 with another pi-web instance appears to be running (state file at ... is locked); exit it first, or remove the file if stale. That message is accurate, actionable, and it is the one thing that finally made me look at process ownership instead of the launchd environment. The exit codes distinguishable (0 clean/SIGTERM, 1 lock/1 config error) is exactly the property the shutdown path is missing.

Priority

The storm is the highest-impact item: it intermittently takes down the remote/phone path, it is self-sustaining, it is caused by pi-web's own extension, and the standard debugging intuition ("fine by hand ⇒ launchd problem") leads away from it. Findings 2 and 3 are each a few lines.

[laulpogan]

Opened #113 with the storm fix (the extension change in Proposed fix above, plus the 1 s → 4 s health-check timeout). Verified here: their extension suite passes 46/46 with the patch, and the restart loop has not recurred. Findings 2 and 3 are not in that PR.