fix(deploy): make-live.sh takes a lock — one deploy at a time per box (v0.448.4)

[vikasprogrammer]

Why

Every live checkout is shared by whoever runs the deploy, so two concurrent make-live.sh runs are two git reset --hards racing each other's builds. The second moves a checkout out from under the first one's npm run build; the first then restarts a service with a binary built from a commit it never resolved — and reports success, because it verifies /health against the version it was told to expect.

On 2026-09-17 two sessions ran this concurrently against all four tenants. It was harmless only because both happened to be deploying the same sha. Luck, not design.

The lock

An atomic mkdir (macOS ships no flock) holding a meta file — the directory is the mutex, the meta only explains who holds it. A second run fails fast naming the holder's pid, user and start time, rather than queueing: a deploy that waits ten minutes and then runs against whatever main has become by then is its own surprise.

another deploy is already running on this box (pid 81341, vikas@mac-mini, started 2026-09-18 10:40:00).
  Deploys share every live checkout, so running two at once can restart a service on a half-synced build.
  Wait for it to finish, or — if you are certain it is dead — re-run with --force-lock.

A lock that can't be released would be worse than none

Every way it could wedge has an exit, and each is pinned by a test:

Situation	Behaviour
Holder PID is dead (Ctrl-C, terminal died)	cleared automatically
Live PID but ancient (wedged run, or PID reuse)	cleared past AOS_LIVE_LOCK_MAX_AGE, default 2h
Meta unreadable (interrupted writer)	cleared
You know it's dead and won't wait	--force-lock
Run fails in preflight	released on EXIT — a crashed deploy never blocks the next one
You lost the race	you never delete the winner's lock

--dry-run is deliberately not gated — it changes nothing and it's what you reach for to see what the other run is doing — but it warns that it's reading checkouts another run is moving. --help is untouched.

Scope is one box. Two different machines deploying to the same remote tenant still race; that needs a lock on the remote side and isn't what bit us. Noted in the script header.

A trap worth recording

Under set -euo pipefail, sed missing-file | head exits non-zero and killed the script outright — silently, with the lock still in place. That's exactly the interrupted-writer case the staleness branch exists to clear, so it would have turned a self-healing path into a permanent wedge. The meta is now read into a variable first. The test caught it.

Testing

• scripts/make-live-lock-test.cjs (new, in the gate) — 24 checks running the real script with a scratch lock path and a deliberately bogus target, so every case fails fast in preflight after the lock has been taken and released. Nothing deploys, no checkout is touched.
• Verified end to end against the real lock path: a real make-live.sh invocation was refused before touching any checkout, and the holder's lock survived. A real --dry-run still reports all four tenants normally.
• Full npm run test:governance green.

🤖 Generated with Claude Code