fix(execution): remove CC's SCM dependency edges so the monitor watchdog's restart isn't refused (ALP-945)

[jatassi]

Summary

• Removes both of AlphaMindCommandCenter's SCM DependOnService edges (on alphamind-scheduler + alphamind-monitor) from the install script. During the 2026-06-10 incident the SCM refused all sixteen of the monitor watchdog's nssm restart attempts for 38 minutes because CC (a running dependent) blocked the stop — the ~150 s wedge bound ALP-941 built was in practice unbounded. The edge's documented rationale (fail-fast SSE consumers) was never true of the shipped code: the consumers reconnect forever with capped backoff and the control proxy hits upstreams per-request.
• Surfaces nssm's stderr at the watchdog restart seam: _checked_run captures output (errors="replace" so undecodable bytes can't displace the CalledProcessError), and restart() logs exc.stderr distinctly — the SCM's refusal reason was previously invisible ("exit status 1" only). Swallow-and-retry semantics unchanged; the seam is shared by both watchdogs.
• Documents the topology invariant where topology is defined: install_monitor_service.ps1 and install_safety_core_service.ps1 now state that no service may declare an SCM dependency on the supervised service.
• Rewrites the runbook passages built on the dependency: plain single-service nssm restart works with everything else running; §8.9's manual fallback drops the stop-CC choreography; §1.4/§1.6 orderings are framed as operational convention. Also fixes ALP-941 staleness in the CC runbook (seven services; monitor-watchdog added to the recovery stop/start lists).

Closes https://linear.app/alphamind-jatassi/issue/ALP-945/monitor-watchdog-cannot-restart-the-monitor-in-production-nssm-restart

Deploy notes (operator-executed, prod box, elevated PowerShell)

The live SCM config was written at install time — pulling this PR changes nothing on the installed service. After merge:

1. sc.exe config AlphaMindCommandCenter depend= /
2. Confirm sc.exe qc AlphaMindCommandCenter shows an empty DEPENDENCIES list (effective immediately; no service restart required).
3. Prove the fix end-to-end: with CC running, nssm restart alphamind-monitor exits 0 and the monitor comes back healthy (fresh PID/StartTime, 8766 heartbeat within ~18 s).

The fix is not live until step 1 runs.

Test plan

• CI (.github/workflows/ci.yml) green on the PR — lint on Linux + full pytest on Windows
• Local lint chain clean (ruff check, ruff format --check, mypy, lint-imports)
• uv run pytest tests/execution/process_supervision/ -n auto — 13 passed (incl. the two new stderr criteria)
• grep -ci depend scripts/install_command_center_service.ps1 returns 0
• ALP-945 acceptance criteria all met (final one is the post-deploy prod proof above)

🤖 Generated with Claude Code

[linear]

ALP-941 Monitor wedges on a frozen event loop — in-loop watchdog can't fire, no out-of-process watchdog (ALP-841 recurrence)

Summary

The alphamind-monitor service silently wedged for ~8 hours on 2026-06-09 — it froze around 00:30 MT (06:30 UTC) and was discovered ~08:20 MT. The process stayed alive (NSSM "Running") but its asyncio event loop stopped entirely: no fills ingested, no :8766 SSE heartbeat, no periodic tasks. The monitor's liveness watchdog is a coroutine running on the same event loop it guards, so a freeze of that loop makes the watchdog unable to ever reach its os._exit(1). Unlike the safety core (ADR-0004 / ALP-857), the monitor has no out-of-process watchdog, so nothing restarted it. This is the ALP-841 failure class recurring through a different freeze cause: the ALP-841 broker-call mitigation did land (get_orders is now offloaded), but the loop can still freeze and remain unrecoverable.

This issue fixes the confirmed disease — an unrecoverable loop freeze — and adds freeze introspectability. The trigger of this specific freeze (sustained WAL write-lock contention) is a distinct, not-yet-confirmed defect class spun off to ALP-942; the watchdog here makes a freeze recoverable regardless of cause, so the two ship independently. The exact blocking frame is still open (a 34 MB minidump was captured for offline analysis); the faulthandler change below ensures the next wedge self-captures its frame.

Impact (high severity)

During the outage the monitor processed no broker fills, so local state diverged from the broker and the ALP-937/938/939 partial-close re-protection never fired. After the 2026-06-09 09:35 ET market_open run the broker had executed GS 2→1, MRVL 8→4 (partial closes) and a new SLB 44 entry, but locally none of it registered. Consequently GS (1sh), MRVL (4sh), and TSM (5sh) sat with no protective orders at the broker during RTH — the broker-bracket "floor" that is supposed to make a monitor freeze fail-safe was defeated, because the partial-close path had already cancelled the old protective legs (ALP-937/939) and the re-protect (ALP-938) depends on the frozen monitor processing the close fill. Separately, a 45 MB un-checkpointed WAL accumulated (the frozen connection pins the DB and starves checkpoints) — system-wide contention that also drags the scheduler; that strand is ALP-942.

Detection / symptom

A wedged monitor presents as: process alive (NSSM "Running", old StartTime) but curl :8766/events returns no heartbeat (TCP accepts, then http_code=000), monitor.log frozen, and recent broker fills not reflected in local positions. Confirm via OS process state — a single thread, zero CPU accumulation over a sample window, WaitReason=UserRequest. Get-Service / NSSM status alone is NOT a liveness signal (the documented ALP-841 lesson).

Evidence (2026-06-09, prod / HEAD 420d2da)

monitor PID 10408, StartTime 2026-06-08 22:45:51 MT (restarted onto the ALP-940 deploy)
monitor.log last line: 2026-06-09 00:30:56 MT (06:30:56 UTC)
  ERROR ...activities_backfill.task  fill_backfill sweep failed; retrying next interval
  sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
  (prior identical caught+retried block at 00:00:55 MT)
:8766/events  -> TCP connect OK, then http_code=000 after 22s (no headers, no heartbeat)
process       -> Threads=1, CPU delta 0s over 3s sample, WaitReason=UserRequest, WS 7.5 MB
WAL           -> alphamind.db-wal = 45 MB, un-checkpointed
broker truth  -> GS 1 / MRVL 4 / SLB 44 / TSM 5 all FILLED; GS+MRVL+TSM had NO broker stops

The contention that triggered the freeze (the database is locked blocks, the 45 MB WAL) is the ALP-942 evidence chain; this issue's evidence is the unrecoverability: a frozen loop with an alive process and no external restarter.

Root cause (confirmed)

MonitorSupervisor._watchdog_loop (src/alphamind/execution/continuous_monitor/supervisor.py:268-334) is a coroutine created in the same asyncio.TaskGroup as the watched tasks (lines 247-250). It detects a stall via await self._sleep(self._check_interval()) (line 279) and only then calls os._exit(1) (line 334) for NSSM to restart. When the loop thread is synchronously blocked, that asyncio.sleep never resumes, so the watchdog can never reach os._exit. It can only catch a task that stops beating while the loop still turns — it is structurally blind to a freeze of the loop itself. The control surface is registered watched=False (lines 518-525), and there is no external watchdog process for the monitor (only alphamind-safety-core-watchdog exists). The monitor also writes no off-loop liveness signal — it has no FileHeartbeatSink (grep: heartbeat wiring exists only under safety_core/). Net: a loop freeze is unrecoverable without a manual restart, and undiagnosable after a restart (the frame is gone).

This is exactly the structural gap ADR-0004 closed for the safety core: an out-of-process watchdog reading a file heartbeat across the process boundary makes "the watchdog dies with the loop it guards" unrepresentable.

Why it escaped tests

The existing watchdog tests (tests/execution/continuous_monitor/test_supervisor*.py) exercise the in-loop watchdog catching a task that stops beating while the event loop keeps turning — they drive _watchdog_loop with a working sleep/monotonic, so the loop is never frozen. A loop-thread freeze is outside what an in-loop-watchdog test can express: by construction the in-loop watchdog cannot observe its own frozen loop, so there was no behavior to assert and no test failed. The safety core proved the external watchdog restarts on staleness (tests/execution/continuous_monitor/safety_core/test_watchdog.py), but the monitor had no equivalent watchdog and thus no equivalent test. The new loop-freeze simulation test (Verification) is the missing one.

Reading

• docs/design/05-execution-layer/continuous-monitor-runtime.md — ADR-0004 (safety-core isolation + dedicated out-of-process watchdog). The architecture this issue extends to the monitor.
• src/alphamind/execution/continuous_monitor/safety_core/watchdog.py — run_watchdog, HeartbeatProbe / ProcessController Protocols, supervised_watchdog_loop. The generic watchdog to hoist + reuse.
• src/alphamind/execution/continuous_monitor/safety_core/heartbeat.py — FileHeartbeatSink / FileHeartbeatProbe (atomic write, wall-clock epoch). Hoist + reuse.
• src/alphamind/execution/continuous_monitor/safety_core/process_control.py — NssmServiceController (parameterized by service_name). Hoist + reuse.
• src/alphamind/execution/continuous_monitor/safety_core/__main__.py:90-229 — the run / watchdog subcommand split, _run_watchdog_daemon, _heartbeat_path, _log_filename_for_subcommand. The entrypoint shape to mirror in the monitor.
• src/alphamind/execution/continuous_monitor/supervisor.py:228-349 — MonitorSupervisor.run / _watchdog_loop / _check_interval. Where the file-heartbeat beat is injected (Scope B).
• src/alphamind/execution/continuous_monitor/__main__.py:305-321,323-531 — the monitor's _parse_args (add the watchdog subcommand) and _run_daemon (wire the heartbeat sink + faulthandler deadman task).
• src/alphamind/config/models/continuous_monitor.py:70-204 — ContinuousMonitorConfig; watchdog_cadence_multiplier (139-155) is reused for the stall bound. New knob lands here (Scope D).
• .importlinter:563-581 — the safety-core-isolation contract. Confirms monitor → safety_core is permitted today and that the hoist must keep the contract green.
• scripts/install_safety_core_service.ps1 — the two-service (core + watchdog) NSSM install pattern to mirror for the monitor (Scope F).
• ALP-942 — the spun-off WAL-contention trigger; do not implement it here.

Scope

Mirror the safety-core out-of-process watchdog for the monitor, sharing the (generic) primitives rather than duplicating them. The fix converts an unrecoverable silent wedge into a bounded auto-restart regardless of freeze cause, and makes the next freeze self-introspecting.

(A) Hoist the generic supervision primitives to a shared package. Move watchdog.py, heartbeat.py, and process_control.py from continuous_monitor/safety_core/ into a new neutral package alphamind.execution.process_supervision (sibling of continuous_monitor, so it is genuinely shared and not a continuous_monitor internal). Add a small HeartbeatSink Protocol (a single beat() -> None) there for the supervisor seam. Update safety_core/__main__.py to import from the new location; move the corresponding unit tests to tests/execution/process_supervision/. The safety-core-isolation contract still passes (the shared package is not in its forbidden list and imports no monitor internals); add a one-line comment to that contract noting the shared home.

(B) Write an off-hot-path file heartbeat from the monitor supervisor. Inject heartbeat: HeartbeatSink | None = None into MonitorSupervisor.__init__; when present, call heartbeat.beat() at the top of each _watchdog_loop iteration (immediately after await self._sleep(...) returns — proving the loop resumed). Default None preserves all existing behavior and tests. Wire it in _run_daemon to a FileHeartbeatSink at ~/AlphaMind/logs/monitor.heartbeat. Because _check_interval() is ~2.5 s in practice (the 1 s bracket-stop cadence floors it), the file is refreshed far inside the stall bound while the loop turns, and goes stale the moment it freezes.

(C) Add the monitor watchdog entrypoint + subcommand. Add a watchdog subparser to continuous_monitor/__main__.py (python -m alphamind.execution.continuous_monitor watchdog) and a _run_watchdog_daemon that mirrors safety_core._run_watchdog_daemon: build FileHeartbeatProbe at the monitor heartbeat path + NssmServiceController(service_name="alphamind-monitor"), then run_watchdog(probe=..., controller=..., stall_bound_seconds=tick*multiplier, loop=supervised_watchdog_loop(tick)). Configure its logging to its own file via configure_monitor_logging(filename="monitor_watchdog.log") (the Windows WinError 32 per-process rotating-log rule, ALP-868).

(D) Add one config knob for the watchdog cadence. Add monitor_watchdog_tick_seconds: float = Field(default=15.0, gt=0.0) to ContinuousMonitorConfig and config/continuous_monitor.yaml. The watchdog ticks at monitor_watchdog_tick_seconds and its stall bound is monitor_watchdog_tick_seconds * watchdog_cadence_multiplier — exactly mirroring the safety core's cadence * multiplier relationship, so no second multiplier is introduced. TRAP: a new ContinuousMonitorConfig field re-pins the resolved-config snapshot hash — update the pinned digest in tests/config/test_snapshot.py following the entry_window_max_reprices precedent (the documented hash-shift comments around lines 175-263).

(E) faulthandler deadman for 3.14 introspectability. Call faulthandler.enable() at monitor startup and register a supervised "faulthandler deadman" task that, each iteration of supervisor.supervised_loop("faulthandler_deadman", monitor_watchdog_tick_seconds), cancels the prior pending dump and re-arms faulthandler.dump_traceback_later(monitor_watchdog_tick_seconds * watchdog_cadence_multiplier, repeat=False, file=<monitor_faulthandler.log handle>). While the loop turns the timer is always re-armed before firing → no output; if the loop freezes, the re-arm stops and faulthandler's dedicated C-thread timer fires and dumps the frozen main-thread stack to the file — capturing the exact blocking frame on the next wedge without py-spy (which cannot introspect CPython 3.14) or a full minidump. Inject the arm/cancel callables so a fake records calls in tests.

(F) Install the watchdog as an NSSM service + update the runbook. Refactor scripts/install_monitor_service.ps1 to install both alphamind-monitor (run) and alphamind-monitor-watchdog (watchdog), mirroring install_safety_core_service.ps1: same AppExit Default Restart + 60 s throttle, delayed-auto start, per-service stdout/stderr logs, and the same-service-account note (the watchdog must run under the account that can nssm restart alphamind-monitor). Add alphamind-monitor-watchdog to the scripts/RUNBOOK_production.md service table and the relevant sections, mirroring the safety-core watchdog entry.

Acceptance criteria

• alphamind.execution.process_supervision holds watchdog / heartbeat / process_control + a HeartbeatSink Protocol; safety_core/__main__.py imports them from there; uv run lint-imports passes (the safety-core-isolation contract stays green); the moved unit tests pass at tests/execution/process_supervision/.
• MonitorSupervisor calls heartbeat.beat() once per _watchdog_loop iteration when a sink is injected, and not at all when heartbeat is None — covered by a unit test with a fake sink.
• python -m alphamind.execution.continuous_monitor watchdog runs a watchdog that calls ProcessController.restart() when the heartbeat age exceeds monitor_watchdog_tick_seconds * watchdog_cadence_multiplier, and does NOT restart when the heartbeat is fresh or the file is absent (startup grace) — covered by a unit test with a fake controller + injected probe ages.
• A loop-freeze simulation (the loop thread blocked) shows the in-loop _watchdog_loop never reaches os._exit(1) while the file-heartbeat-probe-based external watchdog does fire a restart — the test that encodes the core fix.
• monitor_watchdog_tick_seconds exists on ContinuousMonitorConfig and in config/continuous_monitor.yaml, config loads, and tests/config/test_snapshot.py passes with the updated digest.
• The faulthandler deadman writes a thread dump to monitor_faulthandler.log when its re-arm task stops (frozen-loop case) and writes nothing while it keeps re-arming — covered by a unit test with fake arm/cancel callables.
• scripts/install_monitor_service.ps1 installs both alphamind-monitor and alphamind-monitor-watchdog; scripts/RUNBOOK_production.md lists alphamind-monitor-watchdog in the service table.

Verification

• New/moved unit tests: process_supervision primitives (moved from safety_core), the MonitorSupervisor heartbeat seam, the monitor watchdog daemon wiring, and the faulthandler deadman (fake arm/cancel) — each sociable at the sanctioned boundaries only (no patching of internal collaborators).
• The loop-freeze simulation test is the load-bearing regression guard for the disease (in-loop watchdog cannot fire; external watchdog can).
• Lint chain green: uv run ruff check . && uv run ruff format . && uv run mypy && uv run lint-imports (the hoist makes lint-imports the key gate).
• By inspection: the NSSM two-service install script and the RUNBOOK service-table entry.
• Authoritative gate: the ci workflow (full suite, Windows, -n auto).

Artifacts (prod box, C:\Users\jacks\AlphaMind\)

• diagnostics\monitor_wedge_10408_20260609.dmp — full minidump (34 MB) of the wedged process, captured via comsvcs.dll MiniDump before the recovery restart. Offline analysis (WinDbg + symbols, or a 3.14-capable profiler) confirms the blocking frame for ALP-942; not required to land this issue.
• diagnostics\monitor.log.wedge-20260609 — preserved copy of the wedged monitor.log segment.

One-time prod data cleanup

No bulk cleanup needed. This issue adds infrastructure (a hoisted package, an injected heartbeat sink, a new NSSM service + heartbeat/faulthandler files, one config knob) — it introduces no new DB table and mutates no existing rows. The outage's data divergence (un-ingested GS/MRVL/SLB fills; naked GS/MRVL/TSM during RTH) is a consequence of the freeze, not of this fix, and self-heals through mechanisms already in motion: the fill-backfill backstop (ALP-763) re-captures the missed fills on monitor restart, and re-protection is invocation-driven (the next market_hours_rolling run re-protects). Operationally confirm those completed (positions re-protected at the broker, fills present in fill_records) — but that is existing machinery, not a step this issue introduces.

Related

ALP-942 (the spun-off WAL write-lock-contention trigger of this freeze — a distinct, not-yet-confirmed defect class; this issue makes a freeze recoverable regardless of cause, ALP-942 reduces this freeze class's probability). ALP-841 (the original monitor sync-loop-block wedge — this is a recurrence via a different freeze cause; its broker-side get_orders mitigation is in place but the architectural gap remained). ALP-857 / ADR-0004 (the safety core's out-of-process watchdog — the pattern mirrored here). ALP-937 / ALP-938 (the partial-close re-protection chain defeated during the outage).

ALP-945 Monitor watchdog cannot restart the monitor in production — nssm restart refused by the CC service dependency

Summary

The monitor's event loop froze at ~13:00:18 MT on 2026-06-10 (cause: ALP-946, a bare-sync subscribe_quotes call on the loop). Every layer of the ALP-941 supervision design then worked — the heartbeat went stale, the faulthandler deadman dumped the frozen stack, and the out-of-process alphamind-monitor-watchdog detected staleness and issued restarts — but every restart was refused by the Windows SCM for 38 minutes, until the operator ran the manual §8.9 recovery at 13:42. The watchdog is structurally unable to restart the monitor whenever AlphaMindCommandCenter is running, which in production is always. The ~150 s wedge bound ALP-941 was built to guarantee was in practice unbounded.

The original report's root cause (the CC service dependency blocks the stop) is confirmed. Refinement settled the fix direction with the operator: remove both of CC's SCM dependency edges (on alphamind-scheduler and alphamind-monitor) — the edge's documented rationale is refuted by CC's as-built behavior (see Evidence) — plus a diagnosability hardening at the watchdog's restart seam.

Evidence (prod logs + repo — 2026-06-10 incident)

All log timestamps are Mountain Time.

• monitor_watchdog.log (prod): first staleness detection at 13:02:50 — heartbeat stale (152s > bound 150s), placing the last beat at ≈ 13:00:18, matching ALP-946's freeze time. Sixteen restart attempts from 13:02:50 to 13:40:22, spaced ~150 s apart — exactly the run_watchdog debounce (src/alphamind/execution/process_supervision/watchdog.py:91) — each raising CalledProcessError: Command '['nssm', 'restart', 'alphamind-monitor']' returned non-zero exit status 1 from process_control.py:54.
• monitor_watchdog.err.log (prod): sixteen lines of nssm's stderr — alphamind-monitor: STOP: A stop control has been sent to a service that other running services are dependent on. The refusal reason reached this file only via NSSM's redirection of the watchdog process's own stderr; the watchdog's log shows just "exit status 1" (the diagnosability gap Scope B closes).
• Operator intervention: the watchdog service was stopped at 13:41:41 (KeyboardInterrupt traceback in monitor_watchdog.log) and restarted at 13:42:10; the monitor came back via the manual §8.9 stop-CC dance. Total wedge ≈ 42 min against the 150 s design bound.
• The dependency edge: scripts/install_command_center_service.ps1:148 — nssm set $ServiceName DependOnService alphamind-scheduler alphamind-monitor — is the only DependOnService declaration in the repo. The Windows SCM refuses a stop control on a service with running dependents, and nssm restart is stop + start (RUNBOOK §7), so with CC running the stop is refused before anything happens.
• The edge's rationale is refuted by the as-built code it shipped with. The script (lines 13–17) justifies the dependency with "the SSE consumer tasks fail-fast when their upstreams are unreachable, so a too-eager start would cycle through restarts". The actual consumer loop (src/alphamind/command_center/events/multiplexer.py:240–298) reconnects forever with capped jittered backoff (1 s → 30 s cap), never exits, and has no reconnect budget; the control proxy hits upstreams per-request. Both the edge and that loop landed in the same commit (269ff0a, ALP-128, 2026-05-26) — the rationale was never true of the shipped code.
• Timing corroboration: the watchdog shipped 2026-06-09 22:23 MT (ALP-941, commit 33b77e0, PR fix(execution): out-of-process monitor watchdog — frozen event loop now auto-restarts and self-dumps its blocking frame (ALP-941) #346) into a topology that had refused its restart since 2026-05-26. The 2026-06-10 incident was its first production firing — refused on first contact.
• Safety-core check (the report asked to verify): nothing declares a dependency on alphamind-safety-core, so the safety-core watchdog is unaffected today. The gap is monitor-specific in practice; Scope C makes the invariant explicit so it stays that way.

Root cause (confirmed)

1. AlphaMindCommandCenter declares an SCM dependency on alphamind-scheduler and alphamind-monitor (scripts/install_command_center_service.ps1:148).
2. The Windows SCM refuses to stop a service that a running service depends on; nssm restart is stop + start, so with CC running — always, in production — nssm restart alphamind-monitor exits 1 without stopping anything.
3. The watchdog's only restart mechanism is exactly that command (NssmServiceController.restart, src/alphamind/execution/process_supervision/process_control.py:51–54, bound to alphamind-monitor at continuous_monitor/__main__.py:149 and :804), and restart() deliberately swallows failures so a transient nssm error doesn't kill the watchdog (process_control.py:55–58). Correct per-tick — but against a structural refusal it degenerates into an infinite refused-restart loop at the debounce cadence.
4. The disease is the topology: production declares an SCM dependency edge pointing at a watchdog-supervised service, on a rationale the as-built CC refutes. The refused restarts and the 42-minute wedge are symptoms.

Why it escaped tests

The watchdog's restart-on-staleness logic is tested against a fake ProcessController (the designed seam, watchdog.py:35–44); the real seam is an OS shell-out whose failure semantics live in the Windows SCM and are unreachable from pytest. The topology itself is encoded in an install script no test executes — the dependency graph exists only on the prod box. And the two pieces never coexisted before production: the edge shipped 2026-05-26 (ALP-128), the watchdog 2026-06-09 (ALP-941); its first real firing on 2026-06-10 was this path's first execution against the real SCM. No test class can cover install-script topology against SCM stop semantics; the structural fix removes the conflicting topology and documents the invariant where topology is defined (the install scripts).

Reading

1. docs/adr/0004-monitor-fail-safe-isolate-safety-core.md — process topology is an ops/simplicity question once the broker floor holds; the watchdog's design intent.
2. scripts/install_command_center_service.ps1 (lines 13–17, 36–38, 51–56, 117, 144–148) — every dependency artifact Scope A deletes.
3. src/alphamind/execution/process_supervision/process_control.py — the restart seam Scope B hardens.
4. src/alphamind/execution/process_supervision/watchdog.py:84–102 — the debounce loop that made the refusal repeat every ~150 s.
5. src/alphamind/execution/continuous_monitor/__main__.py:147–149, 773–808 — watchdog wiring; the service name the controller restarts.
6. src/alphamind/command_center/events/multiplexer.py:240–298 — the reconnect-forever consumer loop that makes edge removal safe.
7. scripts/RUNBOOK_production.md §1.4, §1.6, §7 (second restart gotcha, lines 1096–1100), §8.9 (manual fallback, lines 1243–1248) — the passages Scope D rewrites.
8. scripts/RUNBOOK_command_center.md (prerequisite 6, lines 59–62; service-config bullet, line 123) — the CC-side passages Scope D rewrites.
9. tests/execution/process_supervision/test_process_control.py — the existing test surface Scope B extends.

Scope — remove CC's SCM dependency edges; surface nssm stderr at the restart seam

(A) Remove both dependency edges from the CC install script. In scripts/install_command_center_service.ps1: delete the DependOnService call (line 148) with its comment block (144–147), the $PipelineDependency / $MonitorDependency variables (51–56), the .DESCRIPTION dependency rationale (13–17), the "pipeline + monitor must already be installed" prerequisite (36–38), and the "Depends on" echo (117). Replace with a short comment stating the as-built contract: the command center starts independently of its upstreams — its SSE consumers reconnect with capped backoff until the scheduler/monitor come up (multiplexer.py), and the control proxy hits upstreams per-request.

(B) Log nssm's stderr at the restart seam. In src/alphamind/execution/process_supervision/process_control.py: _checked_run calls subprocess.run(cmd, check=True, capture_output=True, text=True) so the raised CalledProcessError carries the child's stderr; restart() catches subprocess.CalledProcessError distinctly and includes exc.stderr in the failure log line, keeping the broad except Exception fallback for non-subprocess errors. The log-and-retry-next-tick swallow behavior is unchanged. The seam is shared by both watchdogs (monitor + safety core); no wiring changes.

(C) Document the supervision-topology invariant where topology is defined. Add to scripts/install_monitor_service.ps1 and scripts/install_safety_core_service.ps1 a comment: no service may declare an SCM dependency (DependOnService) on the supervised service — the out-of-process watchdog's nssm restart would be refused by the SCM while the dependent runs (ALP-945, the 2026-06-10 incident).

(D) Runbook updates (Living-document rule). In scripts/RUNBOOK_production.md: §7 — delete the "can be silently refused" gotcha (lines 1096–1100), state that a plain nssm restart <service> works for any single service with the others running, and add the topology-invariant one-liner; §8.9 — the manual fallback (lines 1243–1248) becomes plain nssm stop + nssm start alphamind-monitor with no CC choreography; §1.4/§1.6 — keep the same stop/start command order but frame it as operational convention (quiet migration window; watchdog before its target), since "the command center depends on scheduler + monitor" is no longer true. In scripts/RUNBOOK_command_center.md: update prerequisite 6 (claims the SCM refuses CC start when dependencies are missing) and the "Dependency on alphamind-scheduler + alphamind-monitor" service-config bullet to the no-dependency topology; the "stop all / restart all" orderings stay as convention.

(E) One-time prod deployment step (operator-executed, prod box, elevated PowerShell). The live SCM config was written at install time, so pulling the updated script changes nothing on the installed service. Run sc.exe config AlphaMindCommandCenter depend= / and confirm sc.exe qc AlphaMindCommandCenter shows an empty DEPENDENCIES list. Effective immediately; no service restart required. Then prove the fix end-to-end: with CC running, nssm restart alphamind-monitor exits 0 and the monitor comes back healthy (fresh PID/StartTime, 8766 heartbeat within ~18 s). Include this step in the PR description's deploy notes — the fix is not live until it runs.

Acceptance criteria

• scripts/install_command_center_service.ps1 contains no dependency configuration or prose: a case-insensitive grep for "depend" over the file returns nothing.
• NssmServiceController.restart failure logging carries the shell-out's stderr: a unit test injects a run callable raising CalledProcessError with stderr set (e.g. the SCM dependency-refusal line) and asserts the emitted log record contains that stderr text, and that the exception is still swallowed.
• _checked_run attaches the child's stderr to the raised CalledProcessError: exercised with a real subprocess that exits nonzero after writing to stderr (no patching of subprocess).
• scripts/install_monitor_service.ps1 and scripts/install_safety_core_service.ps1 each carry the topology-invariant comment (no DependOnService may target the supervised service).
• scripts/RUNBOOK_production.md §7 documents plain single-service restart with no stop-CC choreography and carries the invariant; the stop-CC dance appears nowhere in §7 or §8.9.
• scripts/RUNBOOK_production.md §1.4/§1.6 frame stop/start ordering as operational convention; no passage claims the SCM enforces CC's dependencies.
• scripts/RUNBOOK_command_center.md reflects the no-dependency topology in its prerequisite list and service-config summary.
• Deploy-time, operator-verified on the prod box: sc.exe qc AlphaMindCommandCenter shows an empty DEPENDENCIES list, and with CC running nssm restart alphamind-monitor exits 0 with the monitor healthy afterward (fresh StartTime + 8766 heartbeat).

Verification

• uv run pytest tests/execution/process_supervision/ -n auto — existing heartbeat/watchdog/process-control tests plus the two new stderr criteria.
• uv run ruff check . && uv run ruff format . && uv run mypy && uv run lint-imports.
• By inspection: the grep criteria over the three install scripts and both runbooks.
• CI on Windows is the authoritative full-suite gate. The PR touches src/, so CI runs (scripts/** and **.md alone are paths-ignored).
• Post-deploy prod proof (operator): Scope E's restart-with-CC-running check — the exact command and refusal class from the incident. Optional off-hours end-to-end drill: Suspend-Process the monitor's python process; the heartbeat goes stale; within ~150 s the watchdog fires and the restart now succeeds (NSSM escalates the unanswered stop control to termination, §8.3); confirm fresh StartTime + 8766 heartbeat with no manual action.

One-time prod data cleanup

No bulk cleanup needed. No production data was corrupted by this defect — the wedge's uncaptured fills were recaptured by the startup replay + activities backfill after the 13:42 manual recovery (§8.9; ALP-946's domain). The only prod-box action is Scope E's SCM config change, which is the fix's deployment, not data repair.

Separate observation — do NOT bundle

The watchdog has no escalation channel: sixteen consecutive failed restarts over 38 minutes produced only file-log lines — no alert fired, and nothing watches the watchdog's outcomes (the alert engine consumes the CC event stream; config/alerts.yaml has no rule on watchdog restart failure or heartbeat staleness, and the watchdog emits no events). The incident ended only because the operator happened to be watching. That is an alerting-coverage defect with its own root cause and fix surface (alert engine / channels), untouched by this issue's topology fix. File separately if wanted.

Related

• ALP-941 — built the out-of-process watchdog this defect defeats; merged 2026-06-09 (PR fix(execution): out-of-process monitor watchdog — frozen event loop now auto-restarts and self-dumps its blocking frame (ALP-941) #346), one day before its first firing was refused.
• ALP-946 — the bare-sync subscribe_quotes wedge that triggered the 2026-06-10 incident; separate defect (loop-freeze cause vs restart failure). With ALP-945 fixed, an ALP-946-class wedge is bounded at ~150 s plus restart time.
• Commit 269ff0a (ALP-128, 2026-05-26) — introduced both the dependency edge and the reconnect-forever consumer loop that refutes the edge's rationale.

Review in Linear