feat(substrate): v2 Phase 3 PR-A — watchdog from proven parts (WS-C, checks 11-12)

[SamPlvs]

Summary

Phase 3 / WS-C, part 1 of 2: the watchdog — an external checker in the LifecycleWrapper poll loop (not an LLM monitor, not a cron) built from the parts the 2026-08-12 review rated proven. Plan oracle checks 11–12. Part 2 (the fresh-context driver, PR-B) follows on the same branch line.

• Heartbeats — stdlib-only writer (zo._hook_heartbeat, ~60 ms) on a new PostToolUse "*" hook (+ Stop→ready, PreCompact→compacting, SubagentStop/SessionEnd→shutdown), keyed agent_id or lead-<session_id>, written under <memory_root>/heartbeats/, sealed against agent Write/Edit and gitignored (platform root + delivery .zo/ templates).
• Never-block taxonomy first — ported from oh-my-claudecode (MIT): context-limit (#213), rate-limit (#777), auth (#1308), user-abort (incl. ⎿ Interrupted by user); ZO additions: awaiting_input (a permission dialog is never sent Enter), compacting; no bare 429/overloaded; tiered rate-limit patterns.
• Stall predicate by evidence, unknown ≠ dead — heartbeat tick deltas (pre-existing files baselined), normalized pane/stdout digest (spinners/counters stripped), progress-path mtimes (ledger, comms dir, .zo/experiments), process-tree CPU time (a silent 40-min training call is not a stall), PID + process-start-time identity (positive proof only; EPERM = alive).
• Bounded nudges — tmux only, named paste buffer, dwell 30 s, budget 3 (persisted), only when the pane shows the idle prompt; escalation once per stall: tmux logs error_type=stall severity=blocking + STALLED at exit (never kills a human-facing pane); headless kills by default (kill_headless_on_escalate).
• Rate-limit wait-and-resume — paused state evaluated per poll (no blocking sleep), reset time parsed in local tz, resume verified by real progress (static banner gets a bounded resume nudge), timeout excludes paused time. The headless retry-with-backoff loop is removed; exit while limited → RATE_LIMITED + resume_at for the PR-B driver.
• Config — ProjectConfig.watchdog: WatchdogConfig, zo build/continue --no-watchdog, ZO_WATCHDOG=0, ZO_WATCHDOG_STALL_SEC; CLI threads watchdog/memory_root/ZO_SESSION_ID.
• Docs — specs/watchdog.md rewritten to implemented reality (the RFC's cron-scheduled orchestrator tick and respawn/reroute ladder are superseded by the plan; respawn moves to PR-B); mdx status, COMMANDS/build.mdx flag; README badge.
• Recon + contract persisted under memory/zo-platform/research/2026-08-17-phase3-recon/ (integration map with file:line, PR-A build contract, raw mapper output).

Structural finding (shapes PR-B)

Orchestrator.advance_phase() / mark_subtask_complete() have zero runtime callers — the automated gate, _auto_iterate_if_needed, the WS-B mark_phase_passed flip and the only gate-nonce mint are unreachable in production today (zo build = one lead session per phase, then end_session()). PR-B's driver is the first runtime caller. Recorded as PRIORS PR-047; decisions in DECISION_LOG 2026-08-17T09:00.

Verification

• pytest: 929 → 1131 passed / 7 skipped; ruff check src/ clean; scripts/validate-docs.sh 0 failures.
• Seeded-failure tests (both loops): check 11 — a 10-minute stall (stall_threshold_sec=600) is nudged ≤3× then escalated within one poll; a rate-limited session is never nudged; a permission dialog is never nudged. Check 12 — a rate-limit pause auto-resumes on reset with verified progress; timeout accounting excludes the pause.
• Wiring tests start from the entry points: settings.json heartbeat entry asserted; _watchdog_tick asserted from both loops incl. the suspected-dead continue path; zo build passes watchdog=/memory_root=/zo_session_id=; heartbeats sealed (seeded Write denied); .gitignore via git check-ignore.
• Built contract-first with 4 concurrent builders → integrator → 3 adversarial verifier lenses (19 findings, 2 high: reset times parsed in UTC; static banner could never resume) → fixer (11 applied with regression tests, 4 rejected with reasons).
• Live: the heartbeat hook fired in the build session itself for the lead and for workflow subagents (agent_type=workflow-subagent, tick_count 44 → shutdown) — PostToolUse carries agent identity. Also corrected: PostToolUseFailure does fire on nonzero-exit Bash (session-040 caveat withdrawn).

Not in this PR (recorded)

Driver + fresh-context loop + evaluate_loop_state(ledger=) + restore cutover + zo phase set (PR-B); check 13 on the Linux box; _watchdog-ticks.jsonl rotation; wrapper.py 1404 lines (split _wrapper_tmux.py); CPU-evidence idle threshold to verify on a real tmux session.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying zero-operators with    Cloudflare Pages

Latest commit:	52ae20c
Status:	✅  Deploy successful!
Preview URL:	https://98b4b1e1.zero-operators.pages.dev
Branch Preview URL:	https://claude-v2-phase3-substrate.zero-operators.pages.dev

View logs