From 0503985aa1d6ad8144e687bee4e00979abbd3fc8 Mon Sep 17 00:00:00 2001 From: SamT <35964759+SamPlvs@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:38:04 +0100 Subject: [PATCH 1/3] =?UTF-8?q?feat(substrate):=20v2=20Phase=203=20PR-A=20?= =?UTF-8?q?=E2=80=94=20watchdog=20from=20proven=20parts=20(WS-C,=20oracle?= =?UTF-8?q?=20checks=2011-12)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit External checker in the LifecycleWrapper poll loop (not an LLM monitor, not a cron): hook-written per-agent heartbeats, never-block taxonomy applied before any intervention, bounded nudges through a pane-ready guard, rate-limit wait-and-resume replacing the headless retry loop, PID + process-start-time identity that never treats unknown as dead. - src/zo/watchdog.py (+ _watchdog_models/_watchdog_text/_proc): pure policy evaluate(), three-state freshness, taxonomy ported from oh-my-claudecode (MIT) with ZO adjustments (no bare 429/overloaded, awaiting_input added, bare "interrupt" excluded), tiered rate-limit patterns, local-tz reset parsing, positive-proof-only process death, WatchdogConfig/WatchdogState - src/zo/_hook_heartbeat.py + hookkit "heartbeat" handler on a new PostToolUse "*" hook (stdlib-only, ~60 ms; Stop→ready, PreCompact→compacting, SubagentStop/SessionEnd→shutdown); heartbeats sealed and gitignored - src/zo/_wrapper_watchdog.py WatchdogRunner ticked from BOTH loops before the liveness reads; evidence = heartbeat deltas + normalized text digest + progress-path mtimes + process-tree CPU time; tmux nudge via named paste buffer; rate-limit = paused state per poll, verified resume; timeout excludes paused time; escalation: tmux logs+STALLED, headless kills (config) - AgentStatus.PAUSED_RATE_LIMIT/STALLED; LeadProcess identity/resume_at fields; headless retry-with-backoff removed (exit while limited → RATE_LIMITED+resume_at) - ProjectConfig.watchdog, zo build/continue --no-watchdog, ZO_WATCHDOG=0, ZO_SESSION_ID threaded to hooks; delivery .zo/ gitignore templates updated - specs/watchdog.md rewritten to implemented reality (RFC cron tick superseded); docs cascade; README badge 1053 - recon artefacts + build contract under memory/zo-platform/research/2026-08-17-phase3-recon/ Verification: 929 → 1131 passed / 7 skipped, ruff clean, validate-docs 0 failures. Seeded tests for checks 11 (10-min stall escalated within one poll; rate-limited session never nudged) and 12 (pause auto-resumes on reset with verified progress) on both loops. Heartbeat hook observed firing live for the lead and workflow subagents in the build session. Memory: STATE session 041, DECISION_LOG (Phase 3 decisions + PR-A + correction of the PostToolUseFailure caveat), PRIORS PR-047/PR-048, session-041 summary. Co-Authored-By: Claude Fable 5 --- .claude/hooks/zo-hookkit.sh | 3 + .claude/settings.json | 10 + .gitignore | 6 + README.md | 4 +- docs/COMMANDS.md | 9 +- docs/cli/build.mdx | 1 + docs/reference/v2-rearchitecture.mdx | 11 + memory/zo-platform/DECISION_LOG.md | 28 + memory/zo-platform/PRIORS.md | 35 + memory/zo-platform/STATE.md | 4 +- .../integration-map.md | 253 ++ .../pr-a-build-contract.md | 303 +++ .../2026-08-17-phase3-recon/raw-mappers.json | 2063 +++++++++++++++++ .../sessions/session-041-2026-08-17.md | 99 + specs/watchdog.md | 139 +- src/zo/_hook_heartbeat.py | 145 ++ src/zo/_proc.py | 237 ++ src/zo/_watchdog_models.py | 178 ++ src/zo/_watchdog_text.py | 300 +++ src/zo/_wrapper_models.py | 18 +- src/zo/_wrapper_watchdog.py | 325 +++ src/zo/cli.py | 111 +- src/zo/hookkit.py | 88 +- src/zo/project_config.py | 13 +- src/zo/scaffold.py | 5 + src/zo/watchdog.py | 451 ++++ src/zo/wrapper.py | 567 ++++- tests/integration/test_hooks_shim.py | 149 ++ tests/unit/test_cli.py | 317 +++ tests/unit/test_hookkit.py | 293 ++- tests/unit/test_project_config.py | 133 +- tests/unit/test_scaffold.py | 5 + tests/unit/test_watchdog.py | 406 ++++ tests/unit/test_watchdog_policy.py | 383 +++ tests/unit/test_wrapper.py | 996 +++++++- 35 files changed, 7874 insertions(+), 214 deletions(-) create mode 100644 memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md create mode 100644 memory/zo-platform/research/2026-08-17-phase3-recon/pr-a-build-contract.md create mode 100644 memory/zo-platform/research/2026-08-17-phase3-recon/raw-mappers.json create mode 100644 memory/zo-platform/sessions/session-041-2026-08-17.md create mode 100644 src/zo/_hook_heartbeat.py create mode 100644 src/zo/_proc.py create mode 100644 src/zo/_watchdog_models.py create mode 100644 src/zo/_watchdog_text.py create mode 100644 src/zo/_wrapper_watchdog.py create mode 100644 src/zo/watchdog.py create mode 100644 tests/unit/test_watchdog.py create mode 100644 tests/unit/test_watchdog_policy.py diff --git a/.claude/hooks/zo-hookkit.sh b/.claude/hooks/zo-hookkit.sh index 4b22dba..0eb1411 100644 --- a/.claude/hooks/zo-hookkit.sh +++ b/.claude/hooks/zo-hookkit.sh @@ -27,5 +27,8 @@ PY="python3" # Pre-set ZO_REPO_ROOT wins (lets tests point the handlers at a sandbox). export ZO_REPO_ROOT="${ZO_REPO_ROOT:-$REPO_ROOT}" +# Wall-clock stamp of the hook event (WS-C heartbeat; cheap, informational — +# the heartbeat handler stamps its own UTC time and does not require this). +export ZO_HOOK_EVENT_TS="$(date -u +%s)" PYTHONPATH="$REPO_ROOT/src${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m zo.hookkit "$EVENT" || exit 0 exit 0 diff --git a/.claude/settings.json b/.claude/settings.json index f244a9e..79be862 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -81,6 +81,16 @@ "timeout": 5 } ] + }, + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "bash .claude/hooks/zo-hookkit.sh heartbeat 2>/dev/null || exit 0", + "timeout": 5 + } + ] } ], "Stop": [ diff --git a/.gitignore b/.gitignore index ca68880..75101ea 100644 --- a/.gitignore +++ b/.gitignore @@ -32,6 +32,12 @@ logs/ # Per-project memory (keep ZO platform memory, ignore everything else) memory/* !memory/zo-platform/ +# Runtime control-plane state under the platform memory root is never +# tracked (WS-B ledger/contracts, WS-C heartbeats). These MUST come after +# the !memory/zo-platform/ re-include above. +memory/zo-platform/heartbeats/ +memory/zo-platform/plan-ledger.json +memory/zo-platform/contracts.json # Design source files contain project-specific examples (client-identifying) docs/source-design/ diff --git a/README.md b/README.md index 45b5f20..5c7a301 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@
[![Status](https://img.shields.io/badge/status-validated-D87A57?style=flat-square&labelColor=12110F)](#status) -[![Tests](https://img.shields.io/badge/tests-854_passing-D87A57?style=flat-square&labelColor=12110F)](#status) +[![Tests](https://img.shields.io/badge/tests-1053_passing-D87A57?style=flat-square&labelColor=12110F)](#status) [![Agents](https://img.shields.io/badge/agents-21_defined-D87A57?style=flat-square&labelColor=12110F)](#agent-teams) [![Docs](https://img.shields.io/badge/docs-zerooperators.com-D87A57?style=flat-square&labelColor=12110F)](https://docs.zerooperators.com) @@ -526,7 +526,7 @@ delivery-repo/ | 1.0.2 | Platform-aware Docker scaffold + reference-project end-to-end demos | Done | | 1.0.2-post | `--low-token` cost-saving preset (two-tier model routing, per-phase trims) + `ZOTrainingCallback` hard gate enforcement | Done | -780 platform tests. ruff clean (`src/zo/`). 21 agents. 24 slash commands. Measured benchmarks tracked in [docs/reference/cost-benchmark.mdx](docs/reference/cost-benchmark.mdx). +1053 platform tests. ruff clean (`src/zo/`). 21 agents. 24 slash commands. Measured benchmarks tracked in [docs/reference/cost-benchmark.mdx](docs/reference/cost-benchmark.mdx). --- diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index 4997ca4..0763d3b 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -22,7 +22,7 @@ Launch an agent team to execute a plan. Parses the plan, shows a phase review, a zo build plans/project.md [--gate-mode supervised|auto|full-auto] [--no-tmux] [--bypass-permissions] [--low-token] [--lead-model opus|sonnet|haiku] [--max-iterations N] - [--no-headlines] + [--no-headlines] [--no-watchdog] ``` **Cost-saving options:** @@ -34,6 +34,9 @@ zo build plans/project.md [--gate-mode supervised|auto|full-auto] [--no-tmux] **Permission prompts:** - `--bypass-permissions`: auto-approve **every** Claude Code tool-call prompt (Bash, Edit, Write, Read, ...) for the run. Independent of `--gate-mode` (which gates ZO's phases, not individual tool calls); implied by `--gate-mode full-auto`. Off by default. Works in tmux (temporary `.claude/settings.local.json` overlay, auto-restored on exit) and headless (`--dangerously-skip-permissions`) modes. See `docs/cli/build.mdx`. +**Anti-stall watchdog:** +- `--no-watchdog`: disable the anti-stall watchdog for this run (no stall detection, nudges, or rate-limit pause). Equivalent to `ZO_WATCHDOG=0`. On by default; policy comes from the `watchdog:` block in `.zo/config.yaml` (see `specs/watchdog.md`). + ### zo continue Resume a paused project. Shorthand for `zo build` with an existing plan -- finds the plan by project name and picks up from the current phase. @@ -42,13 +45,13 @@ Resume a paused project. Shorthand for `zo build` with an existing plan -- finds zo continue [project-name] [--repo PATH] [--gate-mode supervised|auto|full-auto] [--bypass-permissions] [--low-token] [--lead-model opus|sonnet|haiku] [--max-iterations N] - [--no-headlines] + [--no-headlines] [--no-watchdog] ``` **Options:** - `project-name`: optional if cwd contains `.zo/config.yaml` (auto-detected) - `--repo PATH`: path to delivery repo (overrides target file lookup) -- `--low-token`, `--lead-model`, `--max-iterations`, `--no-headlines`, `--bypass-permissions`: same semantics as `zo build` +- `--low-token`, `--lead-model`, `--max-iterations`, `--no-headlines`, `--bypass-permissions`, `--no-watchdog`: same semantics as `zo build` ### zo draft diff --git a/docs/cli/build.mdx b/docs/cli/build.mdx index 272fb49..7641727 100644 --- a/docs/cli/build.mdx +++ b/docs/cli/build.mdx @@ -137,6 +137,7 @@ While `zo build` is running, you have several windows into what the team is doin | `--max-iterations N` | `10` (or `2` if `--low-token`) | Hard cap on Phase-4 experiment iterations. Wins over plan and preset. | | `--no-headlines` | (off) | Skip the end-of-session Haiku bullet summary (~1 Haiku call per run, ~$0.0002). | | `--bypass-permissions` | (off) | Auto-approve Claude Code tool-call prompts. Implied by `--gate-mode full-auto`. See [Permission prompts](#permission-prompts-bypass-permissions). | +| `--no-watchdog` | (off) | Disable the anti-stall watchdog for this run (no stall detection, nudges, or rate-limit pause). Equivalent to `ZO_WATCHDOG=0`. Policy otherwise comes from the `watchdog:` block in `.zo/config.yaml`. | ## Examples diff --git a/docs/reference/v2-rearchitecture.mdx b/docs/reference/v2-rearchitecture.mdx index a73426f..2bdef59 100644 --- a/docs/reference/v2-rearchitecture.mdx +++ b/docs/reference/v2-rearchitecture.mdx @@ -65,6 +65,17 @@ architecture — features from different repos interlock into single mechanisms. | 2 | **Watchdog from proven parts** — heartbeat-file liveness, a taxonomy of stops that must never be fought (context-limit, rate-limit, auth, user abort), bounded nudge budgets, and rate-limit wait-and-resume for overnight runs | oh-my-claudecode + ruflo | P0 | | 6 | **Fresh-context-per-subtask execution loop** — a new agent per iteration re-derives state from the ledger, experiment lineage, and a curated priors digest; git commits are the checkpoints; context rot and compounding hallucination are eliminated structurally | ralph + oh-my-claudecode | P1 | +**Status (Phase 3, in progress):** the watchdog has shipped — an external +checker in the `LifecycleWrapper` poll loop (not an LLM monitor, not a cron) +that reads hook-written heartbeats, classifies never-fight stops before any +nudge, nudges the lead pane at most three times through a pane-ready guard, +pauses on the usage-limit banner until the parsed reset time and verifies the +resume by real progress, then escalates. It is on by default (`watchdog:` block +in `.zo/config.yaml`, `zo build --no-watchdog` or `ZO_WATCHDOG=0` to disable); +design and integration points live in `specs/watchdog.md`. The fresh-context +loop (feature 6) is next and consumes the watchdog's `STALLED` / +`RATE_LIMITED` outcomes. + ### Workstream D — Self-learning & platform oracle *The self-evolution loop gets the same rigor ZO applies to ML work.* diff --git a/memory/zo-platform/DECISION_LOG.md b/memory/zo-platform/DECISION_LOG.md index 84a4964..e43f295 100644 --- a/memory/zo-platform/DECISION_LOG.md +++ b/memory/zo-platform/DECISION_LOG.md @@ -1277,3 +1277,31 @@ The `--no-headlines` flag is preserved (not removed) for backwards compatibility **Scope deferrals (deliberate, to Phase 3):** `evaluate_loop_state` keeps its ExperimentRegistry input (already oracle-derived; signature change touches ~20 test call sites and belongs to the fresh-context loop rework); session-state restore still reads STATE.md (cutover rides the fresh-context substrate to avoid regressing the PR-036 GATED-precedence resume fix). **Outcome:** 908 → 929 passed / 7 skipped (+21: 13 ledger, 5 stories/lint, 2 nonce-flip, 1 status), ruff clean, validate-docs green. Oracle checks 8-10 have passing seeded tests: status renders from the ledger; builder ledger-write denied while the oracle flip lands; vague story rejected. Cascade: specs/plan.md (§10 Stories + lint), specs/workflow.md (state-tracking line), docs/COMMANDS.md (status control plane). Branch `claude/v2-phase2-control-plane`. + +## Decision: 2026-08-17T09:00:00Z +**Type:** SCOPE + ARCHITECTURE +**Title:** v2 Phase 3 sequencing and four build-shaping decisions (Sam), after recon found the oracle-owned gate path has no runtime caller + +**Decision:** Phase 2 (PR #108) merged to main at session start; Phase 3 (WS-C) built on `claude/v2-phase3-substrate` off `1faf53a`. A read-only recon swarm (7 mappers + synthesis, `memory/zo-platform/research/2026-08-17-phase3-recon/`) established, with `file:line` evidence, that `Orchestrator.advance_phase()` and `mark_subtask_complete()` have zero runtime callers: `zo build` launches one lead session per phase and calls `end_session()`; the automated gate, `_auto_iterate_if_needed`, the WS-B `mark_phase_passed` flip and the ONLY gate-nonce mint (`orchestrator.py:788`) are unreachable in production. Phases have only ever advanced through hand-edited STATE.md (the PR-036/037 prod-001 incident). Sam decided: (1) **the driver evaluates gates for ALL phases** — after any session exit the Python side calls `advance_phase()` mechanically (COMPLETED → next phase; GATED → mint nonce, stop; ITERATE → relaunch fresh); only Phase 4 gets fresh headless spawns per iteration until check 13 says extend; (2) **two PRs**: A = watchdog (checks 11–12, buildable here), B = driver + fresh-context loop + the two Phase-2 deferrals; (3) **restore cutover: ledger wins over STATE.md** with a loud warning, plus a sanctioned `zo phase set ` override logged to DECISION_LOG (replaces the hand-edit lever); (4) **tmux nudges default ON** behind a pane-ready / no-permission-dialog guard. + +**Rationale:** The fresh-context loop's driver is not a Phase-4 cost feature — it is the missing runtime caller that makes gates, nonces, ledger flips and `zo gates approve` real (PR-009's own rule, "built ≠ wired", applied to `advance_phase`). Two PRs de-risk the substrate change and let the watchdog merge first. Ledger-wins is the plan's dual-plane doctrine (control plane decides, STATE.md is the projection). Nudges on: the 38-hour stall was an idle lead; a nudge is the resume mechanism after a rate-limit reset too — the guard (idle prompt visible, no `awaiting_input` dialog) prevents Enter from approving a permission prompt. + +**Alternatives considered:** Phase-4-only driver (rejected: leaves five phases on self-asserted completion); one PR (rejected: substrate risk); STATE.md wins / refuse-to-start (rejected: STATE.md is hand-editable and clobbered by `end_session`); nudges log-only in tmux (rejected: no auto-resume after rate-limit reset). + +**Outcome:** PR-A shipped this session (next entry); PR-B queued with the exact seams in the integration map §2–3. + +## Decision: 2026-08-17T12:30:00Z +**Type:** FEATURE + ARCHITECTURE +**Title:** v2 Phase 3 PR-A — watchdog from proven parts (heartbeats + never-block taxonomy + bounded nudges + rate-limit wait-and-resume + PID/start-time identity) + +**Decision:** Implemented WS-C's watchdog per plan Phase 3 and the build contract (`memory/zo-platform/research/2026-08-17-phase3-recon/pr-a-build-contract.md`): pure policy module `zo.watchdog` (+ `_watchdog_models/_watchdog_text/_proc`), stdlib-only heartbeat writer `zo._hook_heartbeat` on a `PostToolUse *` hook (+ Stop/PreCompact/SubagentStop/SessionEnd stamps), `WatchdogRunner` ticked from BOTH wrapper loops before the liveness reads, config in `ProjectConfig.watchdog` + `--no-watchdog` + `ZO_WATCHDOG=0`. `specs/watchdog.md` rewritten: the RFC's cron-scheduled orchestrator-owned tick and respawn/reroute ladder are superseded by the plan's external checker in the wrapper poll loop (a non-LLM process cannot itself stall); respawn moves to PR-B's driver. + +**Design choices worth the record:** (a) three-state freshness — `unknown` (no/unreadable heartbeat, unknown pid identity, EPERM) is never a stall verdict; (b) taxonomy runs before any nudge in both loops, ported from OMC (MIT) with ZO changes: no bare `429`/`overloaded` (0.4291, step 4290, "GPU overloaded"), tiered rate-limit patterns (banner / prose / loose-only-with-vocabulary), `awaiting_input` added (menu cursor `❯ 1.` required — plain numbered lists must not disable nudging), bare "interrupt" excluded but `⎿ Interrupted by user` included; (c) evidence = heartbeat tick deltas (pre-existing files baselined) + normalized text digest (spinner/counters stripped, first observation ≠ progress) + progress-path mtimes + **process-tree CPU time** (added by the fix round so a silent 40-minute training call inside one Bash tool is not a false stall/kill; a CPU-spinning hung process is bounded by `--timeout`); (d) rate-limit = paused state evaluated per poll, never a blocking sleep; reset time parsed in the operator's local tz; a static banner (the real TUI never clears the line) gets a bounded resume nudge after `paused_until`; resume is verified by heartbeat/file progress, never asserted; timeout excludes paused time, capped at an in-pause escalation; (e) escalation once per stall; tmux never kills a human-facing pane (logs blocking + `STALLED` at exit), headless kills by default (`kill_headless_on_escalate`) because it has no other lever until the driver; (f) the headless retry-with-backoff loop is REMOVED — exit while rate-limited → `RATE_LIMITED` + `resume_at` for the PR-B driver; (g) heartbeats live under the per-project memory root, are sealed against agent Write/Edit, and gitignored on the platform root and in the delivery `.zo/` templates (they were tracked before — recon caught it). + +**Correction of the 2026-08-12T15:30 entry:** `PostToolUseFailure` DOES fire on nonzero-exit Bash — `logs/comms/failures-2026-08-17.jsonl` captured this session's own `git checkout` exit-1. The "infrastructure errors only; add PostToolUse error inspection in WS-D" caveat is withdrawn. + +**Live evidence:** the new hook fired in this session for the lead and for workflow subagents (`agent_type=workflow-subagent`, `tick_count` 44 → `shutdown`) — PostToolUse carries `agent_id`/`agent_type` for subagents. + +**Verification method:** contract-first build (4 concurrent builders on disjoint files) → integrator → 3 adversarial verifier lenses (semantics / wiring+sealing / test quality: 19 findings, 2 high — banner reset times parsed in UTC; static banner could never resume) → fixer (11 applied with regression tests, 4 rejected with reasons). 929 → 1131 passed / 7 skipped, ruff clean, validate-docs 0 failures. Seeded tests for checks 11 and 12 on both loops. + +**Follow-ups (not done, recorded):** `_watchdog-ticks.jsonl` unbounded growth; `wrapper.py` 1404 lines (split `_wrapper_tmux.py`); verify the CPU-evidence idle threshold on a real tmux session; sealed-prefix symlink resolution in hookkit; `is_interrupt` from the failure feed not yet fed to `evaluate()`. diff --git a/memory/zo-platform/PRIORS.md b/memory/zo-platform/PRIORS.md index 4bf3bbc..036a94b 100644 --- a/memory/zo-platform/PRIORS.md +++ b/memory/zo-platform/PRIORS.md @@ -1406,3 +1406,38 @@ rm -f package-lock.json # keep the diff to the intend **Evidence:** 2026-08-12 session 040 — `claude`, `uv`, `npm` all exit 127 in the sandbox shell; no `.venv/` in repo; hooks verified live via the session's own runtime instead (see DECISION_LOG 15:30 entry). **Rules learned:** (1) Probe for a binary before building a plan around it (`which X` first, not after failure). (2) Keep every hook/script runnable on bare `python3` + stdlib-adjacent deps. (3) Machine-specific capabilities belong in a preflight check, not in assumptions — `zo preflight` should test for the claude CLI explicitly. **Confidence:** high + +## PR-047: "Built and tested" was never "wired" for the oracle-owned gate path — grep runtime callers before assuming a mechanism fires +**Source:** Session 041 (2026-08-17), Phase 3 recon swarm on `claude/v2-phase3-substrate` +**Root cause category:** ignored_rule (PR-009 rule 1 — "built and tested is not wired and enforced" — was itself never applied to `advance_phase`) + +**Failure:** `Orchestrator.advance_phase()` and `mark_subtask_complete()` — the "single enforcement point" named by PR-009, and the ONLY code that mints a gate nonce (`orchestrator.py:788`), evaluates the automated gate, runs `_auto_iterate_if_needed`, and flips the WS-B ledger — have zero runtime callers outside tests (`grep -rn advance_phase src/ .claude/ scripts/` → definition only). `zo build` launches one lead session per phase and calls `end_session()`, which writes the in-memory (unchanged) phase states back to STATE.md, clobbering any `## Phases` edit the lead made. Phases advanced in production only through hand-edited STATE.md (the PR-036/037 prod-001 incident was that lever failing), and `zo gates approve --nonce` could never find a nonce. Two whole workstreams (WS-A5 nonce gates, WS-B oracle-owned flips) shipped with passing seeded tests but sat behind an unreachable path. + +### Rules + +1. **A mechanism's wiring test must start from the CLI/hook entry point, not from the orchestrator method.** `test_artifacts_present_allows_gate` (PR-009) proved subtasks → gate → notebook — but started at `orch.mark_subtask_complete`, which nothing calls. A wiring test is only a wiring test if its first call is something a user or a hook actually invokes (`zo build`, a settings.json hook event, a slash command). + - **How to apply:** For every "oracle-owned" or "enforced" mechanism, add a test that patches the wrapper/session launch and asserts the CLI path reaches the mechanism (`advance_phase` called after `wait_for_completion` returns). PR-B's first test is exactly this. + +2. **Recon before build: enumerate runtime callers of every enforcement point with grep, and treat "definition only" as a red flag, not a detail.** The 2026-08-12 baseline review (9 agents) missed this; a single `grep -rn advance_phase` found it in seconds. Read-only mapper swarms that return exact `file:line` integration points (session 041 playbook) surface these gaps cheaply — run one before every substrate change. + +3. **STATE.md `## Phases` is a projection, never a control input, once a driver exists.** Until then it is the only lever operators have — so the PR-B cutover must ship the sanctioned `zo phase set` override in the same change (Sam's decision, DECISION_LOG 2026-08-17T09:00). + +### Verified Solution + +Not a code fix in PR-A — the finding shaped Phase 3's design (DECISION_LOG 2026-08-17T09:00: driver evaluates gates for ALL phases; PR-B is the first runtime caller). PR-A's own mechanisms follow rule 1: `tests/integration/test_hooks_shim.py::TestSettingsWiring::test_heartbeat_wired_on_post_tool_use` starts from settings.json; `tests/unit/test_wrapper.py::test_watchdog_tick_runs_on_suspected_dead_path` starts from `wait_for_completion`; `tests/unit/test_cli.py` asserts `build` passes `watchdog=/memory_root=/zo_session_id=` into the wrapper. The rule would have caught the original failure: a wiring test starting at `zo build` would have found no path to `advance_phase` on day one. + +## PR-048: Contract-first spawning applies to the lead's own build plan — a pinned API is the shared context; do not serialize the shared module ahead of parallel builders +**Source:** Session 041 (2026-08-17), PR-A build workflow; Sam's pushback ("why don't you use multiple agents … with shared context") +**Root cause category:** ignored_rule (CLAUDE.md design principle "contract-first spawning: define all agent interfaces before parallel spawn") + +**Failure:** The first PR-A workflow put the core module (`zo.watchdog`) on a serial critical path and gated the wrapper/hooks/config builders behind it, "to reduce integration risk" — even though the build contract already pinned the module's public API verbatim. ~12 minutes of a builder's work was lost restarting; the second run (four builders concurrent, core-finisher reconciling the drafted module while the others imported it) delivered the same integration quality (an integrator + adversarial verify pass caught the two mid-flight mismatches). + +### Rules + +1. **When the contract pins the interface, spawn every builder at once; the integrator step absorbs mismatches.** Subagents share no context window; the contract + integration map on disk ARE the shared context (ZO's own "state on disk" doctrine). Serializing a dependency only pays when the interface is genuinely undecided. +2. **Cross-builder seams go through the contract and an integrator, not live chatter** — pin kwargs/keys/paths in the contract, tell each builder to code defensively across the seam (`getattr` defaults, patched call sites in tests), and let one integrator reconcile. Peer messaging is for undecided interfaces only. +3. **Restart cheaply:** if a serial stage is already mid-flight and its artefact is on disk, keep the artefact and hand it to a "finisher" running concurrently with the rest, rather than throwing it away or waiting. + +### Verified Solution + +`scratchpad/pr-a-build.js` (session 041) rewritten from Core → Build(3) to Build(4 concurrent) → Integrate → Verify(3) → Fix; final: 1131 passed / 7 skipped, 19 verifier findings triaged, no integration defect reached the commit. Same playbook for PR-B. diff --git a/memory/zo-platform/STATE.md b/memory/zo-platform/STATE.md index 6ad190a..0e48075 100644 --- a/memory/zo-platform/STATE.md +++ b/memory/zo-platform/STATE.md @@ -8,7 +8,9 @@ status: complete ## Current Position -**Session 040 (current) — pick up here.** Research + decision session: deep-dive review of three agent-orchestration repos (oh-my-claudecode, ruflo, ralph — cloned to `~/Documents/code/`) to inform the ZO v2 rearchitecture. 9-agent workflow (7 source-reading lenses + ZO baseline + adversarial synthesis, ~1.06M tokens) catalogued **63 features**, distilled to **12 ranked adoptions** + 6 rearchitecture themes + 11 anti-patterns; all findings persisted to `memory/zo-platform/research/2026-08-12-repo-reviews/` (per-repo markdown + `raw-findings.json`). **Sam decided: adopt all 12.** Work organized into **five layer-based workstreams** (A enforcement plane, B control plane, C execution substrate, D self-learning/platform oracle, E operator experience) — NOT source-repo categories, because features from different repos interlock into single mechanisms. Shipped this session: `plans/zo-v2-rearchitecture.md` (full plan: 6 gated phases, 20-check oracle, anti-scope; + `.gitignore` exception), `docs/reference/v2-rearchitecture.mdx` (all 12 features w/ provenance + repo links; added to mint.json Reference nav), `docs/roadmap.mdx` v2 section (4 pillars + repo credits), website §11 "What's next" (new section w/ 3 repo credit cards; quick start renumbered §12; drawer nav updated). **Verification caveat: no Node.js on this machine** — Astro build NOT run; website change verified via HTML-parser balance check + static-server DOM inspection (section text, all 4 links, drawer entry, renumbering all confirmed rendered). CI/deploy build must confirm. **Same session, part 2 — v2 Phase 1 (WS-A enforcement plane) SHIPPED** on branch `claude/v2-phase1-enforcement` (stacked on the plan branch, PR #106): (A1) `src/zo/contracts.py` — contracts.json emitted at decompose into memory_root (gate_mode precedent), `contract_produced` upgraded from prose placeholders to concrete paths (ownership ∩ required_artifacts, ownership-dir fallback), SubagentStop hook validates deliverables (missing/undersized/pattern/empty-dir) and blocks with a violation list; (A2) drift-guard Stop hook — completion-claim regex over the last assistant transcript message + added TODO/FIXME/NotImplementedError lines in `git diff HEAD` → block (env kill-switch `ZO_DRIFT_GUARD=0`); (A3) PreCompact (STATE flush + checkpoint decision), SessionEnd (summary backfill), PostToolUseFailure (`logs/comms/failures-{date}.jsonl` feed) — specs/memory.md recovery section updated to match (replaces the never-built periodic postToolUse checkpoint design); (A4) sealed-paths PreToolUse guard — memory-root control files (gate_mode/gate_nonce/gate_decision/contracts.json/sealed_paths) + user `sealed_paths` prefixes denied, off-limits write-scope enforced per contracts.json when agent identity present in hook input (plan check 6 AMENDED: no disallowedTools frontmatter exists for subagents and verifiers need scoped writes — path-scoped enforcement instead, fail-open without identity); (A5) nonce gates — minted at GATED (`secrets.token_hex(8)` → `gate_nonce` file), surfaced in `prepare_gate_review`, `apply_human_decision` raises PermissionError without it (single-use, cleared on terminal decisions), new `zo gates approve/reject --nonce` CLI writes DECISION_LOG + comms + `gate_decision` file consumed on next decompose, `/approve`+`/reject` slash commands rewritten to route through the CLI (forgeable hand-edit path CLOSED). All via one shim (`.claude/hooks/zo-hookkit.sh` → `python3 -m zo.hookkit`, venv-preferring, fail-open) + 6 new settings.json wirings. **First-ever hook-script tests** (subprocess + stdin JSON pattern). **854 → 904 passed / 7 skipped, ruff `src/` clean, validate-docs green.** 4 pre-existing integration tests updated to pass the nonce (designed behaviour change). Plan oracle checks 1-5 + 7 have passing seeded-failure tests; check 6 as amended. **Part 3 — live pre-PR verification (Sam-directed):** added always-on hook-trace observability (`logs/hook-trace-{date}.jsonl`, `ZO_HOOK_TRACE=0` off-switch) and verified in the live session itself: sealed-paths DENIED a real Write to gate_mode; drift-guard fired correctly-silent on a real Stop; subagent-stop fired with `agent_type`+`agent_id` in the live payload — **agent-identity open question RESOLVED** (per-agent enforcement keys correctly). Drift-guard now prefers the live payload's `last_assistant_message` (transcript parse = fallback). Caveats logged: PostToolUseFailure doesn't fire on nonzero-exit Bash (infrastructure errors only); PreCompact/SessionEnd not yet observed live; full `zo build` demo needs a machine with the claude CLI (this Mac has none — PR-046). **908 passed / 7 skipped, ruff clean.** **Part 4 — Phases merged + Phase 2 (WS-B control plane) SHIPPED:** #106 + #107 merged to main (stack conflict resolved by merging main into the branch, branch side kept — main had nothing unique). Then WS-B on `claude/v2-phase2-control-plane`: `src/zo/ledger.py` (plan-ledger.json: per-subtask entries w/ synthesized criteria, merge-preserving regeneration, atomic writes, phase_status map); oracle-owned flips wired at the two verified-completion sites (automated gate + nonce-verified human PROCEED), resets on ITERATE/loop-CONTINUE, attempts on mark_subtask_complete, ledger sealed via `_SEALED_DEFAULTS`; `## Stories` plan section + sizing lint in validate_plan (fires only when stories declared — legacy plans untouched); `zo status` renders the control-plane table from the ledger (STATE.md = fallback/projection). Phase-1 hardenings from recon: `contracts.set_active_phase` now atomic; `zo build` exports ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_CONTRACTS_PATH so per-project sealing works in delivery sessions. Deferred to Phase 3 (documented): evaluate_loop_state ledger input, session-restore cutover (PR-036 precedence). **929 passed / 7 skipped, ruff clean, validate-docs green. Oracle checks 8-10 seeded tests pass.** **Next:** Phase 2 PR review/merge → Phase 3 (WS-C: watchdog, then fresh-context loop — demo validation needs the Linux box w/ claude CLI, PR-046). +**Session 041 (current) — pick up here.** v2 **Phase 3 (WS-C execution substrate) — part 1 of 2 SHIPPED: PR-A the watchdog** (plan oracle checks 11–12) on branch `claude/v2-phase3-substrate` (cut off main after **PR #108 / Phase 2 merged** at the start of this session, squash `1faf53a`). Method: read-only recon swarm (7 mappers + synthesis → `memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md`, every claim `file:line`) → build contract (`pr-a-build-contract.md`, contract-first: pinned `zo.watchdog` API, heartbeat schema, tick algorithm, per-builder file ownership) → 4 concurrent builders → integrator → 3 adversarial verifiers (19 findings, 2 high) → fixer (11 applied with regression tests, 4 rejected with reasons). **Structural finding that reframes Phase 3:** `Orchestrator.advance_phase()` / `mark_subtask_complete()` have ZERO runtime callers — the automated gate, `_auto_iterate_if_needed`, the WS-B `mark_phase_passed` flip and gate-nonce minting are unreachable in production; `zo build` = one lead session per phase then `end_session()`; phases only ever advanced via hand-edited STATE.md (the PR-036/037 prod-001 incident). PR-B (the driver) is therefore the first runtime caller, not a Phase-4-only optimisation. **Sam decided (this session):** (1) the driver evaluates gates for ALL phases (Phase 4 alone gets fresh headless spawns per iteration until check 13 says extend); (2) two PRs (A watchdog, B driver + fresh-context loop + deferrals); (3) after the restore cutover the ledger wins over STATE.md with a loud warning + a sanctioned `zo phase set` override logged to DECISION_LOG; (4) tmux nudges default ON behind a pane-ready / no-permission-dialog guard. **PR-A shipped:** `src/zo/watchdog.py` (+`_watchdog_models.py`, `_watchdog_text.py`, `_proc.py`): three-state freshness (unknown never = stall), never-block taxonomy ported from OMC (context-limit #213 / rate-limit #777 / auth #1308 / user-abort incl. `⎿ Interrupted by user`) + ZO-added `awaiting_input` (a permission dialog is never sent Enter) + `compacting`; tiered rate-limit patterns (no bare `429`/`overloaded`); `parse_rate_limit_reset` (local tz); PID + process-start-time identity, positive-proof-only (EPERM = alive, recycled pid = dead); pure `evaluate()` policy (progress → never-block → pause/resume → grace → stall → nudge budget 3 with 30 s dwell → escalate once). Heartbeats: stdlib-only `zo._hook_heartbeat.stamp_heartbeat` on a new `PostToolUse *` settings.json entry (+ Stop→ready, PreCompact→compacting, SubagentStop/SessionEnd→shutdown), keyed `agent_id` or `lead-`, written to `/heartbeats/`, **sealed** (`_SEALED_DEFAULTS`) and gitignored (platform root + delivery `.zo/` templates). Wrapper: `WatchdogRunner` (`_wrapper_watchdog.py`) ticked from BOTH loops before the liveness reads (incl. the suspected-dead `continue` path); one pane capture per poll; evidence = heartbeat tick deltas (pre-existing files baselined) + normalized pane/stdout digest (spinners/counters stripped) + progress-path mtimes (ledger, comms dir, `.zo/experiments`) + process-tree CPU time (a silent 40-min training call is NOT a stall); tmux nudge via named paste buffer only when the pane shows the idle prompt; rate-limit = paused state evaluated per poll (no blocking sleep), reset-time parse → probe → verified resume by real progress (static banner gets a bounded resume nudge); timeout excludes paused time; escalation: tmux logs `error_type=stall severity=blocking` + `STALLED` at exit (never kills a human-facing pane), headless kills (`kill_headless_on_escalate`, config). New `AgentStatus.PAUSED_RATE_LIMIT/STALLED`, `LeadProcess.pid_start_identity/resume_at/...`; headless retry loop REMOVED (exit → `RATE_LIMITED` + `resume_at` for the PR-B driver). Config: `ProjectConfig.watchdog: WatchdogConfig` (round-trips), `zo build/continue --no-watchdog`, `ZO_WATCHDOG=0`, `ZO_WATCHDOG_STALL_SEC`; CLI threads watchdog/memory_root/`ZO_SESSION_ID`. Docs: `specs/watchdog.md` rewritten to implemented reality (RFC cron-tick superseded — divergence stated up front), mdx status, COMMANDS.md/build.mdx flag. **Live evidence in this very session:** the heartbeat hook fired for the lead (`lead-.json`, `executing/Bash`) and for the workflow subagents (`agent_type=workflow-subagent`, `tick_count` 44 → `shutdown` on SubagentStop) — PostToolUse DOES carry `agent_id`/`agent_type` for subagents (open question resolved). **Correction to session 040:** `PostToolUseFailure` DOES fire on nonzero-exit Bash (`logs/comms/failures-2026-08-17.jsonl` captured this session's own `git checkout` exit 1) — the "infra errors only, queue for WS-D" caveat was wrong; no WS-D work needed there. **929 → 1131 passed / 7 skipped, ruff `src/` clean, validate-docs 0 failures (badge 1053).** Seeded tests: 10-min stall detected + escalated within one poll (both loops), rate-limited session never nudged (both loops), rate-limit pause auto-resumes on reset with verified progress (check 12, both loops), permission dialog never nudged, heartbeats sealed, settings wiring, tick invoked from both loops, CLI threading. **Next:** PR-B on the same branch after PR-A merges (or stacked): `zo.driver.run_phase_loop` — first runtime caller of `advance_phase`; fresh headless spawns per Phase-4 iteration (`_launch_headless`, prompt on stdin, process-group kill); oracle-side subtask completion from ledger criteria; git checkpoint via `surrogate.commit_worktree`; `evaluate_loop_state(..., ledger=)`; restore cutover ledger>STATE.md (+`LedgerEntry.completed`, HOLD write-through, decompose ordering fix, `zo phase set`); wire `parse_next_md/parse_hypothesis_md` (DEAD_END currently dead code); fix absolute `Experiment.artifacts_dir` (check-13 blocker); check 13 needs the Linux box (PR-046). Follow-ups noted, not done: `_watchdog-ticks.jsonl` unbounded, wrapper.py 1404 lines (split `_wrapper_tmux.py`), CPU-evidence idle threshold to verify on a real tmux session. + +**Session 040 (prior).** Research + decision session: deep-dive review of three agent-orchestration repos (oh-my-claudecode, ruflo, ralph — cloned to `~/Documents/code/`) to inform the ZO v2 rearchitecture. 9-agent workflow (7 source-reading lenses + ZO baseline + adversarial synthesis, ~1.06M tokens) catalogued **63 features**, distilled to **12 ranked adoptions** + 6 rearchitecture themes + 11 anti-patterns; all findings persisted to `memory/zo-platform/research/2026-08-12-repo-reviews/` (per-repo markdown + `raw-findings.json`). **Sam decided: adopt all 12.** Work organized into **five layer-based workstreams** (A enforcement plane, B control plane, C execution substrate, D self-learning/platform oracle, E operator experience) — NOT source-repo categories, because features from different repos interlock into single mechanisms. Shipped this session: `plans/zo-v2-rearchitecture.md` (full plan: 6 gated phases, 20-check oracle, anti-scope; + `.gitignore` exception), `docs/reference/v2-rearchitecture.mdx` (all 12 features w/ provenance + repo links; added to mint.json Reference nav), `docs/roadmap.mdx` v2 section (4 pillars + repo credits), website §11 "What's next" (new section w/ 3 repo credit cards; quick start renumbered §12; drawer nav updated). **Verification caveat: no Node.js on this machine** — Astro build NOT run; website change verified via HTML-parser balance check + static-server DOM inspection (section text, all 4 links, drawer entry, renumbering all confirmed rendered). CI/deploy build must confirm. **Same session, part 2 — v2 Phase 1 (WS-A enforcement plane) SHIPPED** on branch `claude/v2-phase1-enforcement` (stacked on the plan branch, PR #106): (A1) `src/zo/contracts.py` — contracts.json emitted at decompose into memory_root (gate_mode precedent), `contract_produced` upgraded from prose placeholders to concrete paths (ownership ∩ required_artifacts, ownership-dir fallback), SubagentStop hook validates deliverables (missing/undersized/pattern/empty-dir) and blocks with a violation list; (A2) drift-guard Stop hook — completion-claim regex over the last assistant transcript message + added TODO/FIXME/NotImplementedError lines in `git diff HEAD` → block (env kill-switch `ZO_DRIFT_GUARD=0`); (A3) PreCompact (STATE flush + checkpoint decision), SessionEnd (summary backfill), PostToolUseFailure (`logs/comms/failures-{date}.jsonl` feed) — specs/memory.md recovery section updated to match (replaces the never-built periodic postToolUse checkpoint design); (A4) sealed-paths PreToolUse guard — memory-root control files (gate_mode/gate_nonce/gate_decision/contracts.json/sealed_paths) + user `sealed_paths` prefixes denied, off-limits write-scope enforced per contracts.json when agent identity present in hook input (plan check 6 AMENDED: no disallowedTools frontmatter exists for subagents and verifiers need scoped writes — path-scoped enforcement instead, fail-open without identity); (A5) nonce gates — minted at GATED (`secrets.token_hex(8)` → `gate_nonce` file), surfaced in `prepare_gate_review`, `apply_human_decision` raises PermissionError without it (single-use, cleared on terminal decisions), new `zo gates approve/reject --nonce` CLI writes DECISION_LOG + comms + `gate_decision` file consumed on next decompose, `/approve`+`/reject` slash commands rewritten to route through the CLI (forgeable hand-edit path CLOSED). All via one shim (`.claude/hooks/zo-hookkit.sh` → `python3 -m zo.hookkit`, venv-preferring, fail-open) + 6 new settings.json wirings. **First-ever hook-script tests** (subprocess + stdin JSON pattern). **854 → 904 passed / 7 skipped, ruff `src/` clean, validate-docs green.** 4 pre-existing integration tests updated to pass the nonce (designed behaviour change). Plan oracle checks 1-5 + 7 have passing seeded-failure tests; check 6 as amended. **Part 3 — live pre-PR verification (Sam-directed):** added always-on hook-trace observability (`logs/hook-trace-{date}.jsonl`, `ZO_HOOK_TRACE=0` off-switch) and verified in the live session itself: sealed-paths DENIED a real Write to gate_mode; drift-guard fired correctly-silent on a real Stop; subagent-stop fired with `agent_type`+`agent_id` in the live payload — **agent-identity open question RESOLVED** (per-agent enforcement keys correctly). Drift-guard now prefers the live payload's `last_assistant_message` (transcript parse = fallback). Caveats logged: PostToolUseFailure doesn't fire on nonzero-exit Bash (infrastructure errors only); PreCompact/SessionEnd not yet observed live; full `zo build` demo needs a machine with the claude CLI (this Mac has none — PR-046). **908 passed / 7 skipped, ruff clean.** **Part 4 — Phases merged + Phase 2 (WS-B control plane) SHIPPED:** #106 + #107 merged to main (stack conflict resolved by merging main into the branch, branch side kept — main had nothing unique). Then WS-B on `claude/v2-phase2-control-plane`: `src/zo/ledger.py` (plan-ledger.json: per-subtask entries w/ synthesized criteria, merge-preserving regeneration, atomic writes, phase_status map); oracle-owned flips wired at the two verified-completion sites (automated gate + nonce-verified human PROCEED), resets on ITERATE/loop-CONTINUE, attempts on mark_subtask_complete, ledger sealed via `_SEALED_DEFAULTS`; `## Stories` plan section + sizing lint in validate_plan (fires only when stories declared — legacy plans untouched); `zo status` renders the control-plane table from the ledger (STATE.md = fallback/projection). Phase-1 hardenings from recon: `contracts.set_active_phase` now atomic; `zo build` exports ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_CONTRACTS_PATH so per-project sealing works in delivery sessions. Deferred to Phase 3 (documented): evaluate_loop_state ledger input, session-restore cutover (PR-036 precedence). **929 passed / 7 skipped, ruff clean, validate-docs green. Oracle checks 8-10 seeded tests pass.** **Next:** Phase 2 PR review/merge → Phase 3 (WS-C: watchdog, then fresh-context loop — demo validation needs the Linux box w/ claude CLI, PR-046). **Session 039 (prior).** Test-only change making the CLI command-set test plugin-tolerant: `tests/unit/test_cli.py::TestCliGroup::test_cli_group_has_all_commands` asserted strict equality on `cli.commands`, which contradicts the extension points ZO itself shipped in PR #99 — any downstream build installing a `zo.commands` entry-point plugin (the documented mechanism in `zo.extensions.load_cli_plugins`) adds commands and turned the core suite red in that environment, despite zero core changes. Changed `assert expected == actual` → `assert expected <= actual` with an explanatory comment. Trade-off accepted: equality also caught *core* commands added without updating the test; subset still catches removals, and additions have always required updating this test's `expected` set anyway. Surfaced by the first real downstream plugin registering a new command group. No code, version, agents, or docs touched — single-assertion diff (+ memory). **854 passed / 7 skipped on Python 3.11 AND 3.12, ruff `src/` clean, validate-docs 0 failures.** **Next:** unchanged — Batch **D** / **E**, standing Tier-1 (caveman, onboarding). diff --git a/memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md b/memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md new file mode 100644 index 0000000..c1cdb15 --- /dev/null +++ b/memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md @@ -0,0 +1,253 @@ +# ZO v2 Phase 3 (WS-C) — Build-Ready Integration Map + +Branch: `claude/v2-phase3-substrate` (recon done on `claude/v2-phase2-control-plane` @ 8466d29). Authoritative design source: `plans/zo-v2-rearchitecture.md:107-115` (external checker in the LifecycleWrapper poll loop; NOT a monitor agent; NOT a cron). `specs/watchdog.md` §3.1/§3.4/§3.5/§4/§5 are superseded (see §5, §7). + +Line refs re-verified 2026-08-17 where mappers disagreed (see §5). Three ground-truth facts that constrain everything below: + +1. **One `zo build` = one lead session = one phase; there is no loop anywhere in cli.py.** `cli.py:1134 get_current_phase` → `:1143 build_lead_prompt` → `:1150 _launch_and_monitor` → `:800 launch_lead_session` → `:920 wait_for_completion` → `:942 orchestrator.end_session()` → `:953 deregister_session` / `:955-957 consolidate_all`. `zo continue` (`cli.py:1206`) is `click_ctx.invoke(build, ...)` at `cli.py:1274`. `Orchestrator.advance_phase` (`orchestrator.py:752`) and `mark_subtask_complete` (`:848`) have **zero runtime callers**; therefore `_auto_iterate_if_needed` (`:1239`), `evaluate_loop_state` call (`:1276`), `_finalize_experiments` (`:1163`), `mark_phase_passed` (`:829`) are all unreachable in production today. Phase 3 supplies the missing driver, it does not replace one. +2. **Two structurally different poll loops.** `_wait_tmux` (`wrapper.py:669-761`, default path via `use_tmux = not no_tmux` `cli.py:798`) has NO pid, NO rate-limit handling, NO stdout log; `_wait_headless` (`wrapper.py:763-822`) has a pid but its rate-limit code is effectively dead (reads only `stdout_log` at `:792`, which under `--print --output-format json` `:436-437` is a single blob at exit) and is a retry loop (`:793-808`). +3. **No process identity anywhere.** tmux `LeadProcess(pid=None, ...)` `wrapper.py:278`; `surrogate._pid_alive` `:279` is bare `os.kill(pid,0)`; `register_session` stores lock-write time as `started_at` `:261`; psutil absent (`pyproject.toml:34-40`); no `pane_pid|pgrep|lstart|etime` anywhere in src. + +--- + +## 1. Watchdog (oracle checks 11–12) + +### 1.1 Module boundary + +**New `src/zo/watchdog.py`** (pure logic, no subprocess in the classifier path; <500 L): +- `HeartbeatRecord(BaseModel)`: `schema_version:int=1`, `agent_id:str`, `agent_type:str`, `session_id:str`, `pid:int|None`, `process_start_identity:str|None` (tagged `"{platform}:{value}"`, port of OMC `team-owner-epoch.ts:69-112`), `last_tick_at:str` (ISO), `status: Literal["ready","polling","executing","compacting","shutdown"]` (OMC `types.ts:103` + ZO-added `compacting`), `last_event:str`, `progress_marker:str|None` (ledger mtime or last tool_use_id — see §5 "fresh mtime, zero progress"). +- `load_heartbeat(path) -> HeartbeatRecord | None` (fail-open, `contracts.py:164-169` shape) BUT the checker converts to a **three-state** `Freshness = fresh|stale|unknown`; `unknown` is never a stall verdict (hazard: `contracts.py:167` inversion; OMC `team-owner-epoch.ts:146-147` "unknown identity is never positive proof of death"). +- `NeverBlockReason(StrEnum)`: `context_limit|rate_limit|auth_error|user_abort|compacting|unknown` + `classify_never_block(text_tail: str, *, is_interrupt: bool|None=None) -> NeverBlockReason|None` — pattern tables ported verbatim from OMC `todo-continuation/index.ts:370` (context #213), `:390` (rate #777), `:442` (auth #1308, 16 patterns), `:295` (user abort; bare `interrupt` deliberately excluded per #2478), normalizer `:141`; plus OMC `tmux-detector.ts:34` rate-limit text patterns AND its false-positive layer (`stripGitOutputLines :153`, saved-transcript reject `:70/:74`, new-lines-only cursor `:376-389`). Include MIT attribution comment (`oh-my-claudecode/LICENSE:1-3`). +- `StallVerdict(BaseModel)` modelled on `LoopDecision` (`experiment_loop.py:183-202`, `reason` field for DECISION_LOG, `evaluated_at`). +- `WatchdogState` per-run dataclass (tick count, `nudges_used`, `paused_until`, `last_progress_at`, `last_scan_cursor`, `consecutive_stale`) — reset at top of `wait_for_completion` (hazard `wrapper.py:576-580` ad-hoc self attrs; mapper 5). Persist nudge budget to `/heartbeats/_watchdog.json` so a wrapper restart cannot refill it (OMC `idle-nudge.ts:66` in-memory hole). +- `process_start_identity(pid) -> str|None`: linux `/proc//stat` field 22 after last `)`; darwin `ps -o lstart= -p N` under `LC_ALL=C`; validator regex allowlist ≤1024 chars (OMC `team-owner-epoch.ts:112`). `identities_may_match` with darwin `usec=='0'` wildcard (`:128`). `is_process_identity_dead(pid, recorded)` positive-proof-only (`:138-147`); EPERM = alive (ruflo `swarm-tools.ts:85`). +- Consider `src/zo/_proc.py` for `_pid_alive` + start-identity, shared with `surrogate.py` (mapper 1 recommends not importing the private `surrogate._pid_alive`). + +**Wrapper owns the tick** (`wrapper.py`): `_watchdog_tick(process, *, pane_text|stdout_tail)` called from both loops; nudge/pause/escalate side effects live here because they need `self._comms`, tmux, and `LeadProcess`. + +**Escalation-to-restart lives ABOVE the wrapper** in the new driver (§2) — `kill_session` (`wrapper.py:824-861`) returns a terminal `ERRORED`/`exit_code=-9` LeadProcess by contract (`:836-840`, `:857-861`); do not relaunch inside `_wait_tmux`. + +### 1.2 Heartbeat writer (hook side) + +- **Handler:** add `"heartbeat": _handle_heartbeat` to `_HANDLERS` (`hookkit.py:382-389`); `main()` fail-open dispatch (`:392-407`, bare except `:402`, `_trace` on both paths `:403/:405`, always return 0) gives fail-open for free. +- **Wiring:** append a SECOND object to the `PostToolUse` array in `.claude/settings.json:74` (currently only `Write|Edit` → `cascade-reminder.sh` `:74-85`) with matcher `"*"` (or `""`) → `bash .claude/hooks/zo-hookkit.sh heartbeat 2>/dev/null || exit 0`, timeout 5. Do not touch the Write|Edit entry. Also call the same writer from `_handle_subagent_stop` (`hookkit.py:120`, wired `:103-114`) and `_handle_precompact` (`:233`, wired `:115-127`, write `status="compacting"`) and Stop/`drift-guard` (`:196`, wired `:86-102`). +- **Identity:** new `_agent_identity(data) -> (agent_type, agent_id)`; do NOT change `_agent_name` (`hookkit.py:112-117`, contract lookup at `:355-356` depends on it). Evidence: `agent_id`+`agent_type` present in SubagentStop (11/57 lines, `logs/hook-trace-2026-08-12.jsonl:3,:57`) and PostToolUseFailure (`:55`, `hook-trace-2026-08-17.jsonl` last line); ABSENT in all 34/34 PreToolUse `sealed-paths` traces. PostToolUse identity is unverified (not routed today) — first commit ships the handler as trace-only, run one live session, read `stdin_keys` (same method as `DECISION_LOG.md:1261`). Filename key = `agent_id` when present, else `session_id` namespaced (`lead-.json`). +- **Path:** `/heartbeats/.json` where memory_root = `hookkit._memory_root(repo_root)` (`hookkit.py:64-69`: `$ZO_MEMORY_ROOT` → `/memory/zo-platform` if dir → None; `repo_root` = `$ZO_REPO_ROOT` else cwd `:60-61`). Return early on None (pattern `:236`, `:260`). Write via `ledger._atomic_write` (`ledger.py:81-91`). +- **Perf:** move `from zo.contracts import ...` (`hookkit.py:34`) lazy into the handlers that need it (style at `:226,:238,:262`); debounce: skip write if existing mtime < 2 s old. +- **Env for hooks:** `ZO_REPO_ROOT` (shim `.claude/hooks/zo-hookkit.sh:29`), `ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_CONTRACTS_PATH` (`cli.py:1076-1078` → `_launch_and_monitor(extra_env=)` `cli.py:1163` → `launch_lead_session` `wrapper.py:143-151` → tmux inline prefix `:230-231` / headless `env.update` `:456-463`). Add `ZO_SESSION_ID` (comms session id minted `cli.py:1059`/`:1094`) so heartbeat ↔ comms correlate. +- **Gitignore + sealing (same PR, before first write):** `memory/zo-platform/heartbeats/` is currently TRACKED (`.gitignore:33-34` re-includes `memory/zo-platform/`; `git check-ignore` matched only `logs/heartbeats/x.json` via `.gitignore:27`) — add `memory/zo-platform/heartbeats/`, and also `memory/zo-platform/contracts.json` + `plan-ledger.json` (`contracts.py:10` docstring "gitignored" is false for the platform root). Add `"heartbeats"` to `_SEALED_DEFAULTS` (`hookkit.py:47-50`; prefix match `:340-342` seals the subtree) so agents cannot forge liveness via Write/Edit; hook-internal python writes never traverse PreToolUse (`.claude/settings.json:63-72` matcher `Write|Edit`, handler `hookkit.py:328-379` reads `tool_input.file_path`). +- **Shim scope constraint:** `.claude/hooks/zo-hookkit.sh:21` `[[ -d "$REPO_ROOT/src/zo" ]] || exit 0` — heartbeats exist only for sessions whose `.claude/` is the platform repo's. Holds today because lead is launched `cwd=str(zo_root)` (`cli.py:801`, `--add-dir` `wrapper.py:226`); no hooks are scaffolded into delivery repos (`scaffold.py` has none). Assert as a Phase-3 invariant with a test; fresh-context builders (§2) MUST also be launched with `cwd=zo_root` + `ZO_MEMORY_ROOT`. +- **Non-build sessions:** report (`cli.py:3491`), init-architect (`:2002`), draft (`:3168`) call `_launch_and_monitor` without `extra_env` → no `ZO_MEMORY_ROOT` → hookkit falls back to platform root. Document heartbeats as build/continue-only, or set env inside `_launch_and_monitor` from `delivery_repo`. + +### 1.3 External checker insertion points (both loops) + +**tmux (`_wait_tmux` `wrapper.py:669-761`):** +- Insert `self._watchdog_tick(...)` at **line 712**, immediately after `self._check_gate_mode_change()` (`:710`) and `self._maybe_open_training_pane()` (`:711`), BEFORE liveness reads `:713-714`. NOT after `:749` — the suspected-dead branch `:741-745` calls `on_status`, sleeps `min(poll_interval, _DEAD_RECHECK_INTERVAL)` (`:744`) and `continue`s (`:745`), skipping the timeout check `:754`. Regression test: watchdog fires on a poll that takes the `:745` path. +- Hoist ONE `pane_text = self._capture_tmux_pane(pane_id, lines=200)` per poll (def `:998`, default 50; today only 5 lines and only inside `if on_status:` `:749-752`) and share with `on_status` — otherwise 4 subprocess spawns/poll (`_tmux_pane_alive :920-931` does `list-panes -a`; `_tmux_claude_running :934-960` display-message). +- Available per-poll: `pane_exists`(713), `claude_running`(714), `poll_count`(706/716), `consecutive_dead`(707), `start_time=time.monotonic()`(702), `process.started_at`(279). Debounce constants `_STARTUP_GRACE_POLLS=2 :74`, `_DEAD_CONFIRM_POLLS=2 :78`, `_DEAD_RECHECK_INTERVAL=2.0 :81`, rationale docstring `:677-701` — reuse for stall confirmation, don't add a second debounce. +- Do NOT let the watchdog inherit `_tmux_claude_running` semantics (`:947-960` compares `#{pane_current_command}` to a shell set `:959-960`; true for any non-shell foreground; may flip to `bash` during long Bash tool calls — see open Q). + +**headless (`_wait_headless` `wrapper.py:763-822`):** +- Insert at **line 778** (after `_check_gate_mode_change()` `:777`, before `rc = self._proc.poll()` `:779`). +- Never-block classification runs between `:792` (`output = self._read_tail(process.stdout_log)`) and `:793` (`if self._detect_rate_limit(output)`), i.e. BEFORE any backoff decision. Read `stderr_log` too (`:453/:455`, never read today); track a byte cursor rather than re-tailing 100 lines (`_read_tail :1029-1037`) — otherwise a single `429` re-matches forever and `retries` never resets (`:808`). +- Fix `self._proc` init: assigned only at `:287` (None, tmux) and `:474` (headless); `__init__` `:83-99` never sets it → `AttributeError` on any fresh `LifecycleWrapper.wait_for_completion(reconstructed_process)` (the restart pattern) at `:779` and `kill_session :847`. Add `self._proc: subprocess.Popen | None = None` next to `:99`. Tests only pass because they set `wrapper._proc` by hand (`tests/unit/test_wrapper.py:468,488,510,531`). + +**Shared:** factor `_poll_tick(process, *, pane_text, stdout_tail)` used by both loops (parity already broken: `_wait_headless` never calls `_maybe_open_training_pane`, passes `""` to `on_status` `:813`). `wait_for_completion` (`:547`, `poll_interval=10.0 :551`, `timeout=None :552`) gains keyword-only `watchdog: WatchdogConfig|None`, `memory_root: Path|None`; stash alongside `:574-578` inside the same `try/finally` (`:585-586`, teardown mirrors `_close_training_pane :636-646`). Only production caller `cli.py:920` passes on_status/gate_mode_file/project_name/delivery_repo — thread `memory_root` explicitly (do not derive from `gate_mode_file.parent`, `cli.py:1159`). + +**Timeout accounting:** `timeout` is `time.monotonic() - start_time` since `:702/:772` — a rate-limit pause would count against it (`:754`). Track `last_progress_at` separately and exclude paused intervals. + +### 1.4 Never-block taxonomy — detection sources + +| Reason | tmux source | headless source | Hook/feed source | +|---|---|---|---| +| rate_limit | pane tail (last N lines, cursor-tracked) w/ OMC patterns + git-strip | stderr/stdout tail via cursor | — | +| context_limit | pane tail (`context low`/compact banner) | stdout json envelope + transcript % estimate (OMC `persistent-mode/index.ts:1003-1008`) | PreCompact → `status=compacting` heartbeat | +| auth_error | pane tail | stderr tail | — | +| user_abort | — | rc + `is_interrupt` | `PostToolUseFailure.is_interrupt` (present in every live trace `logs/hook-trace-2026-08-12.jsonl:40,55`) — currently DROPPED by `_handle_post_tool_failure` `hookkit.py:287-308`; add `is_interrupt`, `agent_id`, `agent_type`, `never_block_reason` to the failure-feed record (`:294-302`, file `/failures-.jsonl`, live records `logs/comms/failures-2026-08-17.jsonl`) | +| thinking-only streak | — | — | Stop payload `transcript_path` (`hookkit.py:197,205-207`); extend `_last_assistant_text` (`hookkit.py:153`) into a port of OMC `classifyLastAssistantTurn` (`persistent-mode/index.ts:1513-1607`, streak max 3, TTL 5 min `:1479-1481`, fail-open) | + +Rules: taxonomy runs FIRST in the tick, before any nudge (tmux: a rate-limited TUI still shows `claude` as foreground → looks RUNNING forever until `timeout :754`; naive watchdog at 712 would nudge it — the exact check-11 failure). Do NOT reuse `_RATE_LIMIT_PATTERNS` (`wrapper.py:51-56`, bare `r"429"` at `:52`, matcher `:894-896`) against pane text (`0.4291`, `step 4290`). Ambiguous → do not nudge. Bypass order ported from OMC `persistent-mode/index.ts:2268-2373`. + +### 1.5 Nudge mechanism + budget + +- **tmux only.** Extract `_paste_and_submit(pane_id, text)` from `_launch_tmux :255-270` (`load-buffer :256`, `paste-buffer -t :260`, `sleep(1) :266`, `send-keys Enter :267-270`; retry variant `_verify_prompt_submitted :364-398`). Use a **named buffer** (`tmux load-buffer -b zo-nudge -` from stdin; `paste-buffer -b zo-nudge -d`) so the operator's paste buffer isn't clobbered and no temp file is needed. +- **Headless cannot be nudged**: `Popen(cmd, stdout=fh, stderr=fh, text=True, env=env)` `:462-463` inherits parent stdin; prompt is one-shot argv `-p` `:450` under `--print` `:436`. Ladder = stale-heartbeat → `kill_session :824` → fresh relaunch with re-derived prompt (§2). +- **Budget** (port OMC `idle-nudge.ts:29-131`): `delay 30 s` dwell before first nudge, `max 3` per agent per run, `scan throttle 5 s`, idle timer resets after each successful nudge, message "Continue working on your assigned task and report concrete progress (not ACK-only)". Predicate = heartbeat stale AND (positive process death OR zero progress-delta over N ticks) AND never-block reason is None. Never nudge the lead pane while `paused_until` is set. Persist `nudges_used` (§1.1). Exhaustion → `AgentStatus.STALLED` returned from `wait_for_completion` → driver escalates to iteration restart (`_record_learning` `orchestrator.py:1323` so stalls become priors). Model on `_maybe_open_training_pane :588-634` (fires once, sentinel `self._training_pane_id = ""` `:632/:634`, torn down in `finally`). +- Global backstop: an un-raisable hard cap on total nudges + restarts per run (OMC `security-config.ts:44-108`: overrides may only lower). + +### 1.6 Rate-limit wait-and-resume (both modes) + +- Replace `:793-808` retry loop (`_backoff_wait :898-900` = `30*2^n + U(0,5)`, `_max_retries=3 :89/:96`, terminal `RATE_LIMITED :795`) with a **paused state evaluated each poll**: `while time.monotonic() < resume_at: tick(); sleep(min(poll_interval, remaining))` — never a single blocking `time.sleep(wait_secs)` `:807` (kills `on_status`, timeout, gate-mode re-read, and watchdog for 30–120 s). +- Reset-time source, in order: (1) parse `resets? .+ at` / `resets at HH:MM` from the matched tail; (2) poll-until-clear with an injectable clock + injectable `is_still_limited` probe (edge-triggered: `was_limited and not now_limited and not degraded`, OMC `daemon.ts:305`; 30 s per-probe timeout `:360`); (3) fallback existing backoff. No OAuth usage API (`rate-limit-monitor.ts:19-58`) dependency. +- Add `AgentStatus.PAUSED_RATE_LIMIT`, `STALLED`, `RESTARTED` (`_wrapper_models.py:15-23`); `LeadProcess` gains `pid_start_identity`, `nudges_used`, `paused_until`, real `pid` in tmux (`:26-37`). `cli.py:926` compares `== "completed"` (StrEnum-safe); handle new members before the generic else `:924-928` and BEFORE teardown `:930-960`. +- tmux coverage: same paused state, text from hoisted pane capture. +- Resume must be **verified**: require heartbeat delta or ledger delta within a bounded window after resume, else mark resume failed (OMC's `sendResumeSequence` `tmux-detector.ts:414-434` returns true unverified). +- Existing tests asserting retry semantics must be rewritten: `tests/unit/test_wrapper.py:504` `test_detects_rate_limit_and_backs_off`, `:525` `test_rate_limit_exhausts_retries`. + +### 1.7 PID + start-time identity + +- tmux: capture `#{pane_pid}` after pane creation (`:214-219`) — that is the shell pid (claude typed via send-keys `:239-242`); resolve claude child via `pgrep -P ` (one hop, poll until present during `_wait_for_tui_ready :304-350`); store `pid` + `process_start_identity` on `LeadProcess`. +- headless: `pid=proc.pid` `:466`; add identity at `:465-469`. +- Fix `surrogate.register_session` (`:240-268`) to also store `proc_start`; `sweep_locks :293-311` and `live_sessions :314-321` compare the tuple (recycled pid today keeps a dead lock alive → blocks consolidation `cli.py:763,949-958`). ruflo reconcile-on-every-load pattern (`swarm-tools.ts:107-149`) as hygiene; 24 h TTL only as legacy-record fallback. +- No psutil (`pyproject.toml:34`; adding it requires `uv.lock` regen or `.github/workflows/ci.yml:41` fails). Use `ps -o lstart=` (darwin) / `/proc` (linux, CI `ci.yml:26-28` ubuntu, py 3.11/3.12). + +### 1.8 Config threading + +- `ProjectConfig` (`project_config.py:28-51`) has no `model_config` → `watchdog:` block silently dropped, and `save_project_config :132-152` round-trips `model_dump()` (data loss). Add nested `watchdog: WatchdogConfig = WatchdogConfig()` (`enabled`, `poll_interval_sec`, `stall_threshold_min` — reconcile spec's 20 (`specs/watchdog.md:65`) vs check-11's 10-min stall, `nudge_delay_sec`, `nudge_budget`, `hard_max_restarts`, `rate_limit_probe_timeout_sec`); drop `tick_cron`/`reroute` from `specs/watchdog.md:60-70`. +- `ProjectContext.make_target` (`cli.py:75-87`) loads then discards ProjectConfig; add `make_project_config()` (None for legacy layout `cli.py:88`; legacy default = enabled with hardcoded defaults). Precedence via a `resolve_policy`-style merge (`experiment_loop.py:136-180`: CLI > plan > clamp > default). Thread `cli.py:1071` → `_launch_and_monitor` (new kwarg, sig `:709-731`) → `wait_for_completion :920`. `--low-token` preset (`cli.py:262-269`) gains a sessions cap key. + +### 1.9 Comms logging + +Five event types only (`comms.py:31-38`); do NOT add a sixth. Use `self._comms.log_checkpoint(agent="watchdog", phase="lifecycle", subtask="heartbeat|nudge|rate-limit-pause|rate-limit-resume|escalate", progress=..., blockers=[reason])` (`comms.py:381-416`) and `log_error(agent="watchdog", error_type="stall", severity="warning"|"blocking", description=..., escalated_to=...)` (`:344-379`, severity vocab `:61-67`). Both render live in `_print_status` (`cli.py:869-913`: checkpoint `:904`, error `:908`; `message` events NOT rendered — add branch only if needed). Files: `zo_root/logs/comms/.jsonl` under `fcntl.flock` (`comms.py:200-209`). Also `_trace`-style JSONL of tick decisions (`hookkit.py:81-96` shape) so check-11 can assert on a log line. Every escalation → `DecisionEntry` (`orchestrator.py:1279-1288` pattern) + `_record_learning` (`:1294-1302`). + +--- + +## 2. Fresh-context loop (check 13, Linux later) + +### 2.1 The seam +`cli.py:1134-1166`. Split `_launch_and_monitor` (`:709-960`) into `_launch_once` (`:798-939`: register-per-attempt `:789-794`, launch `:800`, wait `:920`, status print) and once-per-run teardown (`:942` end_session, `:944` semantic.close, `:953-957` deregister+consolidate; permissions overlay reclaim `:773-783` once). New driver `zo.driver.run_phase_loop(...)` (or `experiment_loop.run_fresh_context_loop`; keep `experiment_loop.py` pure — `__all__ :56` is decision-only, 16 pure tests) called from `build`: + +``` +while (phase := orch.get_current_phase()) is not None: + orch._refresh_gate_mode() # gate_mode may change mid-run (orchestrator.py:729) + zo_contracts.set_active_phase(memory_root, phase.phase_id) # contracts.py:172 — today only on GATED (orchestrator.py:789) + assert no RUNNING exp for phase (or _finalize/_abort first) # orchestrator.py:1388 mint side-effect + prompt = orch.build_lead_prompt(phase) + ledger digest section + process = _launch_once(prompt, ...) # headless claude -p, fresh context + if process.status in {STALLED, ...}: restarts += 1; check hard cap; continue + ev = orch.advance_phase(phase.phase_id) # FIRST runtime caller ever + if ev.requires_human / phase GATED: print nonce (prepare_gate_review orchestrator.py:902-904); break + if ev.decision == ITERATE and phase is phase_4: git checkpoint; iterations += 1; check caps; continue +``` +Break cleanly on mid-loop switch to SUPERVISED (`_auto_iterate_if_needed` returns None `:1258-1259`). `_consume_gate_decision` (`:403-427`) applies only at decompose — mid-loop approvals need an explicit re-check. + +### 2.2 Inputs & producers (all re-derived from disk) +- Ledger entry/digest: `load_ledger` `ledger.py:93`, `summarize :248`; written by `_emit_plan_ledger` `orchestrator.py:363-396` (merge-preserving `ledger.py:125-170`). New `_prompt_ledger_digest()` section in `build_lead_prompt` list `orchestrator.py:541-554` (iteration N-of-M banner). +- Experiment lineage: `ExperimentRegistry.lineage :233`, `render_checklist :287-321` (auto-refreshed via `_safe_refresh_checklist :343-354`); active/parent exp via `_ensure_experiment_for_phase :1124-1161` (idempotent on RUNNING `:1150-1152`; parent = `latest_in_phase :1154` regardless of status). +- Priors digest: `_prompt_memory :1784-1805` (priors[:8] `:1798`, semantic top_k=3 on `plan.objective` `:1801` — consider querying with current hypothesis). +- `next.md`/`diagnosis.md`/`result.md`: written by model-builder agent (`.claude/agents/model-builder.md:68-79`), read from disk per prompt at `orchestrator.py:1483-1492`. NOT produced by Python; `parse_next_md :650`/`parse_hypothesis_md :636` have zero prod callers → `Experiment.hypothesis`/`next_ideas` always empty → DEAD_END (`experiment_loop.py:303-319`, `check_dead_end :395`) is dead code. Wire both parsers into `_finalize_experiments :1163` (`:1220-1226`). +- Ledger write-through per iteration: `record_attempt` at iteration start (`ledger.py:227`), only oracle path calls `mark_phase_passed :184`; `reset_phase :203` on CONTINUE (`orchestrator.py:1312-1315`) — `attempts` is the only monotonic per-subtask signal. + +### 2.3 Spawn mechanism +Reuse `_launch_headless` (`wrapper.py:414-461`: `[claude, --print, --output-format, json, --model, M, --max-turns, N, --add-dir, cwd]` `:427-441`, `_resolve_claude_bin :904`, env `os.environ.copy()+extra_env :456-459`, Popen to log files `:461-463`, LeadProcess w/ pid `:465-469`). Improvements from ruflo `headless-worker-executor.ts:1397-1431`: prompt on **stdin** (`child.stdin.end(prompt)`; argv `-p` at `wrapper.py:450` risks tokenization/ARG_MAX), `start_new_session=True` + kill process group (SIGTERM → SIGKILL after 5 s) because `claude --print` spawns grandchildren; hard per-iteration timeout. `--dangerously-skip-permissions` only works with `--print` (PRIORS PR-001 `memory/zo-platform/PRIORS.md:16-23`) — confirm `permissions_overlay.py` + sealed-paths hook are honored in `--print` mode (open Q). Do NOT copy `_generate_session_summary` (`cli.py:686-696`: bare `"claude"`, no env/cwd, `except Exception: pass`). `cwd=zo_root` + `ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_REPO_ROOT` mandatory (§1.2 shim constraint; sealing depends on it). + +### 2.4 Git checkpoint +`surrogate.commit_worktree(path, *, message)` (`:360`, `add -A` + commit, True only if a commit was made) over `_git :97-101` — the only git subprocess code in ZO (precedent `consolidate.py:174-182`; `orchestrator.py`, `cli.py`, `scaffold.py` have none). Call after `advance_phase` returns ITERATE (i.e., after `reset_phase` at `orchestrator.py:1312`), message `zo: checkpoint {exp.id} ({verdict})`; then update `SessionState.git_head` (`_memory_models.py:40`, verified by `recover_session` `memory.py:287-301`) so the fresh session doesn't raise a git_head mismatch blocker. Scope question: `add -A` may commit weights (delivery `.gitignore` scaffolded `scaffold.py:227`) — prefer `.zo/experiments/` + artifacts allowlist. Fix `Experiment.artifacts_dir` absolute path (`experiments.py:491`, docstring `:176-178` says relative; consumers `orchestrator.py:1203`, `experiments.py:448,455`) — a check-13 blocker if `registry.json` is committed and checked out on the Linux box. + +### 2.5 Caps +`LoopPolicy` (`experiment_loop.py:113-118`: max_iterations 10, plateau_epsilon .01, plateau_runs 3, stop_on_tier must_pass, dead_end .9), low-token clamps `:130-133` applied in `resolve_policy :166-177` (CLI > plan > clamp > default `:149-150`); budget enforced at `:261` by counting COMPLETE exps (durable). Driver MUST also enforce max_iterations + `hard_max_restarts` as a session counter (a session that never reaches a gate never consults the evaluator). Never mutate the returned policy — `resolve_policy` returns the shared `DEFAULT_POLICY` singleton (`:178-179`, model not frozen `:120`). Plateau requires exact metric-name match (`experiments.py:515-525`) and `len(deltas)==plateau_runs` (`experiment_loop.py:283`) — log missing deltas. + +### 2.6 Prompt text changes +- `.claude/agents/lead-orchestrator.md`: no Phase-4 loop content (grep hits only `:23,:175,:216,:230`). Add: one-iteration-per-session rule; `specs/watchdog.md:77` liveness-by-evidence rule; STATE.md → plan-ledger.json at `:101,:213,:230`. +- `.claude/agents/model-builder.md:86-110` already disk-based; add "no prior-iteration context survives"; reconcile `:290` (escalate after 2 non-improving iterations) with loop-owned plateau (`experiment_loop.py:274-296`). +- `orchestrator.py` mirrors: `_render_loop_briefing :1456-1519`, `_prompt_experiment_context :1378-1454`, `_prompt_coordination :1809-1838` (add nothing about heartbeats — hook-driven by design). + +### 2.7 Testable on Mac vs Linux-only +- Mac (mock spawn): driver loop with a fake `launch_lead_session`/`_launch_once` returning scripted `LeadProcess` statuses; assert `advance_phase` invoked (first runtime caller — wiring test), restart cap, gate break, git checkpoint via `commit_worktree` on a tmp git repo, prompt digest content, ledger `record_attempt` per iteration, `set_active_phase` per iteration. Reuse `tests/integration/test_auto_iteration.py:154-162` `_run_iteration` as the executable contract. +- Linux-only: real `claude -p` spawn, `--print` permissions behavior, check 13 (GPU demo ≥ 91.62%, cost ≤ 1.15×; `plans/zo-v2-rearchitecture.md:60,114-115`). Preflight `_check_claude_cli` (`preflight.py:84-89`, wired `:70`) is `shutil.which` only — strengthen with `claude --version` (pattern `_check_docker :187-199`). Land loop behind an off-by-default flag; STATE.md records "11-12 green, 13 pending". + +--- + +## 3. Deferrals + +### 3.1 `evaluate_loop_state` ledger input +Signature `experiment_loop.py:205-209` `(registry, phase, policy=None)`; body reads only `registry.experiments` `:230-233`. **Add keyword-only `*, ledger: LedgerFile | None = None`** appended after `policy`. Call-site count: **16 in `tests/unit/test_experiment_loop.py`** (lines 102,111,122,137,145,150,159,176,185,196,218,232,243,259,312,328; 12 pass policy positionally) + 1 prod (`orchestrator.py:1276`, lazy-import block `:1265-1269`) + `test_auto_iteration.py:162` indirect. Zero churn. At `:1276` pass `ledger=zo_ledger.load_ledger(self._memory.memory_root / LEDGER_FILENAME)` (`ledger.py:93` fail-open None; `LEDGER_FILENAME :51`). Ledger has no oracle_tier (`LedgerFile :67-74`, `LedgerEntry :54-64`) — ledger contributes `phase_status`/`passes`/`attempts` only; registry stays for tiers. Add one test: ledger=None → identical verdict. + +### 3.2 Session-restore cutover (STATE.md → plan-ledger.json) +- **Precedence: ledger > STATE.md, per-phase, presence-based**; comms warning on per-phase mismatch. Preserve GATED > ACTIVE > PENDING-deps-met, BLOCKED excluded — lives in `get_current_phase` `orchestrator.py:687-727` (`:709-712`, `:713-716`, `:717-726`, docstring `:704-705`) reading only `phase.status` → **no change**; nine `TestGetCurrentPhase` tests (`tests/unit/test_orchestrator.py:463-566`) are source-agnostic and remain the oracle. Rationale `PRIORS.md:1089-1150` (PR-037 rules `:1125,:1143,:1148`). +- **PR-036 preserved by moving validation**: reuse `_VALID_PHASE_STATUSES` (`_memory_formats.py:30-32`; enforcement `:154-168`, error shape `:158-165`) in a validated `LedgerFile.phase_status` field (`ledger.py:73`) and in `set_phase_status :239-245`; strict-load variant for restore: file exists + parse fails → raise with path/phase/value/valid list; file absent → fall back to STATE.md (`load_ledger :93-98` fail-open stays for hooks). Drift-guard test twin of `tests/unit/test_memory.py:156`. Note `memory.read_state` swallows `(ValueError, KeyError)` at `memory.py:139` (untested; PR-036 error never reaches operators) — fix on the restore path. +- **Schema gap**: no per-subtask completion in ledger (`passes` phase-wide, `mark_phase_passed :184-200`; `record_attempt :227-236` ≠ complete). Add `LedgerEntry.completed: bool` + `mark_subtask_completed(memory_root, phase_id, subtask)`, called from `mark_subtask_complete` `orchestrator.py:858` next to `record_attempt`; `reset_phase :203-213` must clear it (mirrors `completed_subtasks.clear()` `orchestrator.py:953,1311`). Guard: `tests/unit/test_orchestrator.py:1049 test_partial_progress_restored`. Keep oracle-owned `passes` separate (anti-Goodhart). +- **Cutover site**: `_restore_phase_states` `orchestrator.py:468-481` (`:478` is the PR-036 traceback line — keep coercion after boundary validation). Also `orchestrator.py:359-360` (session `phase` pointer gated on STATE.md `phase_states` emptiness) and `start_session :237-256`. +- **Write-through audit** (writers): present at `orchestrator.py:792` (GATED), `:829` (COMPLETED), `:946` (PROCEED), `:954-956` (ITERATE), `:960` (ESCALATE/blocked), `:1312-1315` (loop CONTINUE); **missing** at `:962` (HOLD → GATED; add `_ledger_safe("set_phase_status", pid, "gated")`) and `:476-481` (restore, no write-back). `_ledger_safe :398-401` discards `_mutate`'s False (`ledger.py:173-181`) — log it. Ordering hazard: `decompose_plan :350-353` runs `_consume_gate_decision :351` before `_emit_plan_ledger :353` — first-run gate decision drops into `doc is None` (`ledger.py:176-178`); reorder or ensure ledger exists. `emit_ledger :144-148` prev_status always wins → after cutover STATE.md hand-edits are inert (open Q). +- STATE.md keeps being written (`_capture_phase_states :293-303` from `end_session :284`) as human projection — required by `hookkit.py:236,:260`, `cli.py:2303-2309` (`zo status` exits 1 without STATE.md — relax to either file), `cli.py:1081-1082` (mode derivation from `state.phase`). +- Tests: keep `test_real_resume_via_state_md_round_trip` (`test_orchestrator.py:568-605`) + add ledger twin + conflict test (ledger wins). Two-orchestrators-one-MemoryManager pattern `:940-987`. Also SKIPPED never written / `pending` never re-set (`_orchestrator_models.py:36`; `ledger.py:198,211,239`); subtask_id slug collision `ledger.py:153,77-78`. + +--- + +## 4. Reusable existing code (deduped) +- `ledger._atomic_write` `ledger.py:81-91`, `_mutate :173-181`, `load_ledger :93-98`, `summarize :248-254`, `emit_ledger` merge `:136-148`, `record_attempt :227`, `reset_phase :203`, `mark_phase_passed :184`, `set_phase_status :239`. +- `contracts.emit_contracts` atomic pattern `contracts.py:151-161`, `load_contracts :164-169`, `set_active_phase :172`. +- `hookkit._memory_root :64-69`, `_repo_root :60`, `_trace :81-96`, `main :392-407`, `_agent_name :112-117`, `_last_assistant_text :153`, `_sealed_prefixes :314-348`, `_SEALED_DEFAULTS :47-50`, `_handle_post_tool_failure :287-308`. +- `wrapper._check_gate_mode_change :648-667` (per-poll file-watch, log-on-change), `_maybe_open_training_pane :588-634`, debounce consts `:74-81` + docstring `:677-701`, `_launch_headless :414-461`, `_resolve_claude_bin :904`, `_capture_tmux_pane :998`, `_read_tail :1029`, `_tmux_pane_alive :920`, `_tmux_claude_running :934`, paste/submit `:255-270` + `_verify_prompt_submitted :364-398`, `kill_session :824-861`, `monitor_team :481-497` / `read_task_list :499-512` / `_read_team_config :1011-1026` (secondary signal only; `is_active` True when `tasks_total==0` `:496`). +- `surrogate._pid_alive :279-290`, `register_session :240-268`, `sweep_locks :293-311`, `live_sessions :314-321`, `_git :97-101`, `commit_worktree :360`. +- `comms.log_checkpoint :381-416`, `log_error :344-379`, `_write_event` flock `:200-209`; `_print_status` renderer `cli.py:869-913`. +- `experiment_loop.LoopDecision :183-202`, verdict cascade `:227-283`, `resolve_policy :136-180`; `orchestrator._ledger_safe :398-401`, `_record_learning :1323`, DecisionEntry-per-verdict `:1279-1288`, `_refresh_gate_mode :729`; `experiments.render_checklist :287-321`, `lineage :233`, `save_registry :269-279`. +- `training_display._time_ago :92-105`; `memory.write_state :142-161`, `_append_locked :178`, `_get_git_head :303`; `preflight._check_docker :187-199`; `cli.py:2317-2352` ledger-first status table; `cli.py:2912-2967` `watch-training` external poll loop; `_peers_live` fail-open guard `cli.py:747-796`. +- Test harness: `tests/unit/test_wrapper.py:28-46` fixtures, `:543-641` class-patched staticmethods + `side_effect` scripts + `timeout=-1`, `:504-541` headless harness, `:729-744` parametrized classifier; `tests/unit/test_hookkit.py:31-36 _run`, `:39-58`; `tests/integration/test_hooks_shim.py:29-38 _run_shim`, `:41-66 contracts_env`, `:113-146 TestSettingsWiring`; `tests/unit/test_ledger.py:36-52 _workflow`, `:140-157` sealed deny; `tests/unit/test_orchestrator.py:53-72` fixtures; `tests/unit/test_experiment_loop.py:62-92 _exp/_registry`; `tests/integration/test_auto_iteration.py:154-162`. +- Reference ports (MIT): OMC `types.ts:103`, `heartbeat.ts:19-81`, `idle-nudge.ts:29-131`, `persistent-mode/index.ts:1479-1607, 2268-2373, 1003`, `todo-continuation/index.ts:141,295,370,390,442`, `tmux-detector.ts:34,70,74,153,376`, `daemon.ts:305,345,360,405-433`, `team-owner-epoch.ts:29-148`, `security-config.ts:44-151`; ruflo `swarm-tools.ts:80-149`, `repo-supervisor.ts:20-124`, `headless-worker-executor.ts:1378-1431`; ralph `ralph.sh:84-113`, `CLAUDE.md:7-104`. + +--- + +## 5. Hazards & conflicts between mappers (resolved) +| Conflict | Resolution (verified by grep 2026-08-17) | +|---|---| +| `_check_gate_mode_change()` in `_wait_tmux` at 710 (m1,m3,m6) vs 723 (m5) | **710** (`_maybe_open_training_pane()` 711); `_wait_headless` twin at 777 | +| `_maybe_open_training_pane` def 588 vs 592; `_capture_tmux_pane` def 997 vs 998; `poll_interval` 551 vs 554 | **588 / 998 / 551** (`timeout` 552) | +| cli teardown: `end_session` 942 vs 947; deregister 949-958 vs 953-957 | **942**; deregister **953**, consolidate **955-957**; launch **800**, wait **920** | +| Rate-limit sleep 806 vs 807, retries 807 vs 808 | **807 / 808**; RATE_LIMITED 795, `_detect_rate_limit` def 894, `_backoff_wait` 898-900 | +| `self._proc` init hazard (m1) | Confirmed: only assigned at 287 and 474; `__init__` 83-99 has none | +| psutil acceptable (m2 suggests) vs absent (m1,m5,m6) | Absent (`pyproject.toml:34`); use `ps`/`/proc` | +| `evaluate_loop_state` test sites: 15+ / ~20 (DECISION_LOG:1277) / 16 | **16**, all in `test_experiment_loop.py` | +| PID identity attributed to ruflo (digest) | Wrong — OMC `team-owner-epoch.ts:69-148`; ruflo has signal-0 only | +| OMC fresh-spawn per phase | Does not exist; autopilot blocks Stop in-session (`persistent-mode/index.ts:2556-2564`); fresh-spawn precedents are ralph.sh:95 + ruflo headless executor | +| Heartbeat root git status | `memory/zo-platform/heartbeats/` TRACKED (`.gitignore:33-34`); only `logs/` ignored (`:27`) — fix .gitignore first | +| `DECISION_LOG.md:1265` "PostToolUseFailure did not fire for nonzero-exit Bash" | Contradicted by `logs/comms/failures-2026-08-17.jsonl` — correct the entry | +| specs/watchdog.md cron/orchestrator-owned tick (`:40,:74-79`) vs plan | Plan wins (`plans/zo-v2-rearchitecture.md:107-114`); rewrite spec in-PR | +| `_agent_name` "verified against live payloads" (STATE.md:11) | Only for SubagentStop/PostToolUseFailure; PreToolUse 34/34 no identity → sealed-paths off-limits branch (`hookkit.py:351-368`) likely never fired in prod | +| Heartbeat freshness alone = liveness (OMC `heartbeat.ts:81`) | Insufficient — no write during long tool calls (`mcp-team-bridge.ts:746`); require stale AND (dead OR no progress) | +| Sentinel completion grep (ralph.sh:99) | Unsound; use ledger predicate + per-subtask attempt cap | +| `Experiment.artifacts_dir` absolute path | Check-13 blocker if registry.json crosses machines | + +Additional hazards to carry: `_wait_tmux` `timeout` includes paused time (`:754`); `on_status`-gated pane capture (`:749-752`); tmux nudge clobbers global paste buffer (`:255-258`); `_ensure_experiment_for_phase` mint side effect on prompt build (`orchestrator.py:1388`); STATE.md flushed only at `end_session` (`:284`) — ledger is the crash-safe source; `apply_human_decision` nonce single-use (`:928-942`) — driver must break on GATED; `hookkit._trace` cwd fallback lets tests write into real repo (`test_hookkit.py:353-359` `explode` pollution) — new tests must `monkeypatch.setenv` both `ZO_REPO_ROOT` and `ZO_MEMORY_ROOT`; PostToolUse `*` hook cost (bash+python+pydantic import per tool call). + +--- + +## 6. Test plan skeleton +Conventions: module docstring naming workstream + checks (`tests/unit/test_ledger.py:1-6`, `test_contracts.py:1-4`), two-tests-per-mechanism rule (`test_hookkit.py:1-8`), section comments `# ---- (oracle check N) ----` (`test_hookkit.py:61,108,186,245,274`), class docstring stating both halves (`test_ledger.py:137-138`), test names containing "seeded" (`test_hookkit.py:65,139,278`; `test_ledger.py:143`; `test_plan.py:751`). + +**`tests/unit/test_watchdog.py`** — "Tests for zo.watchdog — the WS-C execution substrate (plan oracle checks 11-12)." +- `# ---- never-block taxonomy (oracle check 11, negative half) ----`: parametrized positives per category (pattern `test_wrapper.py:729-744`) + negatives (`0.4291`, `step 4290`, git log line "fix weekly report", cat'ed old transcript) + `is_interrupt` → user_abort. +- `# ---- stall predicate ----`: three-state freshness; unknown → no verdict; stale+dead → stall; stale+progress-delta → no stall; startup grace. +- `# ---- process identity ----`: mocked `subprocess.run` for `ps`/`/proc` read; darwin usec wildcard; malformed → not dead; EPERM alive. +- `# ---- nudge budget ----`: dwell, max 3, throttle, timer reset, persisted budget survives re-instantiation, exhaustion → STALLED. +- `# ---- rate-limit pause/resume (oracle check 12) ----`: injectable clock + probe; edge-triggered resume; degraded ≠ clear; resume verified by heartbeat delta; timeout excludes pause. + +**`tests/unit/test_wrapper.py` additions** (fixtures `:28-46`; `@mock.patch("zo.wrapper.time.sleep")`; class-patch `_tmux_pane_alive/_tmux_claude_running/_capture_tmux_pane/_kill_tmux_window`; instance-patch `monitor_team`; `timeout=-1` or scripted dead sequence; call_count assertions): +- **Seeded stall (check 11)** on `_wait_tmux`: heartbeat file mtime aged via injected clock, pane text neutral → nudge called ≤3 → STALLED within one poll iteration after threshold; assert comms `log_error(error_type="stall")` line. +- **Rate-limited NOT nudged (check 11)** on `_wait_tmux`: pane text contains OMC rate-limit banner → nudge mock never called, status PAUSED, then resume on probe flip. +- Same pair on `_wait_headless` (harness `:504-541`); rewrite `:504` and `:525`. +- Watchdog fires on the `:745` continue path; `_proc` default None; single pane capture per poll shared with `on_status`. + +**`tests/unit/test_hookkit.py` additions**: `heartbeat` handler writes atomic JSON keyed by `agent_id`/`session_id`, `compacting` on precompact, fail-open with unset memory root; `_handle_post_tool_failure` record carries `is_interrupt/agent_id/agent_type/never_block_reason`; heartbeats sealed (`test_ledger.py:140-157` pattern). + +**`tests/integration/test_hooks_shim.py`**: extend `TestSettingsWiring.test_all_ws_a_events_wired` (`:117-146`) with `heartbeat` on PostToolUse; **new wiring test** asserting `_watchdog_tick` is invoked from both poll loops (grep-free: patch and assert called). `test_hooks_shim.py::_run_shim` (`:29-38`) end-to-end heartbeat write with `ZO_MEMORY_ROOT` in tmp. + +**Driver / fresh-context** (`tests/integration/test_fresh_context_loop.py`): fake `_launch_once`; assert `advance_phase` called (first runtime caller), restart cap, GATED break with nonce, git checkpoint on tmp repo via `commit_worktree`, `set_active_phase` per iteration, `record_attempt` per iteration, no RUNNING exp reuse (`orchestrator.py:1388` guard), gate-mode re-read. + +**Deferrals**: `evaluate_loop_state(..., ledger=None)` unchanged verdict + ledger-aware case; ledger validator PR-036 triple (`test_memory.py:156,163,184` twins), strict-load raises on corrupt existing file, ledger round-trip resume twin of `test_orchestrator.py:568-605`, ledger-wins conflict, HOLD write-through, `mark_subtask_completed`/`reset_phase` clear, first-run gate decision lands in ledger, `zo status` without STATE.md. + +**Config**: `WatchdogConfig` round-trip through `save_project_config`, legacy default, precedence. + +--- + +## 7. Docs/memory cascade +- `specs/watchdog.md`: `:3` Status RFC→implemented; `:39-40` §3.1 cron→poll loop; `:53-58` §3.4; `:60-70` §3.5 config keys; `:74-79` §4 → `wrapper.py` `_wait_tmux :669`/`_wait_headless :763`, delete `:79` monitor-agent line; `:83-88` §5 → checks 11-12; `:92-93` "spec-only" false. Reconcile 20-min default (`:65`) vs 10-min check. +- `specs/workflow.md:547` Subtask 4.3 fresh-context semantics; `specs/memory.md:286,:291`; `specs/comms.md:48` untouched (no new event type); `specs/agents.md` untouched (no new agent). +- `plans/zo-v2-rearchitecture.md:67,:132` and `docs/reference/v2-rearchitecture.mdx:99` "854" (real count 917 → post-Phase-3 recount); mdx `:65-66` no Status column (decide: add to all five tables `:44,:63,:78,:92` or prose `:87-103`); `docs/roadmap.mdx:30`; `docs/COMMANDS.md:13` only if `zo watchdog` ships; `docs/cli/build.mdx:133` if headless full builds become supported. +- `README.md:13` badge (`tests-854`), `:529` ("780 platform tests"), `:307` slash count only if a `.claude/commands/*.md` is added (validate-docs Check 3 HARD FAIL `scripts/validate-docs.sh:110-129` with `STATE.md:65`). Checks 1/2/7 (`:46-103,:215-222`) fire only on new agent file; Check 4 (`:136-148`) on version bump; Check 6 (`:192-208`) warn-only, already tripping (diff 63); Check 8 inert locally (`:244`). +- `.gitignore` (heartbeats/, contracts.json, plan-ledger.json under memory/zo-platform); `contracts.py:10` docstring; `.claude/agents/lead-orchestrator.md:101,:213,:230` + Phase-4 section; `.claude/agents/model-builder.md:86-110,:290`; `pyproject.toml`/`uv.lock` only if a dep is added. +- Memory: `memory/zo-platform/STATE.md:9-11` (prepend Session 041 "pick up here", demote 040; front-matter `:3-7`); `DECISION_LOG.md` append EOF (`:7-14` template): watchdog RFC-vs-plan divergence, `evaluate_loop_state` keyword-only choice, mdx status-column decision, correction of `:1265`; `PRIORS.md` only on real failure (`:1057-1101` template, "+ 7 skipped" = `tests/unit/test_semantic.py:451`, not e2e — `tests/e2e/` has zero .py); `memory/zo-platform/sessions/session-041-.md` (`session-040-2026-08-12.md:1-3,:92` shape, mandatory "## Next session — pick up here"). + +--- + +## 8. Open questions for Sam (build-changing only) +1. **tmux nudge semantics mid-turn**: `paste-buffer`+Enter into a BUSY Claude TUI (only launch-time uses exist, `wrapper.py:255-270,386-398`) — queued as next turn, swallowed, or interrupt? Determines whether nudges are safe to repeat (budget 3) or must be single-shot → restart. +2. **Heartbeat write source**: is PostToolUse `*` acceptable (process spawn per tool call), or must a second source (pane-text delta / comms mtime) cover thinking / long single tool calls? And does PostToolUse carry `agent_id` (unverified — trace-only first commit resolves it). +3. **Rate-limit reset time**: is a `resets at` timestamp visible in the TUI banner / stderr, or does check 12 accept poll-until-clear with injectable probe (deterministic in tests, no OAuth API)? +4. **Heartbeat root**: `ZO_MEMORY_ROOT` = delivery `.zo/memory` (`cli.py:69,1076`) vs platform `zo_root` for comms/hook-trace (`cli.py:1096`, `hookkit.py:94`). One root or both threaded through `wait_for_completion`? +5. **Driver ownership + scope**: new `zo.driver` module vs orchestrator method; wire the FULL `advance_phase` gate path (`orchestrator.py:797,830-832` pytest/notebook/snapshot per iteration) or only the phase_4 loop branch? And how do subtasks get marked complete in prod (`mark_subtask_complete :848` has no caller → `advance_phase` always ITERATE "Subtasks remaining" `:838-844`)? +6. **Per-agent vs lead-only liveness**: wrapper tracks one `LeadProcess`; teammates are files only (`monitor_team :481-497`, no pids). Phase 3 = lead process identity + hook heartbeats for teammates? +7. **STATE.md hand-edit semantics after cutover**: ledger wins silently / with warning / refuse-to-start? (prod-001 fix workflow in PR-036/037 edits STATE.md.) +8. **`--dangerously-skip-permissions` for fresh builders**: is `permissions_overlay.py` + sealed-paths honored under `--print`? If not, the fresh loop voids Phase-1 enforcement. +9. **Linux demo mode for check 13**: headless end-to-end (contradicts `docs/cli/build.mdx:133`) or tmux lead spawning headless children? Also: is registry.json/absolute `artifacts_dir` crossing machines (fix becomes a blocker)? +10. **Git checkpoint scope**: whole delivery tree (`add -A`) or `.zo/experiments/` + artifacts allowlist? \ No newline at end of file diff --git a/memory/zo-platform/research/2026-08-17-phase3-recon/pr-a-build-contract.md b/memory/zo-platform/research/2026-08-17-phase3-recon/pr-a-build-contract.md new file mode 100644 index 0000000..bc443d3 --- /dev/null +++ b/memory/zo-platform/research/2026-08-17-phase3-recon/pr-a-build-contract.md @@ -0,0 +1,303 @@ +# PR-A build contract — Watchdog (v2 Phase 3 / WS-C, oracle checks 11–12) + +Companion to `integration-map.md` (same directory). The map says WHERE; this +contract says WHAT. Builders follow this contract exactly; the map supplies the +exact `file:line` anchors. Where the two disagree, this contract wins. + +Ground rules (from CLAUDE.md + plan anti-scope): Python 3.11+, PEP8, type hints, +Google docstrings, files < 500 lines, functions < 50 lines, ruff clean on +`src/`; every mechanism ships WIRED with a seeded-failure test; fail-open for +advisory paths (heartbeats, logging), never for control decisions; no psutil +(stdlib + `ps` / `/proc`); no new comms event types; control-plane files under +the existing per-project memory root only. Ported OMC code carries an MIT +attribution comment (`oh-my-claudecode/LICENSE`, © 2025 Yeachan Heo). + +--- + +## 0. File ownership (disjoint per builder) + +| Builder | Owns (create/edit) | Must not touch | +|---|---|---| +| **core** | `src/zo/watchdog.py` (new), `tests/unit/test_watchdog.py` (new) | everything else | +| **wrapper** | `src/zo/wrapper.py`, `src/zo/_wrapper_models.py`, `tests/unit/test_wrapper.py` | hookkit, cli, config | +| **hooks** | `src/zo/hookkit.py`, `.claude/hooks/zo-hookkit.sh`, `.claude/settings.json`, `.gitignore`, `tests/unit/test_hookkit.py`, `tests/integration/test_hooks_shim.py` | wrapper, cli, config | +| **config-cli-docs** | `src/zo/project_config.py`, `src/zo/cli.py`, `tests/unit/test_project_config.py`, `tests/unit/test_cli.py`, `specs/watchdog.md`, `docs/reference/v2-rearchitecture.mdx`, `docs/roadmap.mdx` (only if it names the watchdog as pending), `plans/zo-v2-rearchitecture.md` (check-11 wording only if needed) | wrapper, hookkit, watchdog.py | +| **integrator** (later) | anything, to reconcile | — | + +`src/zo/watchdog.py` is the shared dependency: builders wrapper/hooks/config +import it and MUST use the API below verbatim (names, signatures, semantics). + +--- + +## 1. `zo.watchdog` public API (builder: core) + +Module docstring: "Watchdog — WS-C execution substrate (plan oracle checks +11-12). Pure logic: no I/O in the classifier and predicate paths; process +identity and file helpers are small, injectable, and fail-open." + +```python +SCHEMA_VERSION = 1 +HEARTBEATS_DIRNAME = "heartbeats" # /heartbeats/ +WATCHDOG_STATE_FILENAME = "_watchdog.json" # /heartbeats/_watchdog.json +HEARTBEAT_STALE_SWEEP_SEC = 24 * 3600 + +class HeartbeatStatus(StrEnum): + READY = "ready" # turn ended / idle at prompt (Stop hook) + EXECUTING = "executing" # tool activity (PostToolUse) + COMPACTING = "compacting" # PreCompact + SHUTDOWN = "shutdown" # SubagentStop / SessionEnd for that key + +class HeartbeatRecord(BaseModel): + schema_version: int = SCHEMA_VERSION + agent_key: str # filename stem: agent_id or f"lead-{session_id}" + agent_id: str | None = None + agent_type: str | None = None + session_id: str # Claude Code session_id from the hook payload + zo_session_id: str | None = None # from env ZO_SESSION_ID (comms correlation) + pid: int | None = None + process_start_identity: str | None = None + last_tick_at: datetime # tz-aware UTC + status: HeartbeatStatus = HeartbeatStatus.EXECUTING + last_event: str = "" # hook_event_name or tool_name + tick_count: int = 0 + +def heartbeat_path(memory_root: Path, agent_key: str) -> Path +def load_heartbeat(path: Path) -> HeartbeatRecord | None # fail-open (None on any error) +def load_all_heartbeats(memory_root: Path) -> list[HeartbeatRecord] # ignores _watchdog.json + unparsable +def write_heartbeat(memory_root: Path, record: HeartbeatRecord) -> Path # atomic tmp+os.replace; mkdir -p +def sweep_stale_heartbeats(memory_root: Path, *, now: datetime, older_than_sec: int = HEARTBEAT_STALE_SWEEP_SEC) -> int + +class Freshness(StrEnum): FRESH = "fresh"; STALE = "stale"; UNKNOWN = "unknown" +def classify_freshness(record: HeartbeatRecord | None, *, now: datetime, stale_after_sec: float) -> Freshness + # None → UNKNOWN; naive datetimes → treat as UTC; UNKNOWN is never a stall verdict. + +class NeverBlockReason(StrEnum): + USER_ABORT = "user_abort"; CONTEXT_LIMIT = "context_limit"; RATE_LIMIT = "rate_limit" + AUTH_ERROR = "auth_error"; AWAITING_INPUT = "awaiting_input"; COMPACTING = "compacting" + +def normalize_terminal_text(text: str) -> str + # strip ANSI escapes and \r; drop lines matching GIT_OUTPUT_LINE_PATTERNS + # (git log/diff/commit lines: r"^(commit [0-9a-f]{7,}|Author:|Date:|diff --git|index [0-9a-f]+\.\.|@@ |[-+]{3} [ab]/)"); + # drop lines that are saved-transcript commands (cat|bat|less|more|tail|head ... transcript|output|hud|.txt); + # lowercase is NOT applied here (callers use re.I). + +def progress_digest(text: str) -> str + # sha1 of normalize_terminal_text(text) with volatile UI lines removed: + # spinner glyphs [⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏·✻✽✶✳✢], "esc to interrupt", elapsed counters r"\(\d+[smh]\b.*?\)", + # r"\b\d+\s*tokens?\b", r"\b\d+[smh]\s+elapsed\b", "? for shortcuts", the idle prompt line, blank lines. + +RATE_LIMIT_TEXT_PATTERNS: tuple[re.Pattern[str], ...] # port of OMC tmux-detector RATE_LIMIT_PATTERNS + + # OMC_HUD_RATE_LIMIT_SCREEN_PATTERNS: r"rate limit", r"usage limit", r"quota exceeded", r"too many requests", + # r"try again later", r"limit reached", r"hit your limit", r"hit .+ limit", r"resets? .+ at", r"5[- ]?hour", + # r"\bweekly\s+(?:usage\s+)?(?:limit|quota|cap|allowance|allocation)\b", + # r"you(?:'|’)ve\s+(?:hit|reached)\s+(?:your\s+)?(?:session\s+|usage\s+)?limit", r"\blimit\s+resets?\b", + # r"stop\s+and\s+wait\s+for\s+limit\s+to\s+reset", r"\b429\b(?!\d)(?=.*(?:rate|limit|request))" , + # r"rate_limit(?:ed|_error)?", r"too_many_requests", r"quota_(?:exceeded|limit|exhausted)" + # NOTE: NO bare r"429" and NO bare r"overloaded" (false positives: 0.4291, step 4290, "GPU overloaded"). +CONTEXT_LIMIT_PATTERNS # OMC #213 tokens: context_limit, context_window, context_exceeded, context_full, + # max_context, token_limit, max_tokens, conversation_too_long, input_too_long, plus TUI text: + # r"context (?:window )?(?:is )?(?:full|low|exceeded)", r"prompt is too long", r"compact(?:ing)? (?:the )?conversation" +AUTH_ERROR_PATTERNS # OMC #1308 — the 16 tokens verbatim (authentication_error … insufficient_scope) as + # word-bounded matches; plus r"\bplease (?:run )?/login\b", r"\bnot logged in\b", r"\binvalid api key\b". + # NOTE: '401'/'403' only when adjacent to auth vocabulary: r"\b40[13]\b(?=.*(?:unauthori[sz]ed|forbidden|auth))". +USER_ABORT_PATTERNS # exact tokens aborted|abort|cancel (word-bounded) + substrings user_cancel, user_interrupt, + # ctrl_c, manual_stop, "^\s*interrupted by user"; bare "interrupt" is deliberately EXCLUDED (OMC #2478). +AWAITING_INPUT_PATTERNS # permission/question dialogs — never send Enter into these: + # r"do you want to (?:proceed|allow|continue)", r"^\s*❯?\s*\d+\.\s", r"\[\d+\]", r"esc to cancel", + # r"yes,? (?:and )?(?:don't|do not) ask again", r"allow (?:once|always)", r"press enter", r"enter to confirm", + # r"select an option", r"choice:", r"waiting for (?:your )?(?:input|response|approval)", r"trust (?:this|the) (?:folder|workspace)" + +def classify_never_block(text: str, *, is_interrupt: bool | None = None, + heartbeats: Sequence[HeartbeatRecord] = (), now: datetime | None = None, + compacting_window_sec: float = 300.0) -> NeverBlockReason | None + # Precedence: is_interrupt → USER_ABORT; then over normalize_terminal_text(text) restricted to the LAST 60 + # non-empty lines: USER_ABORT > CONTEXT_LIMIT > RATE_LIMIT > AUTH_ERROR > AWAITING_INPUT; + # then COMPACTING if any heartbeat has status COMPACTING with last_tick_at within compacting_window_sec of now. + # Rationale (OMC persistent-mode bypass order): the platform's own stops win over "waiting for the user". + +def parse_rate_limit_reset(text: str, *, now: datetime, tz: tzinfo | None = None) -> datetime | None + # Recognize: "resets? (?:at )?(\d{1,2}(?::\d{2})?\s*(?:am|pm)?)" (today; if already past → tomorrow), + # "reset(?:s)? in (\d+)\s*(min|minute|hour|second)s?", "try again in N (s|sec|min|minutes|hours)", + # "retry[- ]after[: ]+(\d+)" (seconds), ISO-8601 timestamps. Local tz = `tz or now.tzinfo`. None if absent. + +def compute_pause_until(now: datetime, reset_at: datetime | None, *, attempt: int, config: "WatchdogConfig") -> datetime + # reset_at + 15 s jitter-free slack if given; else now + min(config.rate_limit_backoff_base_sec * 2**attempt, + # config.rate_limit_backoff_max_sec). + +def pane_ready_for_nudge(text: str) -> bool + # OMC paneLooksReady AND NOT paneHasActiveTask AND no AWAITING_INPUT pattern in the last 40 lines: + # ready := last non-empty line matches r"^\s*(?:[│┃║▌▐▏▕╎┆┊]\s*)?[›>❯]\s*" (idle prompt) or any such line + # exists in the last 5 non-empty lines; active := "esc to interrupt" | "background terminal running" | + # r"^[·✻]\s+[A-Za-z][A-Za-z0-9''-]*(?:\s+[A-Za-z][A-Za-z0-9''-]*){0,3}(?:…|\.{3})$" in last 40 lines. + +class WatchdogConfig(BaseModel): + enabled: bool = True + stall_threshold_sec: int = 1200 # 20 min (specs/watchdog.md); tests induce 10-min stalls with a lower value + startup_grace_sec: int = 120 + nudge_enabled: bool = True # tmux only; Sam: default ON with the pane-ready guard + nudge_delay_sec: int = 30 # dwell before first nudge and between nudges (OMC idle-nudge) + nudge_budget: int = 3 # per run (OMC maxCount) + nudge_message: str = "Continue working on your assigned task and report concrete progress (not ACK-only)." + resume_nudge_budget: int = 2 # after a rate-limit pause ends and nothing moves + escalate_grace_sec: int = 120 # stall persists this long with nudges impossible/exhausted → escalate + kill_headless_on_escalate: bool = True # tmux never kills (human-facing); headless has no other lever + rate_limit_backoff_base_sec: int = 60 + rate_limit_backoff_max_sec: int = 1800 + rate_limit_max_pause_sec: int = 6 * 3600 + hard_max_restarts: int = 3 # consumed by PR-B's driver; declared now so config is stable + progress_paths: list[str] = [] # extra files/dirs whose mtime advance counts as progress + model_config = ConfigDict(extra="forbid") + +def resolve_watchdog_config(project: WatchdogConfig | None = None, *, env: Mapping[str, str] | None = None) -> WatchdogConfig + # env kill switch: ZO_WATCHDOG=0 → enabled=False; ZO_WATCHDOG_STALL_SEC overrides stall_threshold_sec (tests/ops). + +class WatchdogState(BaseModel): # persisted at /heartbeats/_watchdog.json + zo_session_id: str = "" + started_at: datetime + last_tick_at: datetime | None = None + ticks: int = 0 + last_progress_at: datetime + baseline_ticks: dict[str, int] = {} # agent_key → tick_count seen at start (pre-existing files don't count) + seen_ticks: dict[str, int] = {} + last_digest: str | None = None + last_file_mtimes: dict[str, float] = {} + stall_since: datetime | None = None + nudges_used: int = 0 + last_nudge_at: datetime | None = None + resume_nudges_used: int = 0 + escalated_at: datetime | None = None + stall_events: int = 0 + paused_at: datetime | None = None + paused_until: datetime | None = None + paused_reason: str | None = None + pause_attempts: int = 0 + total_paused_sec: float = 0.0 + last_never_block: str | None = None + +def new_state(*, now: datetime, zo_session_id: str = "", heartbeats: Sequence[HeartbeatRecord] = ()) -> WatchdogState +def load_state(memory_root: Path) -> WatchdogState | None # fail-open +def save_state(memory_root: Path, state: WatchdogState) -> Path # atomic + +# Evidence observers — each returns True iff it observed NEW progress and updates state in place. +def observe_heartbeats(state: WatchdogState, heartbeats: Sequence[HeartbeatRecord]) -> bool # any key whose tick_count > seen (and > baseline) +def observe_text(state: WatchdogState, text: str) -> bool # progress_digest changed (first observation = False) +def observe_files(state: WatchdogState, paths: Sequence[Path]) -> bool # any mtime advanced / new file (dirs: newest entry mtime, one level) + +class StallAction(StrEnum): NONE = "none"; NUDGE = "nudge"; RESUME_NUDGE = "resume_nudge"; PAUSE = "pause"; RESUME = "resume"; ESCALATE = "escalate" + +class StallVerdict(BaseModel): + action: StallAction + stalled: bool + reason: str # human sentence for comms/DECISION_LOG + never_block: NeverBlockReason | None = None + freshness: Freshness = Freshness.UNKNOWN + process_dead: bool | None = None + progress: bool = False + evaluated_at: datetime + +def evaluate(state: WatchdogState, config: WatchdogConfig, *, now: datetime, text: str, + heartbeats: Sequence[HeartbeatRecord], progress: bool, process_dead: bool | None, + is_interrupt: bool | None = None, can_nudge: bool) -> StallVerdict +``` + +`evaluate()` is the whole decision policy, pure and unit-testable. Contract: + +1. If `progress`: `state.last_progress_at = now`; clear `stall_since`; if paused → this is a verified resume → `action=RESUME` (accumulate `total_paused_sec`, clear pause fields). +2. `reason = classify_never_block(text, is_interrupt=…, heartbeats=…, now=now)`; `state.last_never_block = reason`. +3. `reason == RATE_LIMIT`: if not paused → `action=PAUSE` (`paused_at=now`, `paused_until=compute_pause_until(...)`, `pause_attempts+=1`); if paused and `now >= paused_until` → extend (`pause_attempts+=1`, new `paused_until`), still `PAUSE`; if `now - paused_at > rate_limit_max_pause_sec` → `ESCALATE` with reason "rate-limit pause exceeded max". Return. +4. Paused and reason is None (banner gone) and no progress: if `now < paused_until` → `NONE`; if `now >= paused_until` and `can_nudge` and `resume_nudges_used < resume_nudge_budget` → `RESUME_NUDGE`; if `now >= paused_until` and cannot nudge (headless) → `NONE` until `paused_until + rate_limit_backoff_base_sec`, then `ESCALATE` if still nothing (headless RESUME requires progress). Return. +5. Any other never-block reason → `NONE` (never nudge; `stall_since` untouched but stall clock does not run: set `last_progress_at = now` for COMPACTING only — compaction is progress; for CONTEXT_LIMIT/AUTH_ERROR/USER_ABORT/AWAITING_INPUT leave the clock — they may become stalls but they are still never nudged; escalation is allowed for AUTH_ERROR/CONTEXT_LIMIT after `stall_threshold_sec` so a dead session doesn't hang forever, with `reason` naming the cause). Return. +6. Startup grace: `now - started_at < startup_grace_sec` → `NONE`. +7. `stalled = process_dead is True or (now - last_progress_at) >= stall_threshold_sec`. If not stalled → `NONE` (clear `stall_since`). +8. Stalled: set `stall_since` (first time: `stall_events += 1`). Decide: + - `process_dead is True` → `ESCALATE`. + - `can_nudge and nudges_used < nudge_budget and (last_nudge_at is None or now - last_nudge_at >= nudge_delay_sec) and (now - stall_since >= nudge_delay_sec or nudges_used > 0)` → `NUDGE`. + - `nudges_used >= nudge_budget and now - last_nudge_at >= nudge_delay_sec` → `ESCALATE`. + - `not can_nudge and now - stall_since >= escalate_grace_sec` → `ESCALATE`. + - else `NONE` (waiting on dwell). + - `ESCALATE` fires ONCE per stall (guard on `escalated_at` newer than `stall_since`); subsequent ticks return `NONE` with `stalled=True`. + +Note: `evaluate()` mutates counters that are *decisions* (pause fields, `stall_since`, `escalated_at`, `stall_events`, `last_progress_at`); the CALLER increments `nudges_used`/`last_nudge_at`/`resume_nudges_used` only after the nudge was actually delivered. + +Process identity (in `watchdog.py`, section "Process identity"; may delegate to a new `src/zo/_proc.py` if `watchdog.py` would exceed 500 lines — core builder decides; `surrogate.py` is NOT modified in PR-A): +```python +def pid_alive(pid: int) -> bool # os.kill(pid, 0); ESRCH → False; EPERM → True (alive, not ours) +def process_start_identity(pid: int, *, platform: str | None = None, run: Callable = subprocess.run) -> str | None + # linux: /proc//stat → field 22 after the last ')' → f"linux:{starttime}"; darwin: `ps -o lstart= -p PID` + # under LC_ALL=C → f"darwin:{epoch_seconds}:0"; other: f"{platform}:{ps lstart raw}"; None on any failure. +def is_valid_process_start_identity(value: object, *, platform: str | None = None) -> bool # OMC regex allowlist, ≤1024 chars +def identities_may_match(recorded: str, observed: str) -> bool # equal, or darwin same-second with usec wildcard "0" +def is_process_dead(pid: int | None, recorded_identity: str | None, *, platform=None, run=subprocess.run) -> bool + # POSITIVE PROOF ONLY: pid None → False; not alive (ESRCH) → True; alive and recorded identity valid and observed + # identity valid and not may_match → True (recycled pid); anything unknown/EPERM → False. +``` + +Tests for core (`tests/unit/test_watchdog.py`, module docstring names WS-C + checks 11–12, section comments `# ---- (oracle check N) ----`, "seeded" in seeded-failure test names): +- taxonomy: parametrized positives per reason (≥3 each incl. the real Claude Code banner "You've hit your usage limit · resets at 3pm", "prompt is too long", "please run /login", "Do you want to proceed? ❯ 1. Yes"), negatives (`val_loss 0.4291`, `step 4290`, `GPU overloaded`, `commit 8466d29 fix weekly report`, `$ cat transcript.txt ... rate limit`), precedence (rate-limit banner + permission menu → RATE_LIMIT), `is_interrupt=True` → USER_ABORT, compacting heartbeat inside/outside window. +- `parse_rate_limit_reset`: "resets at 3pm" (today/tomorrow rollover), "resets at 14:30", "try again in 5 minutes", "retry-after: 90", none. +- freshness three-state; `evaluate()` scenario table with an injected clock: (a) **seeded 10-min stall** with `stall_threshold_sec=600`: ticks at t=0..11 min, no progress → `NUDGE` on the first tick past threshold+dwell, `NUDGE` ×3 spaced by `nudge_delay_sec`, then `ESCALATE` exactly once, then `NONE`; (b) **rate-limited is NOT nudged**: same clock but text carries the banner → `PAUSE` then `NONE`/`PAUSE`, never `NUDGE`; (c) **check 12 resume**: banner at t0 with "resets at" → `paused_until` parsed; at t≥paused_until banner gone, no progress → `RESUME_NUDGE`; then progress → `RESUME` with `total_paused_sec` accumulated; (d) headless (`can_nudge=False`) resume requires progress; (e) `process_dead=True` → `ESCALATE` immediately after grace; (f) startup grace suppresses; (g) `awaiting_input` never nudged; (h) compacting resets the clock; (i) progress via each observer (heartbeat tick delta ignoring baseline files, digest change ignoring spinner/counter churn, file mtime advance). +- identity: linux `/proc` parse (fixture string with a `)` inside comm), darwin `ps` mocked, malformed → None/not-dead, EPERM → alive, recycled pid → dead, unknown observed → not dead. +- persistence: `write_heartbeat`/`load_all_heartbeats` round-trip + atomicity (no partial file), `_watchdog.json` excluded, `sweep_stale_heartbeats`, `save_state`/`load_state` round-trip, `resolve_watchdog_config` env kill switch, `WatchdogConfig(extra="forbid")`. + +--- + +## 2. Heartbeat writer — hooks + shim + wiring (builder: hooks) + +- `.claude/settings.json`: append a SECOND entry to the existing `PostToolUse` array (keep the `Write|Edit` cascade-reminder entry untouched): matcher `"*"`, command `bash .claude/hooks/zo-hookkit.sh heartbeat 2>/dev/null || exit 0`, timeout 5. Do not add new events. +- `.claude/hooks/zo-hookkit.sh`: unchanged routing (`python3 -m zo.hookkit heartbeat`), but export `ZO_HOOK_EVENT_TS="$(date -u +%s)"` (cheap; not required by handler). Keep the `src/zo` guard. +- `src/zo/hookkit.py`: + - Make the top-level `from zo.contracts import …` LAZY (inside the handlers that use it: subagent-stop, sealed-paths) so `heartbeat` costs no pydantic import. `_SEALED_DEFAULTS` gets the literal `"contracts.json"` and adds `"heartbeats"` (prefix match seals the whole subtree — verify with the existing `_sealed_prefixes` logic; agents' Write/Edit into `/heartbeats/…` must be denied — seeded test). + - New `_agent_identity(data) -> tuple[str | None, str | None]` returning `(agent_type, agent_id)`; DO NOT change `_agent_name`. + - New `_handle_heartbeat(data)`: stdlib-only (json/os/datetime), NO import of `zo.watchdog` (pydantic). Resolve `memory_root = _memory_root(_repo_root())`; None → return. `agent_key = agent_id or f"lead-{session_id}"`; path `/heartbeats/.json`; debounce: if file mtime < 2 s ago and event is PostToolUse → return; read existing `tick_count`; write JSON with EXACTLY the `HeartbeatRecord` fields (schema_version, agent_key, agent_id, agent_type, session_id, zo_session_id (env `ZO_SESSION_ID`), pid (env `ZO_LEAD_PID` if set else None), process_start_identity (env `ZO_LEAD_PID_IDENTITY` or None), last_tick_at ISO-8601 UTC with `+00:00`, status, last_event, tick_count+1) via tmp + `os.replace`. Status by `hook_event_name`: PostToolUse → `executing`; Stop → `ready`; SubagentStop → `shutdown`; PreCompact → `compacting`; SessionEnd → `shutdown`. `last_event` = `tool_name` for PostToolUse else `hook_event_name`. + - Call `_handle_heartbeat(data)` (fail-open, wrapped in `contextlib.suppress(Exception)`) at the END of `_handle_drift_guard`, `_handle_subagent_stop`, `_handle_precompact`, `_handle_session_end` so those events also stamp — without changing their outputs. + - `_handle_post_tool_failure`: add `is_interrupt`, `agent_id`, `agent_type` to the failure record (do not remove existing fields). + - Register `"heartbeat": _handle_heartbeat` in `_HANDLERS`. +- `.gitignore`: add `memory/zo-platform/heartbeats/`, `memory/zo-platform/plan-ledger.json`, `memory/zo-platform/contracts.json` (verify with `git check-ignore -v`; note the `!memory/zo-platform/` re-include order — negations must come before these ignores or use paths after it). +- Tests: `tests/unit/test_hookkit.py` — heartbeat written keyed by `agent_id` (subagent payload) and by `lead-` (no identity); JSON validates against `zo.watchdog.HeartbeatRecord` (import allowed IN TESTS); tick_count increments; debounce; status mapping per event; fail-open when memory root unresolvable (no file, exit 0); heartbeats **sealed**: seeded Write into `/heartbeats/x.json` → deny JSON (pattern of `test_ledger.py::…sealed…`); failure record carries `is_interrupt`. `tests/integration/test_hooks_shim.py` — extend `TestSettingsWiring` to assert the heartbeat PostToolUse entry; end-to-end `_run_shim("heartbeat", payload, env={ZO_MEMORY_ROOT: tmp})` writes the file; shim still exits 0 with malformed stdin. + +--- + +## 3. Wrapper integration (builder: wrapper) + +`src/zo/_wrapper_models.py`: +- `AgentStatus`: add `PAUSED_RATE_LIMIT = "paused_rate_limit"`, `STALLED = "stalled"`. +- `LeadProcess`: add `pid_start_identity: str | None = None`, `nudges_used: int = 0`, `stalled: bool = False`, `paused_until: datetime | None = None`, `resume_at: datetime | None = None`, `pause_total_sec: float = 0.0`. + +`src/zo/wrapper.py` (keep < 500 lines… it is 1043 today; the file-size rule is already broken — do not make it worse: put the tick logic in a new `src/zo/_wrapper_watchdog.py` mixin/helper module (`WatchdogRunner` class holding config, state, memory_root, comms, clock, evidence paths) and keep wrapper.py changes to wiring + the pause loop + `_paste_and_submit` + pid capture): +- `LifecycleWrapper.__init__`: add `self._proc: subprocess.Popen | None = None` and `self._wd: WatchdogRunner | None = None`. +- `wait_for_completion(..., watchdog: WatchdogConfig | None = None, memory_root: Path | None = None, zo_session_id: str = "", delivery_repo already exists)`: build `self._wd = WatchdogRunner(...)` when `watchdog and watchdog.enabled and memory_root`; runner `start(now)` snapshots baseline heartbeats (`new_state`), sweeps stale files, seeds `progress_paths` = [`memory_root/plan-ledger.json`, comms log dir (`self._comms` log dir if exposed), `delivery_repo/.zo/experiments` if it exists] + config.progress_paths. Tear down in the existing `finally` (persist state). +- **tmux loop `_wait_tmux`**: hoist ONE `pane_text = self._capture_tmux_pane(pane_id, lines=200)` per iteration (before the liveness reads) and reuse it for `on_status` (5-line snapshot = last 5 lines of it). Call `self._watchdog_tick(process, text=pane_text, can_nudge=True)` right after `_maybe_open_training_pane()` and BEFORE the liveness reads, so it also runs on the suspected-dead `continue` path. Timeout check: use `elapsed = time.monotonic() - start_time - self._wd.paused_seconds()` when a runner exists. +- **headless loop `_wait_headless`**: replace the retry loop with: `new_text = self._read_new_output(process)` (byte cursor over BOTH stdout_log and stderr_log; keep a rolling window of the last 16 KB as `self._wd_text_window`); call `self._watchdog_tick(process, text=window, can_nudge=False)` after `_check_gate_mode_change()` and before the `rc` check; if `rc is not None` and the runner's last never-block is RATE_LIMIT (or the final window matches) → status `RATE_LIMITED`, `resume_at` = parsed reset (may be None) — no retries in the wrapper (PR-B's driver relaunches). Delete `_backoff_wait`/`_max_retries` retry semantics from the loop (keep `_detect_rate_limit` for the exit-classification path only, tightened to the watchdog patterns — remove bare `429`/`overloaded`). +- `_watchdog_tick(process, *, text, can_nudge)`: delegate to `self._wd.tick(process=process, text=text, can_nudge=can_nudge, now=self._wd.clock())` which returns the `StallVerdict`; the WRAPPER performs side effects by action: + - `NUDGE`/`RESUME_NUDGE`: guard `pane_ready_for_nudge(text)`; if ready → `self._paste_and_submit(pane_id, config.nudge_message)` then runner `record_nudge(now, resume=...)`; comms `log_checkpoint(agent="watchdog", phase="lifecycle", subtask="nudge", progress=f"nudge {n}/{budget}: {reason}")`; if NOT ready → comms checkpoint `subtask="nudge-skipped"` (pane busy/awaiting input) — no keys sent. + - `PAUSE` (first entry only, i.e. `paused_at == now`): `process.status = PAUSED_RATE_LIMIT`, `process.paused_until = …`; comms checkpoint `subtask="rate-limit-pause"` with the reset time. + - `RESUME`: `process.status = RUNNING`, `pause_total_sec` updated; comms checkpoint `subtask="rate-limit-resume"` (verified=True/False). + - `ESCALATE`: comms `log_error(agent="watchdog", error_type="stall", severity="blocking", description=verdict.reason, escalated_to="human")`; `process.stalled = True`; if headless and `config.kill_headless_on_escalate` → `self.kill_session(process)`; the loop returns with `status=STALLED`. tmux: keep waiting; when the session eventually ends, final status = `STALLED` if `process.stalled` and no progress since escalation, else the normal completion status. + - First stall detection (`stall_since` set this tick): comms `log_error(agent="watchdog", error_type="stall", severity="warning", ...)` once per stall. + - Persist runner state every tick (fail-open) and write a one-line JSONL trace per tick to `/heartbeats/_watchdog-ticks.jsonl` (`ts, action, stalled, reason, never_block, progress`) — check-11's seeded test asserts on this line. +- `_paste_and_submit(pane_id, text)`: extracted from the launch path; uses a NAMED buffer (`tmux load-buffer -b zo-nudge -` from stdin, `tmux paste-buffer -b zo-nudge -d -t pane`, sleep 1, `send-keys -t pane Enter`); refactor `_launch_tmux` to call it (behaviour-preserving; the existing `_verify_prompt_submitted` retry stays). +- tmux PID + identity (best-effort): after pane creation capture `#{pane_pid}` (shell pid), then during/after `_wait_for_tui_ready` resolve the claude child via `pgrep -P ` (first hit) — store `pid` and `pid_start_identity = process_start_identity(pid)` on `LeadProcess`; None if not resolvable (unknown ≠ dead). Headless: set `pid_start_identity` next to `pid=proc.pid`. Export `ZO_LEAD_PID`/`ZO_LEAD_PID_IDENTITY` is NOT required (hooks run inside the lead; skip). +- Existing tests: rewrite `test_detects_rate_limit_and_backs_off` and `test_rate_limit_exhausts_retries` to the new semantics (running process + rate-limit text → PAUSED, no sleep-backoff; exited process + rate-limit text → RATE_LIMITED with `resume_at`). +- New tests in `tests/unit/test_wrapper.py` (follow the class-patch pattern for `_tmux_pane_alive/_tmux_claude_running/_capture_tmux_pane/_kill_tmux_window`, `mock.patch("zo.wrapper.time.sleep")`, scripted `side_effect` lists, injected clock via `WatchdogRunner(clock=...)`): + - **`test_seeded_10min_stall_detected_and_escalated_within_one_poll` (check 11)** — tmux loop, `stall_threshold_sec=600`, `nudge_delay_sec=0`, heartbeat files aged, pane text neutral+idle prompt: assert `_paste_and_submit` called ≤ `nudge_budget`, an `error_type="stall"` comms line, `_watchdog-ticks.jsonl` shows `escalate` on the first tick after budget exhaustion, and the loop's final `LeadProcess.status == STALLED` when the pane dies afterwards. + - **`test_seeded_rate_limited_session_is_never_nudged` (check 11)** — pane text carries the Claude usage-limit banner: `_paste_and_submit` never called; `process.status == PAUSED_RATE_LIMIT` observed via `on_status`/state; comms `rate-limit-pause` line. + - **`test_seeded_rate_limit_pause_auto_resumes_on_reset` (check 12)** — banner with "resets at HH:MM", injected clock advances past it, banner disappears, no progress → one `RESUME_NUDGE` (paste called once with the nudge message), then heartbeat delta → `rate-limit-resume` checkpoint, `status == RUNNING`, `pause_total_sec > 0`, timeout accounting excludes the pause. + - Same three on the headless harness (`can_nudge=False`: no paste ever; running+banner → PAUSED; exited+banner → RATE_LIMITED with `resume_at`; stall → `kill_session` called once → STALLED). + - `test_watchdog_tick_runs_on_suspected_dead_path`, `test_single_pane_capture_per_poll`, `test_wrapper_proc_defaults_none`, `test_permission_dialog_is_never_nudged`, `test_paste_and_submit_uses_named_buffer`, `test_watchdog_disabled_when_config_off_or_no_memory_root` (no runner, loops behave exactly as before). + +--- + +## 4. Config + CLI threading + docs (builder: config-cli-docs) + +- `src/zo/project_config.py`: `ProjectConfig` gains `watchdog: WatchdogConfig = Field(default_factory=WatchdogConfig)` (import from `zo.watchdog`); `save_project_config` must round-trip it (nested dict). Add `model_config = ConfigDict(extra="ignore")` explicitly with a comment (documented choice, not accidental) — do NOT forbid, legacy configs may carry unknown keys. +- `src/zo/cli.py`: `ProjectContext.make_project_config()` returning `ProjectConfig | None` (None for legacy layouts); in `build` resolve `wd_cfg = resolve_watchdog_config(pcfg.watchdog if pcfg else None)`; add `extra_env["ZO_SESSION_ID"] = `; `_launch_and_monitor(..., watchdog: WatchdogConfig | None = None, memory_root: Path | None = None, zo_session_id: str = "")` → `wrapper.wait_for_completion(..., watchdog=..., memory_root=..., zo_session_id=...)`; after the wait: handle `AgentStatus.STALLED` (red "Session stalled — watchdog escalated; see logs/comms") and `RATE_LIMITED` with `resume_at` ("rate-limited; resets at …; rerun `zo continue` after") BEFORE the generic else. Add `--no-watchdog` flag to `build`/`continue` mapping to `enabled=False` (single concern per flag, PR-038). `_print_status` needs no change (checkpoint/error already render) — verify. +- `tests/unit/test_project_config.py`: watchdog block round-trip through save/load; legacy config without the block → defaults; unknown key ignored. `tests/unit/test_cli.py`: `--no-watchdog` plumbing → `wait_for_completion` receives `enabled=False`; default path passes `memory_root` and a config; `STALLED` status prints the stalled message (patch `wait_for_completion`). +- Docs: `specs/watchdog.md` — Status → implemented (PR-A), §3.1 cron tick → wrapper poll-loop external checker (plan supersedes), §3.4 remediation = taxonomy → nudge (bounded, tmux, pane-ready guard) → escalate (log; headless kill); reroute/respawn deferred to the PR-B driver; §3.5 config keys = `WatchdogConfig` fields; §4 integration points = real files/functions; §5 acceptance = checks 11–12 test names; §6 rewritten. `docs/reference/v2-rearchitecture.mdx`: watchdog feature row/status text (Phase 3 in progress: watchdog shipped, fresh-context loop next). Do NOT bump README test badge or counts (integrator does after the final count). + +--- + +## 5. Definition of done (integrator verifies) +- `python3 -m pytest -q` green (baseline 929 + new), `ruff check src/` clean, `bash scripts/validate-docs.sh` 0 failures. +- Every mechanism has (a) a seeded-failure test and (b) a wiring test proving it is invoked from the runtime path (settings.json entry asserted; `_watchdog_tick` asserted from both loops; CLI passes config to the wrapper). +- No new event types in comms; no writes outside `/heartbeats/`; heartbeats sealed; `.gitignore` verified with `git check-ignore`. +- MIT attribution comment present in `watchdog.py` for the ported pattern tables and identity logic. diff --git a/memory/zo-platform/research/2026-08-17-phase3-recon/raw-mappers.json b/memory/zo-platform/research/2026-08-17-phase3-recon/raw-mappers.json new file mode 100644 index 0000000..6344015 --- /dev/null +++ b/memory/zo-platform/research/2026-08-17-phase3-recon/raw-mappers.json @@ -0,0 +1,2063 @@ +[ + { + "surface": "LifecycleWrapper poll loop + process identity + rate limits (src/zo/wrapper.py, src/zo/_wrapper_models.py, callers in src/zo/cli.py::_launch_and_monitor). Read in full: wrapper.py (1043 L), _wrapper_models.py (60 L), surrogate.py (383 L), comms.py public API, cli.py:709-960 + 1040-1166, tests/unit/test_wrapper.py:455-641, pyproject.toml, .claude/settings.json hooks, .claude/hooks/zo-hookkit.sh, src/zo/hookkit.py. Headline: there are TWO structurally different poll loops (_wait_tmux, _wait_headless) with different data, different identity, and different remediation ceilings; the wrapper today has NO pid at all in tmux mode, NO process start time anywhere, NO psutil, NO heartbeat/never-block taxonomy, and rate-limit handling exists ONLY in headless (where it is also effectively dead code, see hazards).", + "integration_points": [ + { + "ref": "src/zo/wrapper.py:709", + "what": "(a) tmux poll loop top. `while True:` then `self._check_gate_mode_change()` (710) and `self._maybe_open_training_pane()` (711) \u2014 the established per-poll side-effect hook pattern. This is the ONLY unconditional point in the tmux loop: the suspected-dead branch does `continue` at 745 and the timeout check at 754 is skipped on those cycles.", + "action": "Insert `self._check_watchdog(process, pane_id)` as the third call at line 712, i.e. before the liveness reads at 713-714, so it runs on every cycle including dead-confirmation cycles. Do NOT place it after 749 (skipped by the 745 continue)." + }, + { + "ref": "src/zo/wrapper.py:713", + "what": "(a) per-poll data available in tmux mode: `pane_exists = self._tmux_pane_alive(pane_id)` (713), `claude_running = ...` (714), `poll_count` (706/716), `consecutive_dead` (707), `start_time = time.monotonic()` (702) for elapsed, `process.started_at` (wall clock, set at 279). Pane text is NOT computed on the hot path \u2014 it is only captured inside `if on_status:` at 751 as `self._capture_tmux_pane(pane_id, lines=5)`, i.e. 5 lines and only when a callback was passed.", + "action": "Hoist one pane capture per poll to a local (e.g. `pane_text = self._capture_tmux_pane(pane_id, lines=200)`) before the watchdog call and pass the SAME string to both the watchdog and `on_status` at 752, so a watchdog does not double the per-poll `tmux capture-pane` subprocess cost. 5 lines is too small for never-block classification (rate-limit / context-limit banners scroll)." + }, + { + "ref": "src/zo/wrapper.py:776", + "what": "(a) headless poll loop top: `while True:` then `self._check_gate_mode_change()` (777). Per-poll data: `rc = self._proc.poll()` (779), `output = self._read_tail(process.stdout_log)` (792, last 100 lines), team status only inside `if on_status:` (811-813, note it passes `\"\"` as the second arg \u2014 headless never surfaces output to the CLI).", + "action": "Insert the watchdog call at line 778 (before the `poll()` at 779) for liveness/heartbeat, and reuse `output` from 792 for the never-block taxonomy by moving the taxonomy classification between 792 and 793 so it runs BEFORE `_detect_rate_limit` decides to back off." + }, + { + "ref": "src/zo/wrapper.py:547", + "what": "`wait_for_completion(self, process, *, poll_interval=10.0, timeout=None, on_status=None, gate_mode_file=None, project_name='', delivery_repo=None)` \u2014 the single entry point for both loops; per-run state is stashed as instance attrs at 574-578 (`_gate_mode_file`, `_last_gate_mode`, `_training_pane_id`, `_project_name`, `_delivery_repo`) and dispatch is at 580-584 on `process.tmux_pane_id`.", + "action": "Add watchdog knobs here as keyword-only args (e.g. `heartbeat_dir: Path | None`, `watchdog: WatchdogConfig | None`) and stash them alongside 574-578 in the same `try:`/`finally:` (585-586) so teardown mirrors `_close_training_pane()`. Keep defaults None so the 5 existing call sites keep working." + }, + { + "ref": "src/zo/cli.py:920", + "what": "the only production caller: `process = wrapper.wait_for_completion(process, on_status=_print_status, gate_mode_file=gate_mode_file, project_name=project_name, delivery_repo=delivery_repo)`. Note the memory root is NOT passed \u2014 it only arrives implicitly as `gate_mode_file` (`memory.memory_root / \"gate_mode\"`, cli.py:1159), so today the wrapper can only reach the memory root via `gate_mode_file.parent`.", + "action": "Thread an explicit `memory_root: Path | None` through `_launch_and_monitor` (signature at cli.py:709-731) and into `wait_for_completion` \u2014 do not derive it from `gate_mode_file.parent`, which is fragile and None-able." + }, + { + "ref": "src/zo/cli.py:1076", + "what": "`extra_env[\"ZO_MEMORY_ROOT\"] = str(memory.memory_root)` (+ `ZO_DELIVERY_ROOT` 1077, `ZO_CONTRACTS_PATH` 1078) is already injected into the Claude Code subprocess env for both launch modes (consumed at wrapper.py:230-231 tmux inline prefix / 458-460 headless env). The hook side already resolves it identically at src/zo/hookkit.py:64-69.", + "action": "Heartbeat writers (hooks/agents) and the external checker can share exactly this root with zero new plumbing: heartbeat dir = `$ZO_MEMORY_ROOT/heartbeats/.json`. Reuse `hookkit._memory_root()` semantics verbatim so writer and checker cannot disagree." + }, + { + "ref": "src/zo/hookkit.py:81", + "what": "`_trace(event, data)` already appends one JSONL line per hook invocation to `logs/hook-trace-{date}.jsonl` with `ts`, `event`, `agent_identity`, `session_id` \u2014 an existing, already-wired, per-activity liveness signal. Registered hooks that fire during a live session: PreToolUse/sealed-paths (.claude/settings.json:68), Stop/drift-guard (:97), SubagentStop (:109), PreCompact (:121), SessionEnd (:134), PostToolUseFailure (:146). Called unconditionally from `main` at hookkit.py:403 and 405 (both the exception and success paths).", + "action": "Write the heartbeat from `_trace` (or a sibling `_heartbeat()` called from the same two lines) \u2014 it is the single choke point every hook event already passes through, and it is fail-open by contract. Do NOT invent a second activity path." + }, + { + "ref": "src/zo/wrapper.py:255", + "what": "(d) the nudge mechanics already exist inline in `_launch_tmux`: `tmux load-buffer ` (255-258) -> `tmux paste-buffer -t ` (259-262) -> `time.sleep(1)` (266) -> `tmux send-keys -t Enter` (267-270), with a full retry variant at `_verify_prompt_submitted` (386-398). All of it is pane-targeted, so it CAN be re-pointed at a live pane mid-session. It is not factored into a reusable method today.", + "action": "Extract `def _paste_and_submit(self, pane_id: str, text_file: Path) -> None` from 255-270 and have both `_launch_tmux` and the watchdog nudge call it. That is the whole physical nudge capability in tmux mode." + }, + { + "ref": "src/zo/wrapper.py:933", + "what": "(b) tmux identity today: `_tmux_claude_running` runs `tmux display-message -t -p '#{pane_current_command}'` (947-951) and compares the string against a shell set `{bash, zsh, fish, sh, dash, tcsh, csh}` (959-960). No pgrep, no `#{pane_pid}`, no PID anywhere \u2014 repo-wide grep for `pane_pid|pgrep|psutil|lstart|etime|start_time` returns zero hits in wrapper/surrogate/hookkit.", + "action": "Add `_tmux_pane_pid(pane_id)` via `tmux display-message -t -p '#{pane_pid}'`. NOTE: that yields the pane's SHELL pid, because the pane is created as a bare shell (214-218) and claude is then typed into it with send-keys (239-242) \u2014 so the claude pid must be found by walking children (`pgrep -P `), one extra hop. Persist it on `LeadProcess` (see _wrapper_models.py:26)." + }, + { + "ref": "src/zo/_wrapper_models.py:26", + "what": "`LeadProcess` carries `pid, status, started_at, completed_at, exit_code, team_name, stdout_log, stderr_log, tmux_pane_id` \u2014 no `pid_start_time`, no `last_heartbeat`, no `nudge_count`, no `paused_until`. `AgentStatus` (15-23) has SPAWNING/RUNNING/COMPLETED/ERRORED/RATE_LIMITED/TIMED_OUT \u2014 no STALLED, NUDGED, PAUSED, RESTARTED.", + "action": "Extend both: add `pid_start_time: str | None` (ps lstart string, the identity tuple half), `nudges_used: int = 0`, `paused_until: datetime | None`; add `AgentStatus.STALLED` / `PAUSED_RATE_LIMIT` / `RESTARTED`. cli.py:926 compares `process.status == \"completed\"` as a raw string (StrEnum makes this work), so new members are safe there but must be handled in the `else` branch at 928-929." + }, + { + "ref": "src/zo/wrapper.py:824", + "what": "(escalation) `kill_session` \u2014 tmux path does `tmux kill-pane -t ` (828-831), headless path does SIGTERM (845) / wait 5s (848) / SIGKILL (851). Both return `ERRORED` + `exit_code=-9` (836-840, 857-861). There is no relaunch anywhere in the wrapper.", + "action": "Escalation-to-restart must be built ABOVE the wrapper (in `_launch_and_monitor` or the new experiment_loop orchestrator): `kill_session` -> re-derive prompt from ledger -> `launch_lead_session` again. Do not build a restart inside `_wait_tmux`, which returns a terminal LeadProcess by contract." + }, + { + "ref": "src/zo/cli.py:869", + "what": "(e) comms plumbing: `_print_status` tails `zo_root/logs/comms/*.jsonl` (869-872), dedupes by `event_id` (885-888) and renders `decision` (891), `gate` (897), `checkpoint` (904 -> `\u21b3 agent: progress`), `error` (908 -> red `\u2717 ERROR`). `self._comms.log_checkpoint(agent=, phase=, subtask=, progress=)` (comms.py:381-416) and `log_error(agent=, error_type=, severity=, description=)` (comms.py:344-379, severity \u2208 info|warning|blocking|critical, comms.py:61-67) are already used from the wrapper at wrapper.py:283, 335, 353, 380, 404, 470, 625, 733, 756, 787, 802, 832, 853.", + "action": "Watchdog logs via `self._comms.log_checkpoint(agent=\"watchdog\", phase=\"lifecycle\", subtask=\"heartbeat|nudge|pause|escalate\", ...)` and `log_error(..., error_type=\"stall\", severity=\"warning\"|\"blocking\")` \u2014 both render live in the CLI for free with no cli.py change. `log_message(...message_type=\"escalation\", priority=\"critical\")` (comms.py:221) matches specs/watchdog.md:76 (\"emits nudge/respawn messages through the existing comms bus\") but is NOT rendered by _print_status (891-913 has no `message` branch) \u2014 add a branch if nudges should be visible." + }, + { + "ref": "src/zo/surrogate.py:279", + "what": "(f) `_pid_alive(pid)`: `os.kill(pid, 0)` with ProcessLookupError->False, PermissionError->True (exists, other user), OSError->False. Consumed by `sweep_locks` (293-311) and `live_sessions` (314-321). Lock files are written by `register_session` (240-268) with `pid`, `role`, `surrogate_id`, `worktree`, `started_at` \u2014 where `started_at` is the WALL-CLOCK time the lock was written (261), not the process start time.", + "action": "Reuse `_pid_alive` as the liveness primitive (read-only dependency; move it to a shared `zo/_proc.py` if the watchdog needs it too, rather than importing a private name). It is NOT sufficient for the PID-identity requirement \u2014 pair it with a start-time probe." + }, + { + "ref": "pyproject.toml:34", + "what": "(b) dependencies = pydantic, pyyaml, click, rich, nbformat. psutil is NOT a dependency (and appears nowhere in the repo).", + "action": "Get process start time via subprocess `ps -p -o lstart=` (and/or `-o etime=`), which works on both macOS 25.6 (dev box, no /proc) and Linux (oracle #13 box). Do not add psutil for this; do not use /proc//stat (macOS has no /proc)." + }, + { + "ref": "src/zo/orchestrator.py:687", + "what": "(deferral 3) `get_current_phase` already implements the PR-036/PR-037 precedence: GATED loop (710-712), ACTIVE loop (713-716), PENDING-with-deps-met (717-726); docstring 690-705. State is read from STATE.md via `self._memory.read_state()` (232) / `start_session` (237-240).", + "action": "Read-only dependency for the ledger cutover: any ledger-backed resume MUST reproduce this exact 3-tier order and the 'BLOCKED is intentionally skipped' rule (704-705). Evidence/rationale: memory/zo-platform/PRIORS.md:1089-1150 (priority order is contract, tests at :1142-1146)." + }, + { + "ref": "src/zo/orchestrator.py:1265", + "what": "(deferral 3) `evaluate_loop_state(registry, phase.phase_id, policy)` is called at 1276 with a registry from `load_registry(exp_dir)` (1270) \u2014 experiment-registry only, no ledger read. `experiment_loop.evaluate_loop_state` signature at src/zo/experiment_loop.py:205 is `(registry, phase, policy=None)`.", + "action": "To make evaluate_loop_state read the ledger, add a keyword-only `ledger: LedgerFile | None = None` at experiment_loop.py:205 and pass `load_ledger(memory_root / LEDGER_FILENAME)` (ledger.py:93, :51) from orchestrator.py:1276 \u2014 a purely additive signature change; 15+ existing call sites in tests/unit/test_experiment_loop.py pass positionally." + } + ], + "hazards": [ + { + "ref": "src/zo/wrapper.py:277", + "hazard": "TMUX MODE HAS NO PID AT ALL: `LeadProcess(pid=None, ..., tmux_pane_id=pane_id)`. tmux is the DEFAULT launch mode (`use_tmux=True` at 111, `_is_in_tmux()` at 144/974). So the watchdog's 'PID + process-start-time identity' requirement has no input in the default path, and `_pid_alive`-style checks are unavailable. Liveness is a command-NAME string comparison (933-960) which returns True for any non-shell foreground command (`vim`, `node`, `git`, a stray `python`) \u2014 a dead claude with anything else running in the pane reads as ALIVE, and a live claude whose pane foreground briefly flips to `bash`/`sh` reads as DEAD (this is exactly what `_DEAD_CONFIRM_POLLS`=2 at :78 and the 693-696 docstring exist to paper over).", + "mitigation": "Capture `#{pane_pid}` at launch (after 219), resolve the claude child via `pgrep -P`, store pid + `ps -p -o lstart=` on LeadProcess, and make the watchdog decide on the (pid, start_time) tuple rather than on `pane_current_command`. Keep `_tmux_claude_running` for the existing completion path; do not let the watchdog inherit its semantics." + }, + { + "ref": "src/zo/wrapper.py:801", + "hazard": "RATE-LIMIT BACKOFF BLOCKS THE WHOLE POLL LOOP: `time.sleep(wait_secs)` at 807 with `wait_secs = self._base_backoff * (2 ** attempt) + random.uniform(0,5)` (:900, base 30.0 at :90) => sleeps of ~30s, ~60s, ~120s. During each sleep there is no `on_status`, no timeout check (815), no gate-mode re-read, and no watchdog tick. Oracle #11 ('detected and escalated within one poll cycle') and #12 ('pause auto-resumes on reset') both fail if the watchdog piggybacks on this loop.", + "mitigation": "Replace the single blocking sleep with a bounded wait loop (`while time.monotonic() < resume_at: watchdog_tick(); time.sleep(min(poll_interval, remaining))`). The 'wait-and-resume' mode must be implemented as a paused STATE evaluated each poll, not as a sleep." + }, + { + "ref": "src/zo/wrapper.py:792", + "hazard": "HEADLESS RATE-LIMIT DETECTION IS EFFECTIVELY DEAD AND ALSO LATCHES: (1) it reads only `process.stdout_log` (792) \u2014 but headless launches with `--print --output-format json` (436-437), so stdout holds a single JSON blob written at exit; during the run stdout is empty and errors go to `stderr_log` (453/455) which is never read. (2) `_read_tail` re-reads the last 100 lines of an append-only log every poll (1029-1037), so once a '429' or 'rate limit' string lands in the tail it re-matches forever, `retries` only increments (808) and never resets, and the session is forced to RATE_LIMITED (795-800) even after it recovered.", + "mitigation": "Read stderr_log too; track a byte offset / last-seen position rather than re-tailing; reset `retries` when a poll produces no match; and gate the whole thing behind the never-block taxonomy so 'rate limit' text quoted inside the model's own output cannot trigger it." + }, + { + "ref": "src/zo/wrapper.py:51", + "hazard": "`_RATE_LIMIT_PATTERNS` includes bare `re.compile(r\"429\", re.IGNORECASE)` (:52). Applied to a tmux PANE CAPTURE (which is what a tmux-mode rate-limit check would have to use), any '429' substring matches \u2014 a loss value `0.4291`, `step 4290`, a file path, a diff hunk header. Feeding pane text to `_detect_rate_limit` (:894-896) to build the never-block taxonomy will misclassify healthy training output as rate-limited and suppress legitimate nudges.", + "mitigation": "Do not reuse `_RATE_LIMIT_PATTERNS` for the taxonomy. Write a separate anchored pattern set (e.g. `\\b(429|HTTP 429)\\b.*(rate|limit|too many)`, 'resets at', 'usage limit reached', 'Claude usage limit', 'context low'/'compact', 'invalid api key'/'authentication_error', 'ESC to interrupt'), and require a match in the LAST N lines only." + }, + { + "ref": "src/zo/wrapper.py:718", + "hazard": "TMUX MODE HAS NO RATE-LIMIT HANDLING WHATSOEVER \u2014 `_wait_tmux` (669-761) never calls `_detect_rate_limit`, never reads a log, and never sets `AgentStatus.RATE_LIMITED`. Worse: a rate-limited Claude TUI is still the pane's foreground command, so `_tmux_claude_running` keeps returning True (960) and the session sits 'RUNNING' forever until `timeout` (754). A naive watchdog added at 712 will therefore see a frozen-but-alive pane and nudge it \u2014 the exact failure oracle #11 forbids.", + "mitigation": "The never-block classifier MUST run against pane text in tmux mode before any nudge, and it must be the FIRST thing the watchdog does, not a post-filter on the nudge decision." + }, + { + "ref": "src/zo/wrapper.py:462", + "hazard": "HEADLESS SESSIONS CANNOT BE NUDGED AT ALL. `subprocess.Popen(cmd, stdout=stdout_fh, stderr=stderr_fh, text=True, env=env)` does not set `stdin`, so the child INHERITS the parent's stdin (the operator's terminal) \u2014 it is not a writable pipe. Independently, the prompt is passed as argv `-p prompt` (:450) with `--print` (:436): it is a one-shot non-interactive invocation that exits after one response. There is no channel to inject a follow-up.", + "mitigation": "Scope the nudge remediation to tmux mode only. In headless mode the escalation ladder must be heartbeat-stall -> kill_session (824) -> fresh relaunch with a re-derived prompt. Say so explicitly in the design so nobody builds a stdin nudge." + }, + { + "ref": "src/zo/wrapper.py:255", + "hazard": "NUDGE-BY-PASTE USES THE GLOBAL TMUX BUFFER STACK: `tmux load-buffer ` (255-258) is session-global, so a mid-run nudge silently clobbers whatever the human operator had in their tmux paste buffer, and races with any concurrent `_verify_prompt_submitted` retry (386-393). It also requires writing the nudge text to a file first (the launch path writes `-prompt.txt` at 205-206).", + "mitigation": "Use a named buffer (`tmux load-buffer -b zo-nudge -` reading from stdin, then `paste-buffer -b zo-nudge -d`) so the operator's default buffer is untouched and no temp file is needed." + }, + { + "ref": "src/zo/wrapper.py:741", + "hazard": "The suspected-dead branch calls `on_status` (741-743) then `time.sleep(min(poll_interval, self._DEAD_RECHECK_INTERVAL))` (744) and `continue`s (745), SKIPPING the timeout check at 754. Any watchdog code placed at 749+ (the natural-looking 'after liveness' spot) silently stops running for the entire dead-confirmation window \u2014 precisely the window where the session is most likely wedged.", + "mitigation": "Insert the watchdog call at 712 (loop top, before 713) so it is unconditional; add a regression test that asserts the watchdog hook fires on a poll that takes the 745 `continue` path." + }, + { + "ref": "src/zo/wrapper.py:779", + "hazard": "`rc = self._proc.poll() if self._proc else -1` \u2014 `self._proc` is ONLY assigned in `_launch_tmux` (:287, to None) and `_launch_headless` (:474). `__init__` (83-100) never initialises it. Any Phase-3 flow that constructs a fresh `LifecycleWrapper` and calls `wait_for_completion` on a reconstructed/restarted `LeadProcess` (exactly the fresh-context-loop restart pattern) raises `AttributeError: _proc`. `kill_session` has the same bug at :847 (`if self._proc:`). Existing tests only pass because they poke `wrapper._proc = mock_proc` by hand (tests/unit/test_wrapper.py:468, 488, 510, 531).", + "mitigation": "Add `self._proc: subprocess.Popen | None = None` to `__init__` around line 99 (alongside `self._bypass_restore_fn`), or switch 779/847 to `getattr(self, \"_proc\", None)` as already done for the log handles at :1041." + }, + { + "ref": "src/zo/wrapper.py:751", + "hazard": "The only pane text on the tmux hot path is `self._capture_tmux_pane(pane_id, lines=5)` (751) and it is computed ONLY when `on_status` is truthy. A watchdog that reads pane text will therefore either (a) get nothing when `on_status is None`, or (b) add a second `tmux capture-pane` subprocess per poll on top of `_tmux_pane_alive` (925-928, a full `list-panes -a`) and `_tmux_claude_running` (947-951) \u2014 4 subprocess spawns per 10s poll per session.", + "mitigation": "Hoist a single capture per poll and share it; consider replacing the `list-panes -a` scan (which enumerates every pane in every session) with a targeted `tmux has-session`/`display-message -t ` probe while you are in there." + }, + { + "ref": "src/zo/surrogate.py:261", + "hazard": "The existing PID-liveness registry is itself vulnerable to the recycled-PID failure the watchdog is meant to eliminate: `register_session` stores `started_at` = the time the LOCK was written (:261), not the process start time, and `sweep_locks` (:293-311) trusts `_pid_alive` (:279) alone. A recycled PID keeps a dead surrogate's lock alive forever, which suppresses auto-consolidation in `_launch_and_monitor` (cli.py:763, 949-958).", + "mitigation": "Add the process start time to the lock payload and compare it in `_pid_alive`'s callers. Building the start-time probe for the watchdog is the natural place to fix this \u2014 share one helper." + }, + { + "ref": ".claude/hooks/zo-hookkit.sh:21", + "hazard": "`[[ -d \"$REPO_ROOT/src/zo\" ]] || exit 0` \u2014 the hook shim silently no-ops unless the repo containing `.claude/hooks/` has `src/zo`. Hook-written heartbeats therefore only exist for sessions whose settings resolve to the ZO PLATFORM repo. That happens to hold today because the lead session is launched with `cwd=str(zo_root)` (cli.py:801, wrapper.py:226 `--add-dir `), but any future delivery-repo-cwd session (or a fresh-context builder spawned in the delivery repo for ML Phase 4) gets NO heartbeat and will look permanently stalled.", + "mitigation": "Either make the heartbeat writer independent of the platform-repo gate (write from the wrapper/agent side keyed on $ZO_MEMORY_ROOT), or explicitly assert 'builder sessions are launched with cwd=zo_root' as a Phase-3 invariant with a test." + }, + { + "ref": "src/zo/wrapper.py:766", + "hazard": "`_wait_headless` never calls `_maybe_open_training_pane` (only `_wait_tmux` does, at 711) and passes `\"\"` instead of output to `on_status` (813). Feature parity between the two loops is already broken; adding the watchdog to only one loop (or with different semantics) will compound it, and the tmux loop is the one all local testing exercises while oracle #13 runs headless on Linux.", + "mitigation": "Factor the per-poll logic into one `_poll_tick(process, *, pane_text, stdout_tail)` used by both loops, and write the watchdog unit tests against BOTH `_wait_tmux` and `_wait_headless`." + }, + { + "ref": "src/zo/wrapper.py:754", + "hazard": "The `timeout` parameter is checked with `time.monotonic() - start_time` where `start_time` is set once at 702/772 \u2014 it measures wall time since `wait_for_completion` began, not time since last progress. Tests exploit `timeout=-1` to force the branch (tests/unit/test_wrapper.py:604). If the watchdog's stall window is implemented on top of this, a rate-limit pause counts against the session timeout and a paused-but-healthy session gets TIMED_OUT.", + "mitigation": "Track `last_progress_at` separately from `start_time`, and exclude paused intervals from the timeout accounting (or extend `timeout` by the pause duration on resume)." + } + ], + "reusable": [ + { + "ref": "src/zo/wrapper.py:648", + "what": "`_check_gate_mode_change()` \u2014 the canonical 'per-poll, read a file under the memory root, log via comms only on change' pattern: guarded `getattr` for the per-run attr (650), `.exists()` guard (651), `OSError`-swallowing read (653-655), first-read primes without logging (657-660), change -> `log_checkpoint` (661-667). The heartbeat-freshness checker should be structurally identical." + }, + { + "ref": "src/zo/wrapper.py:588", + "what": "`_maybe_open_training_pane()` \u2014 the precedent for a per-poll side effect that fires at most once, self-disables on failure by setting a sentinel (`self._training_pane_id = \"\"` at 632/634), and is torn down in the `finally:` of `wait_for_completion` (585-586 -> `_close_training_pane` 636-646). Model the nudge-budget and pause state on this (bounded, idempotent, torn down)." + }, + { + "ref": "src/zo/surrogate.py:279", + "what": "`_pid_alive(pid)` \u2014 the only PID-liveness code in the repo: `os.kill(pid, 0)`, ProcessLookupError->False, PermissionError->True, OSError->False. Correct and complete for the liveness half; needs a start-time companion for the identity half." + }, + { + "ref": "src/zo/ledger.py:81", + "what": "`_atomic_write(path, text)` \u2014 mkstemp in the destination dir + `os.replace` (81-91). Heartbeat JSON writes must use this (a torn heartbeat read by the checker mid-write is a false stall). Pair with `load_ledger`'s fail-open parse style at 93-99 (`return None` on OSError/ValueError)." + }, + { + "ref": "src/zo/training_display.py:92", + "what": "`_time_ago(ts)` \u2014 `datetime.fromisoformat(ts)` + `(datetime.now(UTC) - dt).total_seconds()` with `except (ValueError, TypeError)` (92-105). The exact freshness computation the watchdog needs, already written and already used for a JSON-status-file dashboard." + }, + { + "ref": "src/zo/cli.py:2912", + "what": "`zo watch-training` (2912-2967) \u2014 a working, shipped example of an EXTERNAL poll loop over a JSON status file in a tmux split pane (`run_live_display(log_dir, interval=...)` at 2963). If a human-facing `zo watch` view of heartbeats is wanted, clone this shape rather than inventing one; note it is spawned from the wrapper via `tmux split-window ... zo watch-training` at wrapper.py:616-620." + }, + { + "ref": "src/zo/hookkit.py:64", + "what": "`_memory_root(repo_root)` \u2014 `ZO_MEMORY_ROOT` env first, else `repo_root/memory/zo-platform` if it is a dir, else None (64-69). The checker must resolve the heartbeat root with exactly this function so writer (hook) and reader (wrapper) can never disagree." + }, + { + "ref": "src/zo/hookkit.py:392", + "what": "`main()` fail-open dispatch (392-407): unknown event -> return 0 (398), handler exceptions swallowed (401-404), `_trace` called on both paths (403, 405), always `return 0`. Any heartbeat write added on the hook side must keep this contract \u2014 a heartbeat failure must never block a tool call." + }, + { + "ref": "tests/unit/test_wrapper.py:543", + "what": "THE TEST-MOCKING PATTERN for `_wait_tmux`, to copy verbatim for watchdog tests. `@mock.patch(\"zo.wrapper.time.sleep\")` as a decorator on the test (543/576/611) makes the loop free-running; then inside the test, `mock.patch.object(LifecycleWrapper, \"_tmux_pane_alive\", ...)` and `mock.patch.object(LifecycleWrapper, \"_tmux_claude_running\", ...)` are patched ON THE CLASS (they are @staticmethods) \u2014 with `return_value=` for a constant (560, 624) or `side_effect=[...]` for a per-poll script (596, 627); `_kill_tmux_window` (562, 629) and `_capture_tmux_pane` (634) are class-patched too, while `monitor_team` is patched on the INSTANCE (`mock.patch.object(wrapper, \"monitor_team\", return_value=TeamStatus(team_name=\"alpha\"))`, 563-566). Loop exit is forced either by a scripted dead sequence (622: `[True,True,False,True,False,False]`) or by `timeout=-1` (604) which trips the 754 branch immediately. Call-count assertions encode the guard arithmetic: `assert alive_mock.call_count == 4` = 2 grace + 2 confirm (573-574). Fixtures: `comms` (35-40, real CommsLogger into tmp_path), `tmp_log_dir` (28-32), `wrapper` (44-46). For headless, `wrapper._proc = mock.MagicMock()` with `poll.side_effect=[None, 0]` and `wrapper._stdout_fh/_stderr_fh = mock.MagicMock()` (507-513), plus a real stdout log file written with the trigger text (514-515)." + }, + { + "ref": "src/zo/comms.py:381", + "what": "`log_checkpoint(agent, phase, subtask, progress, *, current_best_metric=None, target_metric=None, blockers=None)` and `log_error(agent, error_type, severity, description, *, affected_artifacts=None, resolution='pending', escalated_to='')` (344). `blockers` and `escalated_to` are the natural carriers for stall reason / escalation target. Severity vocabulary at comms.py:61-67." + } + ], + "open_questions": [ + "tmux nudge semantics while Claude is mid-turn: nothing in the codebase exercises `paste-buffer` + Enter against a BUSY TUI (the only uses are at launch, wrapper.py:255-270, and the launch retry, 386-398, both against an idle input box). Does a paste+Enter into a working Claude queue as the next user turn, get swallowed, or send an interrupt? This must be settled empirically before the nudge budget design is fixed \u2014 it determines whether a nudge is safe to repeat.", + "Which side writes the heartbeat? The hook path (hookkit.py:403/405) fires only on PreToolUse(Write|Edit), Stop, SubagentStop, PreCompact, SessionEnd, PostToolUseFailure (.claude/settings.json:68,97,109,121,134,146) \u2014 a session that is thinking, or running long Bash/Read/Grep sequences, emits NO hook event and will look stalled. Is a PostToolUse `*` matcher heartbeat acceptable (write amplification on every tool call), or does the heartbeat need a second source (pane-text delta / comms-log mtime)?", + "`_tmux_claude_running` (wrapper.py:947-960) reads `#{pane_current_command}`. Does Claude Code's Bash tool put its child in the pane's foreground process group (flipping `pane_current_command` to `bash`/`sh`, which IS in the shells set at 959)? The `_DEAD_CONFIRM_POLLS` docstring at 693-696 ('a brief foreground flip') implies it was observed. If yes, long Bash tool calls are already a source of false 'dead' readings and the watchdog must not use this signal at all.", + "Rate-limit reset time: 'wait-and-resume on reset' needs a reset timestamp. Nothing in the repo parses one \u2014 `_RATE_LIMIT_PATTERNS` (wrapper.py:51-56) are boolean matchers only, and headless stdout carries only the final JSON blob (436-437). Is the reset time available in the TUI banner text (pane capture), in stderr, or must the pause fall back to a fixed retry-after? Oracle #12 needs a deterministic answer.", + "Heartbeat root for the fresh-context ML loop: `ZO_MEMORY_ROOT` points at the DELIVERY repo's `.zo/memory` (cli.py:69, 1076) while comms logs and the wrapper's hook-trace go under the PLATFORM `zo_root` (cli.py:1096, hookkit.py:94). Which root owns `heartbeats/`? Splitting them means the checker needs both paths threaded through `wait_for_completion`.", + "Restart identity across a fresh-context iteration: after `kill_session` (wrapper.py:824) + relaunch, the new session gets a new pid/pane and a new `session_id` (cli.py:1094 `s-{uuid4}`), but the ledger/experiment lineage must tie the iterations together. Is the restart counter persisted in `plan-ledger.json` (ledger.py:227 `record_attempt` looks like the right hook) or in a new watchdog state file?", + "Multi-agent scope: the spec says 'per-agent heartbeat' but the wrapper only ever tracks ONE process (the lead, `LeadProcess` singular, _wrapper_models.py:26); teammates are observed only as file-system artefacts (`monitor_team` 481-497 reads `~/.claude/teams//config.json` and `~/.claude/tasks//*.json`, with `TeamMember.status` a free-form string, _wrapper_models.py:42-48). Does Phase 3 watch only the lead, or does it need per-teammate heartbeats with no process handle to correlate against?" + ], + "notes": "Direct answers, quoted.\n\n(a) INSERTION POINTS. tmux: wrapper.py:709-711 `while True:` / `self._check_gate_mode_change()` / `self._maybe_open_training_pane()` \u2014 insert at 712. Available per poll: `pane_exists` (713), `claude_running` (714), `poll_count` (706/716), `consecutive_dead` (707), elapsed via `start_time = time.monotonic()` (702) or `process.started_at` (279). Pane text is NOT on the hot path \u2014 only `self._capture_tmux_pane(pane_id, lines=5)` at 751, inside `if on_status:` (749). Team status likewise only inside `if on_status:` (750). headless: wrapper.py:776-777 \u2014 insert at 778; `output = self._read_tail(process.stdout_log)` at 792 is the stdout tail (100 lines, `_read_tail` at 1028-1037); team status only inside `if on_status:` (811-813, which passes `\"\"` as the pane arg). Both loops end with `time.sleep(poll_interval)` (761, 822); default `poll_interval=10.0` (554).\n\n(b) PROCESS IDENTITY. tmux: `pid=None` (wrapper.py:278). `_tmux_claude_running` (933-960) does NOT use pgrep or `#{pane_pid}` \u2014 it runs `tmux display-message -t -p '#{pane_current_command}'` (947-951) and returns `cmd not in shells and len(cmd) > 0` (959-960). `_tmux_pane_alive` (919-931) runs `tmux list-panes -a -F '#{pane_id}'` and does a substring-line membership test (931). headless: real pid from `subprocess.Popen` (462-466, `pid=proc.pid`). START TIME: not obtainable today \u2014 repo-wide grep for `psutil|pane_pid|pgrep|lstart|etime|start_time` finds nothing in wrapper/surrogate/hookkit; psutil is absent from pyproject.toml:34-40. Use `ps -p -o lstart=` (portable across the macOS dev box and the Linux oracle box; /proc is not an option on darwin).\n\n(c) RATE LIMITS. Patterns, verbatim, wrapper.py:51-56: `re.compile(r\"429\", re.IGNORECASE)`, `re.compile(r\"rate.?limit\", re.IGNORECASE)`, `re.compile(r\"overloaded\", re.IGNORECASE)`, `re.compile(r\"too many requests\", re.IGNORECASE)`. Detector: `return any(pat.search(output) for pat in _RATE_LIMIT_PATTERNS)` (896). Headless only, at 793: `if self._detect_rate_limit(output):` -> exhausted branch 794-800 `if retries >= self._max_retries:` ... `\"status\": AgentStatus.RATE_LIMITED` (795) + `log_error(..., error_type=\"rate_limit\", severity=\"blocking\", description=f\"Rate limited after {retries} retries\")` (796-799); otherwise `wait_secs = self._backoff_wait(retries)` (801), `time.sleep(wait_secs)` (807), `retries += 1` (808), `continue` (809). `_backoff_wait` = `self._base_backoff * (2 ** attempt) + random.uniform(0, 5)` (900); `max_retries: int = 3` / `base_backoff: float = 30.0` (89-90). tmux: `_wait_tmux` is 669-761 and contains NO call to `_detect_rate_limit`, `_backoff_wait`, `_read_tail`, and never assigns `AgentStatus.RATE_LIMITED` \u2014 zero rate-limit handling.\n\n(d) WHAT A NUDGE CAN PHYSICALLY BE. tmux: yes, reusable \u2014 `tmux load-buffer str(prompt_file)` (255-258), `tmux paste-buffer -t pane_id` (259-262), `time.sleep(1)` (266), `tmux send-keys -t pane_id Enter` (267-270); all pane-targeted, plus a full retry copy in `_verify_prompt_submitted` (386-398) that already re-pastes into a LIVE pane after startup. Not factored into a helper. Headless: no \u2014 `Popen(cmd, stdout=stdout_fh, stderr=stderr_fh, text=True, env=env)` (462-463) does not set stdin (child inherits the parent terminal, not a pipe), and the prompt is one-shot argv `cmd.extend([\"-p\", prompt])` (450) under `--print` (436). Remediation in headless = kill (824-861) + relaunch only.\n\n(e) COMMS. `self._comms` is set in `__init__` (92) and already used for both event kinds from inside the loops (e.g. `log_checkpoint` at 733/787/802, `log_error` at 756). Signatures: comms.py:381-416 (`log_checkpoint(agent, phase, subtask, progress, *, current_best_metric, target_metric, blockers)`) and comms.py:344-379 (`log_error(agent, error_type, severity, description, *, affected_artifacts, resolution, escalated_to)`); severity vocabulary comms.py:61-67. Events land in `zo_root/logs/comms/.jsonl` (cli.py:1095-1098) written under `fcntl.flock` (comms.py:200-209), and the CLI tails that same file each poll and renders checkpoint/error/decision/gate (cli.py:869-913) \u2014 so watchdog events are visible live with no CLI change; `message` events are NOT rendered (no branch in 891-913).\n\n(f) SURROGATE PID LIVENESS. `surrogate.py:279-290` `_pid_alive` = `os.kill(pid, 0)` (ProcessLookupError->False, PermissionError->True, OSError->False); consumers `sweep_locks` (293-311, prunes dead + corrupt locks) and `live_sessions` (314-321). No psutil, no start time \u2014 `register_session` records `\"started_at\": datetime.now(UTC).isoformat()` (261), i.e. lock-write time, so recycled PIDs are indistinguishable there too. Wired into the launch flow at cli.py:763 (`live_sessions(..., exclude_pid=os.getpid())`), 789-794 (register) and 953 (deregister).\n\nTEST PATTERN (copy this for watchdog tests): see the `reusable` entry for tests/unit/test_wrapper.py:543 \u2014 `@mock.patch(\"zo.wrapper.time.sleep\")` decorator + `mock.patch.object(LifecycleWrapper, \"_tmux_pane_alive\"/\"_tmux_claude_running\"/\"_kill_tmux_window\"/\"_capture_tmux_pane\", ...)` on the class (they are @staticmethods) with `side_effect=[...]` scripting one entry per poll, `mock.patch.object(wrapper, \"monitor_team\", ...)` on the instance, and either a scripted dead sequence or `timeout=-1` to terminate the loop; assert `call_count` to pin the grace/confirm arithmetic." + }, + { + "surface": "Heartbeat write surface for WS-C. Hooks are the only per-tool-call wake source; they run as `python3 -m zo.hookkit ` via one thin bash shim (`.claude/hooks/zo-hookkit.sh`), fail-open, dispatched from `.claude/settings.json`. Empirical identity evidence comes from `logs/hook-trace-*.jsonl` (written by `hookkit._trace`, 139 real invocations across 6 days).\n\nWIRED HOOK EVENTS (.claude/settings.json) \u2014 every entry:\n- SessionStart / matcher \"\" -> `bash .claude/hooks/session-start.sh` (:37-49, timeout 10)\n- PreToolUse / matcher \"Bash\", `if: Bash(git commit *)` -> `pre-commit-validate.sh` (:50-62, timeout 30)\n- PreToolUse / matcher \"Write|Edit\" -> `zo-hookkit.sh sealed-paths` (:63-72, timeout 10)\n- PostToolUse / matcher \"Write|Edit\" -> `cascade-reminder.sh` (:74-85, timeout 5) <-- the ONLY PostToolUse entry today; not routed to hookkit\n- Stop / matcher \"\" -> `stop-check.sh` then `zo-hookkit.sh drift-guard` (:86-102)\n- SubagentStop / matcher \"\" -> `zo-hookkit.sh subagent-stop` (:103-114, timeout 15)\n- PreCompact / matcher \"\" -> `zo-hookkit.sh precompact` (:115-127)\n- SessionEnd / matcher \"\" -> `zo-hookkit.sh session-end` (:128-139)\n- PostToolUseFailure / matcher \"\" -> `zo-hookkit.sh post-tool-failure` (:140-151, timeout 10)\n\n(a) IDENTITY PER EVENT (measured, not inferred), from logs/hook-trace-*.jsonl aggregated:\n- `subagent-stop` (SubagentStop): 11/57 lines carry `agent_id`+`agent_type`+`agent_transcript_path`; e.g. logs/hook-trace-2026-08-12.jsonl:3 (agent_type resolved to \"Explore\") and :57 (\"workflow-subagent\"). Confirms DECISION_LOG.md:1263.\n- `post-tool-failure` (PostToolUseFailure): 2/25 lines carry `agent_id`+`agent_type` \u2014 logs/hook-trace-2026-08-12.jsonl:55 (\"workflow-subagent\") and logs/hook-trace-2026-08-17.jsonl (last line, \"Explore\"). Full key set: agent_id, agent_type, cwd, duration_ms, effort, error, hook_event_name, is_interrupt, permission_mode, prompt_id, session_id, tool_input, tool_name, tool_use_id, transcript_path. This is the tool-lifecycle payload family, so PostToolUse almost certainly carries the same identity \u2014 but PostToolUse is NOT routed to hookkit, so it is UNVERIFIED.\n- `sealed-paths` (PreToolUse): 34/34 lines carry NO agent_id/agent_type and `agent_identity` is null in every one (e.g. logs/hook-trace-2026-08-17.jsonl line 2 of the tail). Key set is cwd, effort, hook_event_name, permission_mode, prompt_id, session_id, tool_input, tool_name, tool_use_id, transcript_path. PreToolUse identity is UNCONFIRMED/absent in all observed samples.\n- `drift-guard` (Stop): 7/7 no identity; carries last_assistant_message, stop_hook_active, transcript_path, background_tasks, session_crons.\n- `session-end` (SessionEnd): 9/9 no identity; carries `reason` + transcript_path.\n`hookkit._agent_name` (src/zo/hookkit.py:112-117) reads only (\"agent_name\",\"agent_type\",\"subagent_type\",\"name\") \u2014 it never reads `agent_id`, so the unique-instance id present in live payloads is discarded.\n\n(b) RECOMMENDED INSERTION: new handler `heartbeat` in `_HANDLERS` (src/zo/hookkit.py:382-389) + a new PostToolUse settings.json block with matcher `\"*\"` (or \"\") -> `bash .claude/hooks/zo-hookkit.sh heartbeat 2>/dev/null || exit 0`, timeout 5. Keep the existing Write|Edit cascade-reminder entry untouched (add a sibling object in the PostToolUse array). Also call the same writer from `subagent-stop` and `drift-guard` (both already wired) so agent-terminal and turn-boundary ticks are covered for free. Env available to hooks: ZO_REPO_ROOT (exported by the shim, .claude/hooks/zo-hookkit.sh:29), and ZO_MEMORY_ROOT / ZO_DELIVERY_ROOT / ZO_CONTRACTS_PATH assembled in src/zo/cli.py:1076-1078 and passed through `_launch_and_monitor(extra_env=...)` (cli.py:1163) -> `launch_lead_session` (wrapper.py:143-151) -> `_launch_headless` env.update (wrapper.py:458-463). Also available: ZO_HOOK_TRACE (hookkit.py:91), ZO_DRIFT_GUARD (hookkit.py:197), ZO_FAILURE_FEED_DIR (hookkit.py:290).\n\n(c) MEMORY-ROOT RESOLUTION: `_memory_root(repo_root)` src/zo/hookkit.py:64-69 \u2014 (1) `$ZO_MEMORY_ROOT` if set, else (2) `/memory/zo-platform` if it is a dir, else (3) None. `repo_root` = `$ZO_REPO_ROOT` else `os.getcwd()` (hookkit.py:60-61). Proposed path `/heartbeats/.json` lands correctly under the existing per-project root (MemoryManager default is `/memory/`, src/zo/memory.py:122) \u2014 no new root. SEALED-PATHS GUARD WILL NOT DENY IT: `_handle_sealed_paths` (hookkit.py:328-379) is reached only from the PreToolUse Write|Edit hook (settings.json:63-72) and inspects `tool_input.file_path`; a hook writing with plain Python `open()`/`os.replace` never passes through the tool layer, so no PreToolUse fires and no deny is possible. Confirmed by design and by `_SEALED_DEFAULTS` (hookkit.py:47-50) containing no directory that a python write consults.\n\n(d) TEAM-STATUS AS SECONDARY LIVENESS: `monitor_team` (wrapper.py:481-497) fuses `_read_team_config` (`~/.claude/teams//config.json`, wrapper.py:1011-1026) and `read_task_list` (`~/.claude/tasks//*.json`, wrapper.py:499-512). Both are pure filesystem reads, already called once per poll cycle from `_wait_tmux` (wrapper.py:742,750) and `_wait_headless` (wrapper.py:812). Usable as a coarse secondary signal (task status deltas / member.status), but the models carry no timestamp (TeamMember/TeamStatus, src/zo/_wrapper_models.py:41-58) so freshness must be derived from file mtime, and `is_active` is True whenever `tasks_total == 0` (wrapper.py:496) \u2014 i.e. a team that never created tasks always reads \"active\". Not a substitute for heartbeats.\n\n(e) EXISTING COMMS EVENTS: five types only \u2014 message/decision/gate/error/checkpoint (src/zo/comms.py:31-38). `CheckpointEvent` (comms.py:152-161) is the closest fit but is emitted only by the wrapper itself (wrapper.py:470,625,662,733,786,802) and by orchestrator gate paths \u2014 never by agents on idle. `_prompt_coordination` (src/zo/orchestrator.py:1809-1838) instructs agents to use TeamCreate/Agent/SendMessage, log to DECISION_LOG.md and update STATE.md; it contains NO idle/checkpoint/heartbeat instruction. There is no agent-emitted idle event to reuse \u2014 heartbeats must be hook-driven, not prompt-driven (which is also the right call: prompt-driven heartbeats fail exactly when the agent stalls).\n\n(f) NEVER-BLOCK SIGNALS available today: `is_interrupt` (bool) in the PostToolUseFailure payload = user-abort signal, present in every live trace of that event (logs/hook-trace-2026-08-12.jsonl:40,55) \u2014 but `_handle_post_tool_failure` (hookkit.py:287-308) DROPS it. Failure-feed schema written to `/failures-.jsonl`: `{\"event_id\": uuid4, \"event_type\": \"error\", \"timestamp\": iso8601, \"session_id\": data[\"session_id\"]|\"unknown\", \"tool_name\": data[\"tool_name\"]|\"unknown\", \"error\": str(data[\"error\"] or data[\"tool_response\"])[:2000], \"input_preview\": json.dumps(data[\"tool_input\"])[:500]}` (hookkit.py:294-302). Live records exist: logs/comms/failures-2026-08-17.jsonl (2 lines, e.g. `\"Exit code 1\\n(eval):1: no matches found: tests/e2e/*.py\"`). Rate-limit text detection already exists as `_RATE_LIMIT_PATTERNS` = 429 | rate.?limit | overloaded | too many requests (wrapper.py:51-56, matcher wrapper.py:894-896) but is applied ONLY to the headless stdout tail. Stop payload carries `transcript_path` + `stop_hook_active` (drift-guard trace, and consumed at hookkit.py:197,205-207) \u2014 transcript is the only place context-limit/auth-error text could be recovered; nothing reads it for that purpose today. There is NO context-limit or auth-error detection anywhere in src/zo (grep for context.limit/usage limit/authentication_error/invalid_api_key returns only unrelated prose at plan.py:157).", + "integration_points": [ + { + "ref": "src/zo/hookkit.py:382", + "what": "_HANDLERS dispatch dict \u2014 the single registration point for a new hook event; main() (hookkit.py:392-406) rejects any argv[0] not in this dict and always returns 0", + "action": "Add \"heartbeat\": _handle_heartbeat. Keep the fail-open contract (bare except at :402) and let _trace fire for it automatically at :405." + }, + { + "ref": ".claude/settings.json:74", + "what": "PostToolUse array currently holds ONE object: matcher \"Write|Edit\" -> cascade-reminder.sh, timeout 5. This is the only PostToolUse wiring in the repo", + "action": "Append a SECOND object to this array with matcher \"*\" (or \"\") -> `bash .claude/hooks/zo-hookkit.sh heartbeat 2>/dev/null || exit 0`, timeout 5. Do not modify the Write|Edit entry." + }, + { + "ref": "src/zo/hookkit.py:112", + "what": "_agent_name(data) scans (\"agent_name\",\"agent_type\",\"subagent_type\",\"name\") and returns a role string; it never reads agent_id even though live subagent payloads carry it", + "action": "Add a sibling _agent_identity(data) -> (agent_type, agent_id) used by the heartbeat writer so the filename keys on the stable unique instance (agent_id) with agent_type as a field. Leave _agent_name untouched \u2014 subagent-stop/sealed-paths contract-matching depends on its exact return shape (see contracts lookup at hookkit.py:355-356)." + }, + { + "ref": "src/zo/hookkit.py:64", + "what": "_memory_root fallback chain: $ZO_MEMORY_ROOT -> /memory/zo-platform if is_dir() -> None. repo_root from $ZO_REPO_ROOT else cwd (hookkit.py:60)", + "action": "Reuse verbatim for the heartbeat path: /heartbeats/.json; return early (no heartbeat) when it yields None, matching the precompact/session-end pattern at :236 and :260." + }, + { + "ref": "src/zo/hookkit.py:287", + "what": "_handle_post_tool_failure \u2014 the only handler that already writes a JSONL feed; its record drops `is_interrupt` (user abort) and `agent_id`/`agent_type` even though live payloads carry all three", + "action": "Extend the record with \"is_interrupt\": bool(data.get(\"is_interrupt\")), \"agent_type\", \"agent_id\", and a classified \"never_block_reason\" field. This is the cheapest place to materialize the never-block taxonomy \u2014 the feed is already wired and already fires on real nonzero-exit Bash." + }, + { + "ref": "src/zo/wrapper.py:776", + "what": "_wait_headless poll loop head \u2014 runs every poll_interval (default 10.0s, cli.py:920 passes no override); _check_gate_mode_change() at :777 is the existing precedent for a per-cycle side-check", + "action": "Insert the watchdog checker call here (and the twin at wrapper.py:709/710 in _wait_tmux) as self._check_watchdog(process). Mirror _check_gate_mode_change's shape: read-only, contextlib-suppressed, emits through self._comms." + }, + { + "ref": "src/zo/wrapper.py:793", + "what": "Rate-limit branch: on pattern match it sleeps _backoff_wait(retries) and `continue`s the loop, capped by _max_retries, then returns AgentStatus.RATE_LIMITED. This is the retry-loop the plan wants replaced by wait-and-resume", + "action": "Replace with a pause/resume state: record a paused_until timestamp (parsed reset time when available, else backoff), suppress all watchdog nudges while paused, and resume without consuming a retry. AgentStatus (src/zo/_wrapper_models.py:15-23) already has RATE_LIMITED \u2014 add PAUSED or reuse it as a non-terminal state." + }, + { + "ref": "src/zo/wrapper.py:709", + "what": "_wait_tmux poll loop \u2014 the DEFAULT path for `zo build` (use_tmux = not no_tmux, cli.py:798). It has NO rate-limit handling at all: _detect_rate_limit is only called from _wait_headless:793", + "action": "Add the watchdog checker AND rate-limit/never-block detection here, sourcing text from _capture_tmux_pane (wrapper.py:997-1007, already used at :751). Without this, oracle check #12 cannot pass on the default launch path." + }, + { + "ref": "src/zo/wrapper.py:_wrapper_models.py:26", + "what": "LeadProcess model carries pid + started_at but no process start-time (kernel-level) identity, so a recycled PID is indistinguishable from the original", + "action": "Add proc_start_time (from psutil.Process(pid).create_time() or `ps -o lstart=`) captured at _launch_headless (wrapper.py:465-469) and compare before ANY kill/nudge. Plan explicitly requires PID+start-time identity." + }, + { + "ref": "src/zo/cli.py:1076", + "what": "extra_env assembly: ZO_MEMORY_ROOT / ZO_DELIVERY_ROOT / ZO_CONTRACTS_PATH set from memory.memory_root and target.target_repo, immediately after _load_project_context", + "action": "Add ZO_HEARTBEAT_DIR (or let the handler derive it from ZO_MEMORY_ROOT) plus a ZO_SESSION_ID matching the CommsLogger session_id minted at cli.py:1059 \u2014 the hook currently has no way to correlate its heartbeat with the wrapper's comms session." + }, + { + "ref": "src/zo/cli.py:920", + "what": "wrapper.wait_for_completion(process, on_status=_print_status, gate_mode_file=..., project_name=..., delivery_repo=...) \u2014 called with NO poll_interval and NO timeout, so the loop is 10s/cycle and unbounded", + "action": "Read-only dependency for the watchdog trigger, but note oracle check #11 says \"within one poll cycle\": at 10s that is satisfiable. If Phase 3 adds watchdog config, thread stall_threshold_min here rather than changing the default poll_interval." + }, + { + "ref": "src/zo/project_config.py:43", + "what": "ProjectConfig pydantic model (project_name, alias, workflow_mode, branch, agent_working_dirs, zo_only_paths, git_author_name/email, enforce_isolation) \u2014 specs/watchdog.md:60-70 specifies a `watchdog:` block here", + "action": "Add a nested WatchdogConfig model (enabled, tick/poll seconds, stall_threshold_min, nudge_budget, never_block list). Defaults must keep existing .zo/config.yaml files parsing unchanged." + }, + { + "ref": "src/zo/experiment_loop.py:205", + "what": "evaluate_loop_state(registry: ExperimentRegistry, phase: str, policy: LoopPolicy|None) \u2014 the Phase-2 deferral: it derives verdicts from the experiment registry, not the ledger. Called from orchestrator.py:1276 and from tests/unit/test_experiment_loop.py", + "action": "Add an optional memory_root/ledger parameter (keyword-only, defaulted) rather than changing the positional signature \u2014 DECISION_LOG.md:1277 records that the signature change touches ~20 test call sites, which is why it was deferred." + }, + { + "ref": "src/zo/orchestrator.py:687", + "what": "get_current_phase resolution order GATED (:711) > ACTIVE (:715) > PENDING-with-deps (:720-726), with the PR-036/PR-037 rationale written into the docstring at :688-706", + "action": "Session-restore cutover must preserve this exact precedence when reading from plan-ledger.json instead of STATE.md. The ledger has phase_status (ledger.py:239 set_phase_status) but no equivalent precedence logic \u2014 port the three-branch order, don't re-derive it." + }, + { + "ref": "src/zo/ledger.py:81", + "what": "_atomic_write(path, text) via tempfile.mkstemp + os.replace, with cleanup on failure; LEDGER_FILENAME at ledger.py:51 and _mutate read-modify-write at ledger.py:173", + "action": "Read-only dependency \u2014 reuse _atomic_write (or copy the 9-line pattern) for heartbeat writes so a poll-loop reader never sees a torn JSON file. DECISION_LOG.md:1275 records a torn-read bug already fixed once this way in contracts.set_active_phase." + }, + { + "ref": "src/zo/comms.py:381", + "what": "CommsLogger.log_checkpoint(agent, phase, subtask, progress, *, current_best_metric, target_metric, blockers) writing CheckpointEvent (comms.py:152-161) to logs/comms/.jsonl with fcntl.flock (comms.py:200-209)", + "action": "Emit watchdog nudge/escalation/pause events through this existing bus (specs/watchdog.md:76 requires it) rather than inventing a channel. Note log_error (comms.py:344) is already used by the wrapper for rate_limit/timeout/session_killed." + } + ], + "hazards": [ + { + "ref": ".gitignore:33", + "hazard": "memory/* is ignored but `!memory/zo-platform/` re-includes it, so `memory/zo-platform/heartbeats/*.json` would be GIT-TRACKED. Verified: `git check-ignore -v memory/zo-platform/heartbeats/lead.json` returns nothing (not ignored), while logs/heartbeats/x.json matches .gitignore:27. A file rewritten on every tool call would produce permanent `git status` churn in the ZO repo, pollute every `git diff HEAD` (which drift-guard runs at hookkit.py:182-184), and get swept into agent commits.", + "mitigation": "Add `memory/zo-platform/heartbeats/` (and contracts.json / plan-ledger.json \u2014 see next hazard) to .gitignore in the SAME PR that writes the first heartbeat, before any heartbeat file exists." + }, + { + "ref": "src/zo/contracts.py:10", + "hazard": "The contracts.py docstring asserts machine state in memory_root is \"gitignored\" \u2014 this is FALSE for the ZO platform's own memory root. `git check-ignore` confirms memory/zo-platform/contracts.json and memory/zo-platform/plan-ledger.json are not ignored either. Phase 3 will inherit this wrong assumption if it takes the docstring at face value.", + "mitigation": "Treat the docstring as stale; verify with git check-ignore before choosing a control-file location. Fix the .gitignore for all three (contracts.json, plan-ledger.json, heartbeats/) at once." + }, + { + "ref": "logs/hook-trace-2026-08-12.jsonl:2", + "hazard": "PreToolUse payloads carry NO agent identity in 34/34 observed live invocations (agent_identity null in every sealed-paths trace line). The per-agent off-limits branch in _handle_sealed_paths (src/zo/hookkit.py:351-368) is gated on `_agent_name(data) is not None`, so that entire Phase-1 enforcement path has likely NEVER fired in production \u2014 only the sealed-prefix branch has. Building heartbeats on PreToolUse would inherit the same identity blindness.", + "mitigation": "Do not put the heartbeat on PreToolUse. Put it on PostToolUse (same payload family as PostToolUseFailure, which does carry agent_id+agent_type). Separately, flag the sealed-paths off-limits gap: it needs a fallback identity source (transcript_path/agent_transcript_path or a session->agent map maintained by the heartbeat writer)." + }, + { + "ref": "src/zo/hookkit.py:112", + "hazard": "_agent_name returns agent_type (the ROLE, e.g. \"Explore\", \"workflow-subagent\") not agent_id (the INSTANCE). Two concurrent subagents of the same role would write to the same .json and overwrite each other's heartbeats \u2014 a stalled instance would look alive because its sibling keeps ticking. This is the file-level analogue of the recycled-PID problem the plan explicitly calls out.", + "mitigation": "Key the heartbeat filename on agent_id (present in live SubagentStop/PostToolUseFailure payloads) with agent_type as a field inside; fall back to session_id when agent_id is absent (main-session turns)." + }, + { + "ref": ".claude/hooks/zo-hookkit.sh:21", + "hazard": "`[[ -d \"$REPO_ROOT/src/zo\" ]] || exit 0` \u2014 the shim silently no-ops unless it resolves to the ZO platform repo, and REPO_ROOT is derived from the SCRIPT's own location (:16-17), not the session cwd. No hooks are installed into delivery repos (grep for 'hooks' in src/zo/scaffold.py returns nothing). Heartbeats therefore exist ONLY for sessions whose .claude/ is the platform repo's.", + "mitigation": "This happens to work for `zo build` because launch_lead_session is called with cwd=str(zo_root) (src/zo/cli.py:801) and the delivery repo is added via --add-dir. But any agent session started directly inside a delivery repo produces zero heartbeats. State this constraint explicitly in the watchdog design, or the checker will read staleness as a stall for a perfectly healthy delivery-repo session." + }, + { + "ref": "src/zo/wrapper.py:793", + "hazard": "Rate-limit handling exists ONLY in _wait_headless. The default `zo build` path is tmux (_wait_tmux, wrapper.py:669-761, selected at cli.py:798 via use_tmux = not no_tmux) and has no _detect_rate_limit call at all. Oracle check #12 (rate-limit pause auto-resumes) cannot be demonstrated on the default path as the code stands.", + "mitigation": "Implement wait-and-resume in a shared helper called from BOTH loops, sourcing text from _capture_tmux_pane (wrapper.py:997) in the tmux case and _read_tail (wrapper.py:1029) in the headless case." + }, + { + "ref": "src/zo/wrapper.py:51", + "hazard": "_RATE_LIMIT_PATTERNS includes a bare `429` regex matched against the last 100 lines of stdout (_read_tail default, wrapper.py:1029). Any agent output containing the substring 429 \u2014 a test count, a byte offset, a git sha fragment, a metric \u2014 triggers a false rate-limit pause. Under wait-and-resume this becomes a false indefinite stall instead of a bounded retry.", + "mitigation": "Tighten to anchored forms (HTTP 429, status 429, 'rate limit', 'usage limit reached') and require the match to be in a fresh tail slice (track last-seen offset), not in scrollback that was already scanned in a prior cycle." + }, + { + "ref": "src/zo/wrapper.py:496", + "hazard": "TeamStatus.is_active = `len(tasks) == 0 or in_progress > 0 or pending > 0` \u2014 a team with zero task files always reports active. read_task_list (wrapper.py:499-512) silently swallows JSONDecodeError/OSError per file (:510-511), so a corrupt task dir also reads as zero tasks and therefore 'active'.", + "mitigation": "Never let team status alone clear a stall verdict. Use it only as corroboration alongside heartbeat mtime + PID/start-time liveness." + }, + { + "ref": "src/zo/hookkit.py:96", + "hazard": "_trace resolves its directory from _repo_root() = $ZO_REPO_ROOT else os.getcwd(). Under pytest the cwd is the real repo, so tests have written real trace lines into logs/ \u2014 the 7 `explode` entries in logs/hook-trace-*.jsonl come from tests/unit/test_hookkit.py:353-359, which does not set ZO_REPO_ROOT. A heartbeat handler with the same fallback would let tests write heartbeat files into memory/zo-platform/ (which, per hazard 1, is git-tracked).", + "mitigation": "Make the heartbeat writer refuse to write when ZO_MEMORY_ROOT is unset AND the derived default is the platform root, or require ZO_HEARTBEAT_DIR explicitly; and monkeypatch.setenv both ZO_REPO_ROOT and ZO_MEMORY_ROOT in every new test (the TestTrace tests at test_hookkit.py:366-399 already model this correctly \u2014 TestDispatcher does not)." + }, + { + "ref": "src/zo/hookkit.py:47", + "hazard": "_SEALED_DEFAULTS = (gate_mode, gate_nonce, gate_decision, contracts.json, plan-ledger.json, sealed_paths) \u2014 a new heartbeats/ directory is NOT in it. An agent could Write/Edit its own heartbeat file through the Write tool and forge liveness, defeating the watchdog. This is exactly the anti-Goodhart failure the sealed-paths hook exists to prevent.", + "mitigation": "Add \"heartbeats\" to _SEALED_DEFAULTS in the same PR. The prefix match at hookkit.py:342 (`resolved.startswith(anchor.rstrip('/') + '/')`) already handles directory sealing, so one tuple entry seals the whole tree. Hook-internal python writes are unaffected \u2014 they never traverse PreToolUse." + }, + { + "ref": "src/zo/hookkit.py:294", + "hazard": "The failure-feed record drops `is_interrupt`, `agent_id`, and `agent_type`, all of which ARE present in live PostToolUseFailure payloads (logs/hook-trace-2026-08-12.jsonl:55). The never-block taxonomy's user-abort signal is being thrown away at the only place it is currently observable.", + "mitigation": "Add those three keys to the record before building the watchdog's never-block classifier on top of the feed." + }, + { + "ref": "memory/zo-platform/DECISION_LOG.md:1265", + "hazard": "The recorded caveat 'PostToolUseFailure did not fire for nonzero-exit Bash commands' is now contradicted by evidence: logs/comms/failures-2026-08-17.jsonl contains two records with error text 'Exit code 1\\n...'. Phase 3 planning that trusts this caveat would wrongly conclude the failure feed is too sparse to carry never-block signals.", + "mitigation": "Re-verify empirically (the feed is live and cheap to read) before designing a redundant PostToolUse-with-error-inspection path; update the DECISION_LOG entry." + }, + { + "ref": ".claude/settings.json:74", + "hazard": "A PostToolUse hook with matcher \"*\" fires on EVERY tool call including every Read/Grep/Glob. At ~1-3 calls/second in a hot loop this spawns a bash+python3 process per call. The shim prefers .venv/bin/python3 (zo-hookkit.sh:26) and pays full interpreter + pydantic import cost via `from zo.contracts import ...` at hookkit.py:34 \u2014 measured hook timeouts elsewhere in settings.json are 5-30s, implying non-trivial latency already.", + "mitigation": "Make the heartbeat handler return before any heavy import (move the zo.contracts import inside _handle_subagent_stop / _handle_sealed_paths, matching the lazy-import style already used at hookkit.py:226,238,262), and debounce: skip the write if the existing heartbeat mtime is younger than N seconds." + }, + { + "ref": "specs/watchdog.md:40", + "hazard": "The spec's section 3.1 design (a cron/self-invoke tick prompting the ORCHESTRATOR agent) directly contradicts the Phase-3 brief (external checker in the LifecycleWrapper poll loop, NOT a monitor agent). Section 4 (:74-79) also assigns ownership to orchestrator.py and lead-orchestrator.md.", + "mitigation": "Treat specs/watchdog.md sections 3.1 and 4 as superseded by plans/zo-v2-rearchitecture.md:107-114 and update the spec in the same PR \u2014 otherwise the next reader implements the agent-owned tick the plan explicitly rejects." + } + ], + "reusable": [ + { + "ref": "src/zo/ledger.py:81", + "what": "_atomic_write(path, text): tempfile.mkstemp in the target dir + os.replace, with unlink-on-failure. Exactly the durability primitive heartbeats need (poll-loop reader must never see a torn file). DECISION_LOG.md:1275 records this pattern being retrofitted to contracts.set_active_phase after a torn-read bug." + }, + { + "ref": "src/zo/comms.py:200", + "what": "CommsLogger._write_event: append + fcntl.flock(LOCK_EX) around the write. The concurrency primitive for any multi-writer JSONL (nudge log, watchdog event stream)." + }, + { + "ref": "src/zo/hookkit.py:81", + "what": "_trace(): the fail-open observability pattern \u2014 contextlib.suppress(OSError), mkdir(parents=True, exist_ok=True), date-rotated JSONL, env kill-switch (ZO_HOOK_TRACE=0). Copy this shape for the heartbeat writer verbatim; it is the reason live identity evidence exists at all." + }, + { + "ref": "src/zo/hookkit.py:392", + "what": "main() dispatch: reset module state, look up _HANDLERS, bare-except around the handler, always return 0, trace on both paths. New handlers get fail-open for free by registering here." + }, + { + "ref": "src/zo/wrapper.py:648", + "what": "_check_gate_mode_change(): the canonical 'cheap read-only side-check called once per poll cycle, logs only on state change' template. The watchdog checker should be structurally identical (getattr-guarded state on self, OSError-suppressed read, comms emit on transition only)." + }, + { + "ref": "src/zo/wrapper.py:685", + "what": "_STARTUP_GRACE_POLLS (wrapper.py:74) + _DEAD_CONFIRM_POLLS (:78) + _DEAD_RECHECK_INTERVAL (:81) and the docstring at :677-701 explaining why a single negative reading must never tear down a healthy session. The watchdog's stall verdict needs exactly this debounce discipline \u2014 reuse the constants and the rationale rather than re-deriving them." + }, + { + "ref": "tests/unit/test_hookkit.py:32", + "what": "_run(event, payload, monkeypatch, capsys): patches sys.stdin with io.StringIO(json.dumps(payload)), asserts hookkit.main([event]) == 0, parses stdout or returns None. The in-process unit pattern for every handler." + }, + { + "ref": "tests/integration/test_hooks_shim.py:29", + "what": "_run_shim(event, payload, env_overrides): subprocess.run(['bash', SHIM, event], input=json.dumps(payload), capture_output=True, text=True, timeout=30, env={**os.environ, **overrides}, cwd=REPO_ROOT, check=False) -> (returncode, stdout). The subprocess+stdin-JSON end-to-end pattern the brief asked for; drives the real bash script exactly as Claude Code does." + }, + { + "ref": "tests/integration/test_hooks_shim.py:116", + "what": "TestSettingsWiring: asserts every hookkit event name actually appears in .claude/settings.json commands ('unwired mechanisms are the #1 anti-pattern from the v2 review'). Extend test_all_ws_a_events_wired (:120) with the heartbeat/PostToolUse wiring so the new hook cannot ship unwired." + }, + { + "ref": "tests/unit/test_wrapper.py:504", + "what": "@mock.patch('zo.wrapper.time.sleep') + mock_proc.poll.side_effect=[None, 0] + a seeded stdout log \u2014 the deterministic poll-loop test harness. Oracle checks #11 (induced stall detected within one poll cycle) and #12 (rate-limit pause auto-resumes) can both be built on this without real timing." + }, + { + "ref": "src/zo/hookkit.py:314", + "what": "_sealed_prefixes(memory_root): merges _SEALED_DEFAULTS with the user-editable memory_root/sealed_paths file (one prefix per line, # comments skipped), and the prefix match at :342 seals whole subtrees. Adding 'heartbeats' to the tuple at :47 is a one-line change that seals the heartbeat dir against agent Write/Edit." + }, + { + "ref": "src/zo/orchestrator.py:687", + "what": "get_current_phase's GATED > ACTIVE > PENDING-with-deps-met branch structure plus its docstring rationale (:688-706) and the BLOCKED exclusion. The session-restore cutover must reproduce this precedence against the ledger; PRIORS.md:1042 (PR-036, parse-time validation) and PRIORS.md:1089 (PR-037, resume precedence) carry the enforced rules and the exact test names that lock them." + }, + { + "ref": "src/zo/wrapper.py:499", + "what": "read_task_list / monitor_team / _read_team_config (wrapper.py:1011): already-implemented, already-per-poll-cycle filesystem readers of ~/.claude/tasks//*.json and ~/.claude/teams//config.json. Free secondary liveness input; no new I/O needed." + } + ], + "open_questions": [ + "Does PostToolUse (success) actually carry agent_id/agent_type for subagents? It is strongly implied \u2014 PostToolUseFailure shares the payload family and carries them (logs/hook-trace-2026-08-12.jsonl:55, logs/hook-trace-2026-08-17.jsonl last line) \u2014 but PostToolUse is not routed to hookkit today, so there is ZERO direct evidence. Cheapest resolution: wire `zo-hookkit.sh heartbeat` on PostToolUse with a trace-only no-op handler first, run one live session with a subagent, then read the stdin_keys. This is the same in-session verification method DECISION_LOG.md:1261 used for Phase 1.", + "Are the 34 identity-free PreToolUse samples explained by 'no subagent ever performed a Write/Edit' (the observed subagents were Explore/workflow-subagent, largely read-only), or does PreToolUse genuinely omit identity? This determines whether the sealed-paths off-limits branch (hookkit.py:351-368) is dead code in production or merely unexercised.", + "Where does the process-start-time come from on macOS + Linux without adding a psutil dependency? pyproject deps were not audited here. `ps -o lstart= -p ` works on both but the format differs; /proc//stat field 22 is Linux-only. Oracle check #13 runs on Linux, day-to-day dev is Darwin 25.6.0.", + "Should the heartbeat file be per-agent (.json) or a single append-only JSONL that the checker tails? Per-agent files make staleness a trivial mtime read and match specs/watchdog.md:44; a JSONL preserves history for post-mortem. The plan text says 'per-agent heartbeat JSON files', which the brief restates \u2014 but nothing in the repo commits to it yet.", + "What is the authoritative source for the rate-limit RESET time needed for wait-and-resume? _detect_rate_limit (wrapper.py:894) is a pure boolean over stdout patterns \u2014 it discards the matched text entirely, so no reset timestamp is captured anywhere today. Without a parsed reset, 'wait-and-resume' degrades to 'wait a guessed interval', which is the retry-loop the plan is trying to eliminate.", + "Does the heartbeat need to survive PreCompact? _handle_precompact (hookkit.py:233) flushes STATE.md and appends a DECISION_LOG entry; compaction can take minutes with no tool calls, which would look like a stall to a naive mtime check. Should PreCompact write a 'compacting' heartbeat state that suppresses nudges (a sixth never-block category the brief's taxonomy does not list)?", + "How do heartbeats behave for the LEAD session, which is a subprocess rather than a subagent and whose payloads carry session_id but no agent_id? Keying on session_id works but collides with the agent-instance namespace unless the writer namespaces them explicitly.", + "Does evaluate_loop_state's ledger cutover need the registry at all afterwards, or does the ledger subsume oracle_tier/result data? experiment_loop.py:230-265 reads latest.result.oracle_tier and e.status == ExperimentStatus.COMPLETE; ledger.py's LedgerFile has passes/attempts/last_failure/phase_status but no oracle tier \u2014 so a pure-ledger evaluate_loop_state may not be expressible without extending the ledger schema." + ], + "notes": "Evidence base: read src/zo/hookkit.py (410 lines, in full), .claude/hooks/zo-hookkit.sh, .claude/settings.json, src/zo/comms.py (models + logger), src/zo/wrapper.py (launch/poll/rate-limit/team sections), src/zo/cli.py:709-930 + 1040-1170, src/zo/contracts.py header, src/zo/ledger.py API surface, src/zo/orchestrator.py:687-726 + 1809-1838, src/zo/experiment_loop.py:184-265, src/zo/memory.py:105-127, src/zo/project_config.py, specs/watchdog.md (in full), specs/comms.md (grep), plans/zo-v2-rearchitecture.md:100-145, memory/zo-platform/DECISION_LOG.md:1257-1279, memory/zo-platform/PRIORS.md:1042-1152, tests/unit/test_hookkit.py, tests/integration/test_hooks_shim.py, tests/unit/test_wrapper.py:462-545.\n\nIdentity claims are derived from a programmatic aggregation over all six logs/hook-trace-*.jsonl files (139 invocations), not from spot-reading: per-event counts were subagent-stop 57, sealed-paths 34, post-tool-failure 25, session-end 9, drift-guard 7, explode 7 (test pollution). The identity table in `surface` (a) reports exact (event, has agent_id, has agent_type, resolved name) tallies from that pass.\n\nGitignore claims were verified by running `git check-ignore -v` rather than reading .gitignore: memory/zo-platform/heartbeats/lead.json, memory/zo-platform/contracts.json and memory/zo-platform/plan-ledger.json all returned NO match (tracked), while logs/heartbeats/x.json matched .gitignore:27. If Phase 3 prefers zero git surface over the plan's 'control-plane files under the memory root' rule, logs/heartbeats/ is already ignored \u2014 but that would create the new root the plan forbids, so amending .gitignore is the better trade.\n\nTwo source-of-truth conflicts worth resolving before writing code: (1) specs/watchdog.md:40 and :74-79 specify an agent/cron-owned tick, which plans/zo-v2-rearchitecture.md:107-114 and the Phase-3 brief explicitly reject in favour of an external checker in the wrapper poll loop; (2) memory/zo-platform/DECISION_LOG.md:1265 claims PostToolUseFailure does not fire for nonzero-exit Bash, but logs/comms/failures-2026-08-17.jsonl contains exactly such records ('Exit code 1\\\\n(eval):1: no matches found: tests/e2e/*.py'). Both docs should be corrected in the Phase 3 PR.\n\nHighest-leverage single finding for the build: PreToolUse carries no agent identity in 34/34 live samples, which means the per-agent off-limits enforcement shipped in Phase 1 (hookkit.py:351-368) has probably never fired in production. Phase 3 needs a durable session->agent identity map anyway for heartbeats; that same map would retroactively fix the Phase-1 gap, so it is worth designing the heartbeat writer to double as the identity registry rather than as a write-only file.\n\nNo files were created, modified, or deleted; all commands were read-only (git log/branch/check-ignore, ls, grep, sed, wc, find, and python3 -c aggregations that only read)." + }, + { + "surface": "Phase-4 experiment iteration path (experiment_loop.py + experiments.py + orchestrator.py gate/prompt path + cli.py/wrapper.py launch loop), mapped for WS-C Phase 3: watchdog in the LifecycleWrapper poll loop, fresh-context-per-iteration builder spawn, ledger input to evaluate_loop_state, and session-restore cutover.\n\nHOW AN ITERATION IS EXECUTED TODAY (the headline answer): ONE long-lived lead session per `zo build` / `zo continue` invocation. cli.py:1134 `orchestrator.get_current_phase()` -> cli.py:1143 `orchestrator.build_lead_prompt(phase)` (single call, no loop) -> cli.py:1150 `_launch_and_monitor(...)` -> cli.py:800 `wrapper.launch_lead_session(...)` -> cli.py:920 `wrapper.wait_for_completion(...)` -> cli.py:942 `orchestrator.end_session()`. There is NO `while` over phases or iterations anywhere in cli.py; `zo continue` (cli.py:1272) merely re-invokes `build` once. Iteration is therefore purely prompt-driven inside the single lead session (orchestrator.py:1408-1454 experiment section + 1494-1519 \"# Autonomous Iteration Loop\" briefing + .claude/agents/model-builder.md:86-110 auto-proposer).\n\nCRITICAL: `Orchestrator.advance_phase` (orchestrator.py:752) and `mark_subtask_complete` (orchestrator.py:848) have ZERO runtime callers. Verified: the only `orchestrator.*` calls in cli.py are start_session (1121), decompose_plan (1124), check_plan_edited (1125), get_current_phase (1134), build_lead_prompt (1143), end_session (942); no `zo phase`/`zo advance` command exists (cli.py command list: build/continue/init/gates/experiments/learnings/watch-training/report/consolidate). Every other caller of advance_phase is under tests/. Consequence: `_auto_iterate_if_needed` (orchestrator.py:1239), `evaluate_loop_state` (orchestrator.py:1276), `_finalize_experiments` (1163), `mark_phase_passed` (829), `_generate_snapshot` (832) are unreachable in a real run. Only the human path lands: `zo gates approve --nonce` writes a `gate_decision` file, consumed at the NEXT decompose by `_consume_gate_decision` (orchestrator.py:403-427) -> `apply_human_decision` (907). Phase 3's fresh-context loop is therefore not \"replacing\" a running Python loop \u2014 it is supplying the missing driver process that has never existed.", + "integration_points": [ + { + "ref": "src/zo/cli.py:1143", + "what": "`prompt = orchestrator.build_lead_prompt(phase)` \u2014 the single, one-shot prompt build. Immediately followed by cli.py:1144-1145 appending human extras and cli.py:1150 `_launch_and_monitor`.", + "action": "THE SEAM (a). Replace lines 1134-1166 with a driver loop: `while (phase := orchestrator.get_current_phase()) is not None:` { build prompt; launch+wait; then call `orchestrator.advance_phase(phase.phase_id)`; if verdict is ITERATE and phase is phase_4 -> loop again with a FRESH `launch_lead_session` (fresh context) }. Put the loop in a new `zo.experiment_loop.run_fresh_context_loop(...)` (or an orchestrator method) rather than inline in the click command, so it is unit-testable without click." + }, + { + "ref": "src/zo/cli.py:1150", + "what": "`_launch_and_monitor(...)` is a launch->wait->end_session->consolidate one-shot (definition cli.py:709-960). It calls `orchestrator.end_session()` at 942 and surrogate deregistration/consolidation at 949-960 \u2014 all of which must NOT run per-iteration.", + "action": "Split `_launch_and_monitor` into `_launch_once(...)` (lines 798-939: launch, wait_for_completion, status print) and the session-teardown tail (941-960). The fresh loop calls `_launch_once` N times and the teardown once. Note the permissions-overlay cleanup at cli.py:775-783 and surrogate `register_session` at 789-794 are also once-per-run, not per-iteration." + }, + { + "ref": "src/zo/cli.py:920", + "what": "`process = wrapper.wait_for_completion(process, on_status=_print_status, gate_mode_file=..., project_name=..., delivery_repo=...)` \u2014 the single blocking call the CLI makes into the poll loop.", + "action": "Read-only dependency for the fresh loop; for the watchdog, this is where the checker's config (heartbeat dir = memory_root, stall threshold, nudge budget) must be threaded in as new kwargs alongside the existing `gate_mode_file`/`delivery_repo` pattern." + }, + { + "ref": "src/zo/wrapper.py:709", + "what": "`_wait_tmux` poll loop: `while True:` at 709, per-poll side-effect calls `self._check_gate_mode_change()` (710) and `self._maybe_open_training_pane()` (711), then liveness checks 713-747, `on_status` 749-752, timeout 754-760, `time.sleep(poll_interval)` 761. Default poll_interval=10.0 (wrapper.py:551).", + "action": "WATCHDOG INSERTION POINT. Add `self._check_watchdog(process)` immediately after line 711, mirroring the `_maybe_open_training_pane` precedent (wrapper.py:588-634: reads state from disk, fires once, swallows its own errors). This is the tmux path \u2014 the DEFAULT path (`use_tmux = not no_tmux`, cli.py:798)." + }, + { + "ref": "src/zo/wrapper.py:776", + "what": "`_wait_headless` poll loop: `while True:` at 776; rate-limit branch at 792-809 (`output = self._read_tail(process.stdout_log)`; `if self._detect_rate_limit(output)`; backoff+retry).", + "action": "Second watchdog insertion point (after line 777). Also the ONLY place rate-limit is currently detected \u2014 see hazards." + }, + { + "ref": "src/zo/wrapper.py:894", + "what": "`_detect_rate_limit(output)` matching `_RATE_LIMIT_PATTERNS` (wrapper.py:51-56: `429`, `rate.?limit`, `overloaded`, `too many requests`) against the stdout tail.", + "action": "Reuse as the seed of the never-block taxonomy classifier, but move it out of `_wait_headless` into a shared classifier callable from both loops. Extend with context-limit / auth-error / user-abort patterns for the never-nudge set." + }, + { + "ref": "src/zo/wrapper.py:898", + "what": "`_backoff_wait(attempt) -> self._base_backoff * (2 ** attempt) + random.uniform(0, 5)` (base 30.0s, wrapper.py:90; max_retries at 794).", + "action": "This is a retry-with-backoff loop, NOT the plan's 'wait-and-resume on reset'. Add a `_wait_for_rate_limit_reset(reset_at)` that parses/derives the reset time and sleeps to it, leaving the session alive; keep `_backoff_wait` only as the fallback when no reset time is recoverable." + }, + { + "ref": "src/zo/_wrapper_models.py:26", + "what": "`LeadProcess`: `pid: int | None` (29), `started_at: datetime | None` (31), `tmux_pane_id` (37), `status` (30, AgentStatus incl. RATE_LIMITED at _wrapper_models.py:22). No process-start-time-from-OS field.", + "action": "Add a `proc_start_time: float | None` captured from the OS at launch (`psutil`-free: `ps -o lstart=/-p ` or /proc) so PID+start-time identity can reject recycled PIDs. `pid` alone is what the watchdog would otherwise key on." + }, + { + "ref": "src/zo/memory.py:125", + "what": "`MemoryManager.memory_root` property; root resolved at memory.py:119-122 (`memory_root` override or `{project_dir}/memory/{project_name}`). Control-plane files already live here: STATE.md (133), gate_mode (328), gate_nonce (350), gate_decision (375), contracts.json, plan-ledger.json (ledger.py:51).", + "action": "Heartbeat JSON dir = `memory_root/heartbeats/.json`. Plan anti-scope explicitly forbids a new state root (plans/zo-v2-rearchitecture.md:151), so reuse this root." + }, + { + "ref": ".claude/settings.json (hooks block) + .claude/hooks/zo-hookkit.sh", + "what": "Existing hook wiring: PreToolUse Write|Edit -> `zo-hookkit.sh sealed-paths`, Stop -> `drift-guard`, SubagentStop, PreCompact, SessionEnd, PostToolUseFailure. Shim is venv-preferring and fail-open; handlers dispatched in src/zo/hookkit.py:392 `main`.", + "action": "Add a `heartbeat` event to hookkit.py `main` + a PostToolUse (and SubagentStop) wiring so every agent tool call touches its heartbeat file. Uses the existing `_memory_root(repo_root)` helper (hookkit.py:64) and `_agent_name(data)` (hookkit.py:112) for per-agent identity \u2014 `_agent_name` was verified to work against live payloads per memory/zo-platform/STATE.md:11." + }, + { + "ref": "src/zo/orchestrator.py:1276", + "what": "`decision = evaluate_loop_state(registry, phase.phase_id, policy)` \u2014 the sole production call site, inside `_auto_iterate_if_needed` (1239). Registry loaded at 1270 from `_experiments_dir()`; policy resolved at 1271-1275.", + "action": "(b) Least-invasive ledger input: add a keyword-only param to `evaluate_loop_state(registry, phase, policy=None, *, ledger: LedgerFile | None = None)` (experiment_loop.py:205-209). Keyword-only + default None means all 16 existing test call sites and this one keep compiling unchanged. At this call site pass `ledger=zo_ledger.load_ledger(self._memory.memory_root / zo_ledger.LEDGER_FILENAME)` (ledger.py:93 already returns None fail-open)." + }, + { + "ref": "src/zo/experiment_loop.py:205", + "what": "`def evaluate_loop_state(registry: ExperimentRegistry, phase: str, policy: LoopPolicy | None = None) -> LoopDecision`. Body reads ONLY `registry.experiments` (230-233) and `policy`. Third arg is positional in 12 of the 16 test calls.", + "action": "(b) Add `*, ledger: LedgerFile | None = None`. Precise test call-site count: 16, ALL in tests/unit/test_experiment_loop.py (lines 102, 111, 122, 137, 145, 150, 159, 176, 185, 196, 218, 232, 243, 259, 312, 328). Zero test call sites elsewhere \u2014 tests/integration/test_auto_iteration.py drives it indirectly via `orch.advance_phase` (test_auto_iteration.py:162). A thin adapter is unnecessary; keyword-only is strictly cheaper." + }, + { + "ref": "src/zo/orchestrator.py:1310", + "what": "CONTINUE branch: `phase.status = PhaseStatus.ACTIVE` (1310), `phase.completed_subtasks.clear()` (1311), `self._ledger_safe(\"reset_phase\", phase.phase_id, f\"loop CONTINUE: {decision.reason}\")` (1312-1315), returns `GateEvaluation(decision=GateDecision.ITERATE)` (1316-1321).", + "action": "This is the state transition the fresh loop keys on. The driver should treat `GateDecision.ITERATE` on phase_4 as 'spawn a fresh builder session', and add the git-commit checkpoint right here (after reset_phase, before returning) or in the driver right after it sees ITERATE." + }, + { + "ref": "src/zo/orchestrator.py:1388", + "what": "`exp = self._ensure_experiment_for_phase(phase.phase_id)` inside `_prompt_experiment_context` \u2014 i.e. BUILDING THE PROMPT MINTS THE EXPERIMENT (side effect). `_ensure_experiment_for_phase` (1124-1161) is idempotent: returns the existing RUNNING exp for the phase (1150-1152), else mints with `parent_id = registry.latest_in_phase(phase_id).id` (1154-1160).", + "action": "Fresh-context input #1 (experiment lineage): each fresh session's prompt is built by re-calling `build_lead_prompt(phase)`, which re-mints/re-resolves the child exp. Because it is idempotent on RUNNING, the driver MUST first run `_finalize_experiments` (parent -> COMPLETE) or `_abort_running_experiments` (orchestrator.py:1356) before rebuilding the prompt, or the 'fresh' iteration silently reuses the previous exp id." + }, + { + "ref": "src/zo/orchestrator.py:1163", + "what": "`_finalize_experiments(phase)` \u2014 parses `result.md` per RUNNING exp (1220-1226), calls `update_result`, and gates on `metrics.jsonl` + `training_status.json` existence (1208-1219). Returns missing-artifact list consumed at advance_phase:798-799.", + "action": "Read-only dependency; the fresh loop must call it (via advance_phase) between iterations so `Experiment.status` flips to COMPLETE and `delta_vs_parent` is computed (experiments.py:514-525) before `evaluate_loop_state` counts." + }, + { + "ref": "src/zo/experiments.py:233", + "what": "`ExperimentRegistry.lineage(exp_id)` -> ancestor chain root-first, self-last. Plus `find` (215), `children_of` (222), `latest_in_phase` (226).", + "action": "Fresh-context input #2 (lineage digest): build the digest from `lineage()` + `render_checklist` (experiments.py:287-321, which already renders exp/parent/status/hypothesis/metric/delta/tier/top-shortfall as a table and is written to `.zo/experiments/CHECKLIST.md` on every mutation via `_safe_refresh_checklist` at 343-354). Reuse `render_checklist` rather than writing a new digest renderer." + }, + { + "ref": "src/zo/orchestrator.py:1487", + "what": "Auto-proposer prompt text instructing the builder to read `{parent_id}/result.md`, `{parent_id}/diagnosis.md`, `{parent_id}/next.md` from disk (orchestrator.py:1483-1492).", + "action": "Fresh-context input #3 (next.md): next.md is produced on disk by the model-builder agent per .claude/agents/model-builder.md:68-79 and read by the next iteration's agent directly from disk \u2014 it is NOT produced by any Python code. It survives a fresh context for free. Keep this prompt block; it is the existing fresh-context re-derivation mechanism." + }, + { + "ref": "src/zo/orchestrator.py:1784", + "what": "`_prompt_memory()` \u2014 priors digest: reads non-superseded priors (1790-1792), injects at most 8 (`for p in priors[:8]`, 1798), plus `self._semantic.query(self._plan.objective, top_k=3)` (1801-1805). Prefixed with the STATE-derived `Mode/Phase/Blockers` lines (1786-1789).", + "action": "Fresh-context input #4 (priors digest). Already re-derived from disk on every prompt build, so it works per-iteration unchanged. For the loop, consider querying the semantic index with the CURRENT hypothesis/shortfalls rather than the static `plan.objective` so each fresh iteration gets iteration-relevant recall." + }, + { + "ref": "src/zo/ledger.py:93", + "what": "`load_ledger(path) -> LedgerFile | None` (fail-open). `LedgerFile.phase_status` (ledger.py:73) and `LedgerEntry.passes/attempts/last_failure` (63-65). `summarize(doc)` (248) returns per-phase (passed, total).", + "action": "Fresh-context input #5 (ledger entry). Emit a per-iteration ledger digest into the fresh prompt (which subtasks pass, attempts, last_failure) so the fresh session re-derives 'what is done' from the ledger instead of STATE.md. `summarize` is the ready-made renderer." + }, + { + "ref": "src/zo/ledger.py:203", + "what": "`reset_phase(memory_root, phase_id, reason)` sets `passes=False`, `last_failure=reason[:500]`, `phase_status[phase]='active'`. Counterpart `mark_phase_passed` (184) and `set_phase_status` (239).", + "action": "Read-only dependency. Note `reset_phase` wipes `passes` for ALL entries of the phase on every loop CONTINUE \u2014 the fresh loop will call it once per iteration, so `attempts` is the only monotonic signal that survives; use it if the loop wants a per-subtask retry cap." + }, + { + "ref": "src/zo/orchestrator.py:687", + "what": "`get_current_phase()` \u2014 resolution order GATED (710-712) > ACTIVE (713-716) > PENDING-with-deps-met (717-726); BLOCKED intentionally excluded (704-705). This is the PR-036/PR-037 precedence.", + "action": "(deferral 3) Session-restore cutover: keep this exact precedence but source `phase.status` from `plan-ledger.json` `phase_status` instead of STATE.md. Simplest cutover: change `_restore_phase_states` (orchestrator.py:468-481) to prefer the ledger and fall back to `session_state.phase_states`." + }, + { + "ref": "src/zo/orchestrator.py:468", + "what": "`_restore_phase_states()` \u2014 reads `self._session_state.phase_states` / `completed_subtasks_by_phase` and coerces via `PhaseStatus(saved_states[...])` (478). Called from `decompose_plan` at orchestrator.py:350. Parse-time validation lives upstream in `src/zo/_memory_formats.py` (`_VALID_PHASE_STATUSES`, per PRIORS PR-036).", + "action": "(deferral 3) Cutover site. Ledger `phase_status` values are free-form strings written by ledger.py ('active'/'gated'/'completed'/'blocked') \u2014 apply the same `_VALID_PHASE_STATUSES` parse-time validation to the ledger before coercing, or PR-036's crash class reappears via a new file." + }, + { + "ref": "src/zo/orchestrator.py:293", + "what": "`_capture_phase_states()` \u2014 writes phase statuses into session_state; called ONLY from `end_session` (orchestrator.py:284).", + "action": "Hazard-adjacent: on a crash/kill there is no flush, so STATE.md lags. The ledger (written atomically on every mutation, ledger.py:81-91) is strictly more crash-safe \u2014 an argument for making it the restore source rather than a mirror." + }, + { + "ref": "src/zo/surrogate.py:360", + "what": "`commit_worktree(worktree, *, message) -> bool` \u2014 `_git(worktree, 'add', '-A')` then `_git(worktree, 'commit', '-m', message)`, returns True only if a commit was created. Underlying `_git(repo, *args)` at surrogate.py:97-101 (`subprocess.run(['git','-C',str(repo), ...])`).", + "action": "(c) REUSE THIS for git-commit-as-checkpoint. It is repo-agnostic (takes any path) despite the worktree-flavoured name; call `commit_worktree(delivery_repo, message=f'zo: checkpoint {exp.id} ({decision.verdict})')` after each iteration. Existing precedent call: src/zo/consolidate.py:178-182." + }, + { + "ref": "src/zo/surrogate.py:97", + "what": "`_git(repo, *args)` \u2014 the only git subprocess wrapper in ZO. `merge_branch` (379), `remove_worktree` (355), worktree add (196-198).", + "action": "(c) Promote `_git` to a public helper (or add `zo.git`) if the loop needs `rev-parse HEAD` for the checkpoint sha. Confirmed by grep: orchestrator.py, cli.py, scaffold.py have NO git-commit code (scaffold.py:227/244 are a .gitignore line and an apt package list); consolidate.py only re-exports surrogate's." + }, + { + "ref": "src/zo/_memory_models.py:40", + "what": "`SessionState.git_head: str | None`; populated/validated by `MemoryManager.recover_session` via `_get_git_head()` (memory.py:287-301), which raises a `git_head mismatch` blocker when the recorded head diverges from the actual repo head.", + "action": "(c) Reuse as the checkpoint ledger: after each per-iteration commit, update `session_state.git_head` so a fresh session's `recover_session` verifies it resumed at the expected checkpoint instead of raising a spurious mismatch blocker." + }, + { + "ref": "src/zo/experiment_loop.py:113", + "what": "(d) `LoopPolicy` fields: `max_iterations: int = 10` (113), `plateau_epsilon: float = 0.01` (114), `plateau_runs: int = 3` (115), `stop_on_tier = 'must_pass'` (116), `dead_end_threshold = 0.9` (117), `low_token: bool = False` (118).", + "action": "Read-only. The fresh loop must enforce `max_iterations` OUTSIDE the session (as a driver-loop counter), not just inside `evaluate_loop_state`, so a session that never reaches a gate still cannot spin forever." + }, + { + "ref": "src/zo/experiment_loop.py:130", + "what": "(d) `_LOW_TOKEN_LOOP_CLAMPS = {'max_iterations': 2, 'stop_on_tier': 'could_pass'}` applied in `resolve_policy` at 166-168 BEFORE plan overrides (170-175), then CLI `max_iterations_override` last (176-177). Precedence documented at 149-150: CLI > plan > clamp > default.", + "action": "Read-only. Mirror this exact precedence for any new fresh-loop caps (e.g. max_sessions, per-iteration wall-clock) so there is one precedence rule, not two." + }, + { + "ref": "src/zo/cli.py:262", + "what": "(d) `_LOW_TOKEN_PRESET` \u2014 `max_iterations: 2` (263), `stop_on_tier: 'could_pass'` (264), `lead_model: 'sonnet'` (262), `gate_mode: 'full-auto'` (267), `compact_threshold: '60'` (268). Threaded to the orchestrator as `max_iterations_override=max_iterations` (cli.py:1119) and `low_token=effective_low_token` (1118).", + "action": "(d) The fresh loop's session-count cap should be derived from the same preset dict (add a key) so `--low-token` clamps sessions too; otherwise low-token mode caps iterations but not spawned sessions." + }, + { + "ref": "src/zo/experiment_loop.py:261", + "what": "(d) Budget enforcement: `if completed_count >= policy.max_iterations` where `completed_count = len(completed)` (242) counts ALL COMPLETE experiments for the phase in the registry, across sessions and machines.", + "action": "Read-only, and the right semantics for a fresh loop: the cap is durable on disk, so a restart cannot reset the budget." + }, + { + "ref": ".claude/agents/lead-orchestrator.md:207", + "what": "(e) 'Coordination Rules' section (207-217). Contains NO Phase-4 iteration/loop instruction at all: no mention of experiments, exp-NNN, autonomous loop, fresh sessions, or the CONTINUE verdict. Grep for 'iterat|loop|fresh|restart' returns only line 23 (roster blurb: model-builder does 'iteration'), 175 (a STATE.md example with `iteration: 2`), 216 (session end), 230 (checklist item 'Session recovery is possible from current STATE.md').", + "action": "(e) PROMPT CHANGES NEEDED HERE: (1) add a Phase-4 section stating the session handles exactly ONE iteration then ends cleanly (the driver relaunches) \u2014 today nothing tells it that; (2) per specs/watchdog.md:77, add the standing rule 'never end a turn in passive hold-and-wait while critical-path work is outstanding; verify liveness by evidence (real PID, artifact deltas), not by absence of bad news'; (3) line 230's 'Session recovery is possible from current STATE.md' must become 'from plan-ledger.json' for the deferral-3 cutover; line 101 and 213 also make STATE.md the phase-transition record." + }, + { + "ref": ".claude/agents/model-builder.md:86", + "what": "(e) 'Autonomous iteration' block (86-110): tells the builder that when the injected prompt says `parent_id = exp-NNN` it must NOT ask the human, must read `{parent}/result.md` shortfalls, `{parent}/diagnosis.md`, `{parent}/next.md`, then write `{exp_id}/hypothesis.md`; and 108-110 explicitly reserves stop-decisions to the loop ('Your job is to execute one iteration cleanly ... not to decide when iteration ends').", + "action": "(e) This text is ALREADY fresh-context-compatible \u2014 it is a pure disk re-derivation protocol. Minimal changes: state that the session's context contains no memory of prior iterations, and that everything must come from the injected digest + the named on-disk files. Also model-builder.md:290 ('Iteration plateau: escalate to Orchestrator after 2 non-improving iterations') conflicts with the loop owning plateau (experiment_loop.py:274-296) \u2014 reconcile." + }, + { + "ref": "src/zo/orchestrator.py:525", + "what": "`build_lead_prompt(phase)` assembles sections 541-554: role, plan_context, autonomy, phase, contracts, [adaptations], roster, experiment_context, memory, coordination, low_token_overrides, gate_criteria, constraints \u2014 joined with '\\n\\n---\\n\\n'.", + "action": "(a) Fresh-session input assembly point. Add a `_prompt_ledger_digest()` section (ledger entry + lineage digest + iteration N-of-M banner) into this list; that keeps one prompt-composition path rather than a parallel fresh-loop prompt builder." + }, + { + "ref": "src/zo/contracts.py:172", + "what": "`set_active_phase(memory_root, phase_id)` \u2014 atomically updates `active_phase` in contracts.json. Called from advance_phase at orchestrator.py:789 (GATED path only).", + "action": "The fresh loop should call this on each iteration start so the sealed-paths/SubagentStop enforcement plane (WS-A) is pointed at the right phase for the newly spawned session; today it is only updated when a phase becomes GATED." + } + ], + "hazards": [ + { + "ref": "src/zo/orchestrator.py:752", + "hazard": "`advance_phase` \u2014 and with it `_auto_iterate_if_needed`, `evaluate_loop_state`, `_finalize_experiments`, `mark_phase_passed`, `_generate_snapshot`, `_generate_test_report` \u2014 has NO runtime caller. Verified by grep across src/ and .claude/: the only non-test callers are none. The 'autonomous iteration loop' is exercised exclusively by tests/integration/test_auto_iteration.py and tests/unit/test_orchestrator.py. Any Phase 3 design that assumes the Python loop currently runs in production is wrong, and any oracle check that passes only in a test harness repeats the exact anti-pattern the v2 plan bans ('nothing ships unwired', plans/zo-v2-rearchitecture.md:148).", + "mitigation": "Make the fresh-context driver the FIRST runtime caller of `advance_phase`. Add an integration test that drives `zo build` end-to-end (or a fake `launch_lead_session`) and asserts advance_phase was invoked, so the wiring cannot silently regress." + }, + { + "ref": "src/zo/wrapper.py:793", + "hazard": "Rate-limit detection exists ONLY in `_wait_headless` (line 793). `_wait_tmux` (709-761) has no rate-limit branch at all \u2014 and tmux is the DEFAULT path (`use_tmux = not no_tmux`, cli.py:798; `_launch_tmux` chosen when `self._is_in_tmux()`, wrapper.py:145). Oracle check #12 ('a rate-limit pause auto-resumes on reset') and the never-nudge rule ('a rate-limited session is NOT nudged', check #11) would pass in headless tests and fail in the real default configuration.", + "mitigation": "Lift rate-limit classification into a shared per-poll `_classify_session_state(process)` called from BOTH loops. In tmux mode there is no stdout log to tail (`process.stdout_log` is None for tmux launches) \u2014 the classifier must read `self._capture_tmux_pane(pane_id, lines=...)` (wrapper.py:998) instead. Write the oracle-check test against the tmux path." + }, + { + "ref": "src/zo/wrapper.py:898", + "hazard": "`_backoff_wait` implements exponential-backoff RETRY (30s * 2^n + jitter, capped by `_max_retries`, wrapper.py:794), and on exhaustion returns status RATE_LIMITED and gives up (795-800). The plan requires 'wait-and-resume' (pause, resume at reset) \u2014 not a retry loop. Naively wiring the watchdog on top of this produces exactly the retry-storm behaviour the spec rules out.", + "mitigation": "Add a distinct `wait_and_resume` mode: on rate-limit, set an explicit paused state, suppress ALL nudges/escalations for the duration, sleep until the reset instant, then resume the same session. Keep `_backoff_wait` only for the no-reset-time fallback." + }, + { + "ref": "src/zo/experiments.py:491", + "hazard": "`mint_experiment` stores `artifacts_dir=str(artifacts_dir)` where `artifacts_dir = registry_dir / exp_id` and `registry_dir` comes from `Orchestrator._experiments_dir()` = `Path(self._target.target_repo)/'.zo'/'experiments'` (orchestrator.py:1119-1122) \u2014 an ABSOLUTE machine path in practice. The docstring at experiments.py:176-178 claims it is 'relative to delivery repo root'. Consumers use it raw: `Path(exp.artifacts_dir)` in `_finalize_experiments` (orchestrator.py:1203) and `resolve_active_experiment_dir` (experiments.py:448, 455). registry.json lives inside the delivery repo and is git-committed.", + "mitigation": "A fresh-context loop that commits registry.json as a checkpoint bakes machine-absolute paths into git; on a different box (the Linux demo box for oracle check #13, or `zo continue --repo`) `_finalize_experiments` reads a nonexistent path and reports result.md missing forever. Store relative and resolve against the delivery repo at read time, with a back-compat branch for absolute values already on disk." + }, + { + "ref": "src/zo/experiment_loop.py:303", + "hazard": "The DEAD_END verdict requires `e.hypothesis` to be populated on registry entries (303-305, 314-319), and `check_dead_end` (395) compares against `exp.hypothesis` (438-440). But `Experiment.hypothesis` is NEVER written in production: `mint_experiment` is always called with the default empty hypothesis (orchestrator.py:1156-1161), and `parse_hypothesis_md` (experiments.py:636) has ZERO callers outside tests (verified by repo-wide grep \u2014 hits only in experiments.py itself, its __all__, and tests/unit/test_experiments.py). Same for `parse_next_md`/`update_next_ideas`: `next_ideas` is always empty, so `_next_planned` (experiments.py:386) always returns [] and CHECKLIST.md's 'Next planned' section never renders.", + "mitigation": "DEAD_END is dead code and the lineage digest will be hypothesis-less. Wire `parse_hypothesis_md` -> registry update at `_finalize_experiments` time (and `parse_next_md` -> `update_next_ideas`) before building the fresh-context lineage digest, or the fresh session re-derives from a registry with no semantic content." + }, + { + "ref": "src/zo/experiment_loop.py:283", + "hazard": "Plateau detection requires `len(deltas) == policy.plateau_runs` (283) where deltas only includes experiments with a non-None `delta_vs_parent` (276-281). `delta_vs_parent` is computed only when the parent exists AND has a result AND the primary_metric NAMES MATCH exactly (experiments.py:515-525). Any renamed metric, any aborted parent, or any root experiment in the window silently makes plateau unreachable \u2014 the loop then runs to `max_iterations` instead.", + "mitigation": "For the fresh loop, treat a missing delta as an explicit signal (log it) rather than silently skipping, and consider comparing on absolute metric values when names diverge. Also note `_abort_running_experiments` (orchestrator.py:1356) leaves ABORTED experiments in the registry, and `_ensure_experiment_for_phase` picks `latest_in_phase` (1154) REGARDLESS of status \u2014 so a child can be parented on an aborted, result-less experiment, permanently poisoning that window's deltas." + }, + { + "ref": "src/zo/orchestrator.py:1388", + "hazard": "`build_lead_prompt` has a MINTING SIDE EFFECT (`_ensure_experiment_for_phase` at 1388, reached from `_prompt_experiment_context` at 550). A fresh-context loop that rebuilds the prompt per iteration will mint or silently reuse depending on whether the previous experiment is still RUNNING. If the driver rebuilds the prompt before `_finalize_experiments`/`_abort_running_experiments` runs, the 'fresh' iteration writes into the PREVIOUS exp dir with the previous exp id \u2014 silently collapsing two iterations into one and corrupting the lineage that the loop's budget/plateau logic counts.", + "mitigation": "Either make the mint explicit in the driver (call `_ensure_experiment_for_phase` once, pass the exp into prompt building) or assert in the driver that no RUNNING exp exists for the phase before building the next iteration's prompt." + }, + { + "ref": "src/zo/_wrapper_models.py:29", + "hazard": "`LeadProcess` carries `pid` and `started_at` (wall-clock of ZO's launch call) but no OS process start time. PID-only identity is exactly the recycled-PID hazard the plan calls out ('PID + process-start-time identity so recycled PIDs are never acted on'). Additionally, in tmux mode `pid` is often the tmux/shell pid rather than the claude process (the launch path returns a pane id at wrapper.py:154+), so `pgrep`-style liveness on `process.pid` can be checking the wrong process entirely.", + "mitigation": "Capture the real claude PID + its OS start time at launch (both tmux and headless paths) and persist both into the heartbeat JSON; compare the tuple, never the PID alone. `_tmux_claude_running(pane_id)` (wrapper.py:934) is the existing tmux-side liveness primitive to build on." + }, + { + "ref": "src/zo/orchestrator.py:284", + "hazard": "Phase state is flushed to STATE.md ONLY in `end_session` (`_capture_phase_states` at 284). A crash, kill, or watchdog-initiated restart between iterations loses the in-memory phase status. PR-037 (memory/zo-platform/PRIORS.md ~L1100+) exists precisely because this interacts badly with the ACTIVE resume branch.", + "mitigation": "Deferral 3 argues for the ledger as restore source \u2014 plan-ledger.json IS written atomically on every mutation (ledger.py:81-91, `_mutate` at 173-181), so it is already crash-safe where STATE.md is not. Have the fresh loop persist phase state to the ledger per iteration, not per session." + }, + { + "ref": "src/zo/orchestrator.py:928", + "hazard": "`apply_human_decision` requires a matching gate nonce and raises PermissionError otherwise (928-941), and the nonce is single-use (`clear_gate_nonce` at 942). The nonce is minted only in `advance_phase`'s BLOCKING branch (788). If the fresh-context driver starts calling `advance_phase` and a phase hits a BLOCKING gate mid-loop, the loop must stop and surface the nonce \u2014 a driver that blindly relaunches will loop forever on a GATED phase (`get_current_phase` returns GATED first, orchestrator.py:710-712).", + "mitigation": "The driver must break out on `requires_human=True` / `PhaseStatus.GATED` and print the nonce from `prepare_gate_review` (orchestrator.py:902-904). Note `_consume_gate_decision` (403) only applies a stored decision at DECOMPOSE time, so a mid-loop approval needs an explicit re-check, not just a relaunch." + }, + { + "ref": "src/zo/orchestrator.py:1258", + "hazard": "`_auto_iterate_if_needed` returns None (loop disabled) in SUPERVISED gate mode (1258-1259), and `_prompt_experiment_context` picks supervised phrasing via `autonomous = self._gate_mode != GateMode.SUPERVISED` (1404). Meanwhile `_refresh_gate_mode` (729) re-reads the `gate_mode` file at the top of every `advance_phase`, and the wrapper watches the same file per poll (`_check_gate_mode_change`, wrapper.py:648).", + "mitigation": "The fresh loop must re-read gate mode between iterations (it can change mid-run via `zo gates set`) and must handle a mid-loop switch to supervised by stopping cleanly rather than spawning another session with an inconsistent prompt." + }, + { + "ref": "src/zo/experiment_loop.py:178", + "hazard": "`resolve_policy` returns the SHARED module-level `DEFAULT_POLICY` singleton when there are no overrides and low_token is False (`if not overrides and not low_token: return DEFAULT_POLICY`, 178-179). Pydantic BaseModel instances are mutable by default here (no frozen config, experiment_loop.py:120). Any code that mutates the returned policy (e.g. a fresh-loop 'decrement remaining budget' convenience) would corrupt the global default for the whole process and every subsequent project in the same run.", + "mitigation": "Never mutate the returned policy; if the fresh loop needs per-iteration counters keep them in the driver, or make `LoopPolicy` frozen." + }, + { + "ref": "specs/watchdog.md:75", + "hazard": "The spec's stated integration points are stale relative to the v2 plan: watchdog.md:75 says 'orchestrator.py \u2014 arm the heartbeat cron on run/session start' and watchdog.md:65 configures `stall_threshold_min: 20`, while the v2 plan (plans/zo-v2-rearchitecture.md:108-110) puts the checker in the LifecycleWrapper poll loop and the task brief specifies an external checker, NOT a cron and NOT a monitor agent. watchdog.md:66-68 also lists a `remediation: [nudge, respawn, reroute]` ladder and a 10-min oracle threshold that disagrees with the spec's 20-min default.", + "mitigation": "Treat plans/zo-v2-rearchitecture.md as authoritative over specs/watchdog.md, and update the spec in the same PR (the repo's docs-cascade rule), otherwise the next reader implements a cron. Reconcile the 10-min oracle check #11 against the 20-min spec default explicitly." + }, + { + "ref": "src/zo/cli.py:942", + "hazard": "`_launch_and_monitor` ends with `orchestrator.end_session()` (942), `semantic.close()` (944), surrogate `deregister_session` + `consolidate_all` (949-960), and the permissions overlay is reclaimed per-run (773-783). Calling this helper once per iteration would consolidate memory, close the semantic index, and churn the permissions overlay N times per phase.", + "mitigation": "Split launch from teardown (see integration point cli.py:1150) before adding the loop; do not simply call `_launch_and_monitor` in a `while`." + } + ], + "reusable": [ + { + "ref": "src/zo/wrapper.py:588", + "what": "`_maybe_open_training_pane` \u2014 the exact pattern for a per-poll external checker: called unconditionally inside the poll loop (wrapper.py:711), reads state from disk, guards on preconditions, fires once, and swallows its own exceptions (633-634). Copy this shape for `_check_watchdog`." + }, + { + "ref": "src/zo/wrapper.py:648", + "what": "`_check_gate_mode_change` \u2014 per-poll file-watch with last-value memo and comms logging (648-667). The heartbeat-freshness checker is structurally identical (read file, compare, log)." + }, + { + "ref": "src/zo/wrapper.py:920", + "what": "`_tmux_pane_alive(pane_id)` (920) and `_tmux_claude_running(pane_id)` (934) \u2014 existing process-liveness primitives for the tmux path; plus `_capture_tmux_pane(pane_id, lines)` (998) for reading session output when there is no stdout log." + }, + { + "ref": "src/zo/ledger.py:81", + "what": "`_atomic_write(path, text)` (mkstemp in the target dir + os.replace, 81-91) and `_mutate(memory_root, fn)` (173-181) \u2014 the load-modify-write-atomically idiom for control-plane files. Heartbeat writes should reuse it; a torn heartbeat read would misclassify liveness." + }, + { + "ref": "src/zo/experiments.py:269", + "what": "`save_registry` \u2014 the same atomic tmp+replace idiom for the registry (273-279), plus `_safe_refresh_checklist` (343-354) showing the fail-open derived-view pattern (`contextlib.suppress(OSError)`)." + }, + { + "ref": "src/zo/surrogate.py:360", + "what": "`commit_worktree(path, *, message)` + `_git(repo, *args)` (97) \u2014 the ONLY git-subprocess code in ZO and directly reusable for git-commit-as-checkpoint. Precedent call at src/zo/consolidate.py:174-182 shows the guard-then-commit pattern." + }, + { + "ref": "src/zo/experiments.py:287", + "what": "`render_checklist(registry)` \u2014 ready-made lineage digest renderer (exp / parent / status / hypothesis / metric / delta / tier / top shortfall + 'Next planned'), deterministic and idempotent. Use it for the fresh session's lineage digest instead of writing a new renderer." + }, + { + "ref": "src/zo/ledger.py:248", + "what": "`summarize(doc)` \u2014 per-phase (passed, total) counts, already used by `zo status`. Ready-made ledger digest for the fresh prompt." + }, + { + "ref": "src/zo/orchestrator.py:398", + "what": "`_ledger_safe(fn, *args)` \u2014 the fail-open ledger-mutator wrapper (`contextlib.suppress(OSError)`). Any new ledger writes from the fresh loop should go through it, matching the 'a ledger IO problem must never crash a build' rule at ledger.py:18-19." + }, + { + "ref": "src/zo/orchestrator.py:1323", + "what": "`_record_learning(title, root_cause, rule_gap)` \u2014 writes an EvolutionEngine FailureRecord + a durable PriorEntry, best-effort with a comms error fallback (1333-1354). Reuse verbatim for watchdog escalations (stall detected, iteration restarted) so stalls become priors." + }, + { + "ref": "src/zo/hookkit.py:392", + "what": "`main(argv)` event dispatcher + `_read_stdin_json` (53), `_repo_root` (60), `_memory_root` (64), `_emit` (75), `_trace` (81), `_agent_name` (112). Adding a `heartbeat` event is a handler function plus one settings.json wiring \u2014 no new plumbing. `_trace` already writes `logs/hook-trace-{date}.jsonl` for live verification." + }, + { + "ref": ".claude/hooks/zo-hookkit.sh:1", + "what": "The venv-preferring, fail-open shim (exits 0 on any missing precondition; only runs inside the ZO platform repo per the `[[ -d \"$REPO_ROOT/src/zo\" ]]` guard). Heartbeat hooks route through the same script with a new event arg \u2014 no second shim." + }, + { + "ref": "tests/integration/test_auto_iteration.py:154", + "what": "`_run_iteration(orch, phase, ...)` helper: build_lead_prompt (155) -> mark_subtask_complete each subtask (157) -> write result.md -> advance_phase (162). This is the exact call sequence the fresh-context driver must perform in production; the test is the executable spec for the driver's contract." + }, + { + "ref": "src/zo/experiment_loop.py:136", + "what": "`resolve_policy(spec, *, low_token, max_iterations_override)` \u2014 the established multi-level cap precedence (CLI > plan > low-token clamp > default, documented at 149-150 and implemented 165-180). Mirror it exactly for any new fresh-loop cap." + } + ], + "open_questions": [ + "Who owns the fresh-context driver loop \u2014 a new `zo.experiment_loop.run_loop()` / `zo.driver` module, an Orchestrator method, or inline in cli.py's `build`? The task brief says 'experiment_loop / orchestrator spawns a fresh builder session per iteration', but experiment_loop.py today is a pure, dependency-free evaluator (imports only pydantic + zo.experiments) and giving it subprocess/wrapper dependencies would invert the layering and force the 16 pure unit tests in tests/unit/test_experiment_loop.py to grow process fixtures.", + "Since `advance_phase` has never had a runtime caller, does Phase 3 wire the FULL gate path (artifact checks at orchestrator.py:797, `_generate_test_report` 830, `_generate_notebook` 831, `_generate_snapshot` 832, `mark_phase_passed` 829) or only the phase_4 loop branch? Wiring everything at once turns previously-inert code into per-iteration production behaviour (pytest runs, notebook generation) with real cost and failure surface.", + "How do subtasks get marked complete in production? `mark_subtask_complete` (orchestrator.py:848) is the precondition for `all_done` at advance_phase:766, and it also has no runtime caller. Without an answer, `advance_phase` will always return ITERATE with 'Subtasks remaining' (838-844) and the loop verdict path at 820 is never reached. Options: a hook parsing the lead's task list (`wrapper.read_task_list`, wrapper.py:499), a new CLI subcommand the lead invokes, or inferring completion from the ledger.", + "Watchdog heartbeats are per-agent, but ZO only tracks the LEAD process (`LeadProcess`, one per run). Sub-agents are spawned by the lead via Claude Code's Agent/TeamCreate tool and ZO has no PID for them \u2014 `monitor_team` (wrapper.py:481) reads team/task files only. Is per-agent liveness therefore heartbeat-file-only (hook-written), with process liveness available exclusively for the lead?", + "What is the nudge delivery channel? specs/watchdog.md:76 says nudges go 'through the existing comms bus', but `CommsLogger` is append-only JSONL that nothing reads back into a live session. In tmux the only real injection path is `tmux send-keys` into the pane (the `_verify_prompt_submitted` machinery at wrapper.py:364-401 shows the fragility); in headless there is no stdin at all. This determines whether 'nudge' is even implementable before 'restart'.", + "Does the fresh-context loop reuse ONE tmux pane (relaunch in place) or create a new pane per iteration? Affects `_launch_tmux` (wrapper.py:154), the `_wait_for_tui_ready` startup grace (304-350), the permissions overlay lifecycle (cli.py:773-783), and whether the user can still watch the run.", + "For the deferral-3 session-restore cutover: `plan-ledger.json` `phase_status` is a free-form `dict[str, str]` (ledger.py:73) with values written as `str(phase.status)` at emit (ledger.py:147) and literals 'completed'/'active'/'blocked' by the mutators. Does the cutover add a PhaseStatus-validated type to the ledger (PR-036's parse-time-validation rule), and does the ledger also need `completed_subtasks_by_phase` \u2014 which it does NOT currently carry, though `_restore_phase_states` (orchestrator.py:479-481) restores it from STATE.md?", + "Oracle check #13 runs on a Linux box with the claude CLI (this Mac has none, per memory/zo-platform/STATE.md:11 / PR-046). Given the absolute-path hazard in `Experiment.artifacts_dir`, is the demo repo created fresh on that box, or does a registry.json committed here get checked out there \u2014 in which case the path fix is a check-13 blocker, not a nice-to-have?", + "Should the per-iteration git checkpoint commit the delivery repo's working tree wholesale (`git add -A` via `commit_worktree`, surrogate.py:368) or only `.zo/experiments/` + artifacts? A blanket `add -A` mid-training can commit checkpoints/large binaries; the delivery repo's .gitignore is scaffolded at scaffold.py:227 and may not cover model weights." + ], + "notes": "Answers to the five sub-questions, condensed:\n\n(a) SEAM + FRESH-SESSION INPUTS. The seam is cli.py:1134-1166 \u2014 specifically between `build_lead_prompt` (1143) and `_launch_and_monitor` (1150), with `_launch_and_monitor` split into `_launch_once` (cli.py:798-939) and teardown (941-960). Inputs and where each is produced:\n - ledger entry -> ledger.py:93 `load_ledger` + ledger.py:248 `summarize`; file written by orchestrator.py:363 `_emit_plan_ledger` at every decompose (merge-preserving, ledger.py:125-170).\n - experiment lineage digest -> experiments.py:233 `ExperimentRegistry.lineage` + experiments.py:287 `render_checklist`; registry mutated by `mint_experiment` (459), `update_result` (504), `update_status` (533).\n - priors digest -> orchestrator.py:1784 `_prompt_memory` (priors[:8] at 1798, semantic top_k=3 at 1801); priors seeded by `_maybe_seed_priors` (258) and appended by `_record_learning` (1342).\n - next.md -> NOT produced by Python. Written by the model-builder agent per .claude/agents/model-builder.md:68-79 and read from disk by the next iteration per the prompt at orchestrator.py:1487. `parse_next_md` (experiments.py:650) exists but is never called in production, so `Experiment.next_ideas` is always empty.\n - active exp id / parent id -> orchestrator.py:1388 `_ensure_experiment_for_phase` (side effect of prompt build; parent = `latest_in_phase`, 1154).\n\n(b) evaluate_loop_state. Current inputs: `registry: ExperimentRegistry`, `phase: str`, `policy: LoopPolicy | None` (experiment_loop.py:205-209); body touches only `registry.experiments` (230) and policy. Test call sites: exactly 16, ALL in tests/unit/test_experiment_loop.py (lines 102, 111, 122, 137, 145, 150, 159, 176, 185, 196, 218, 232, 243, 259, 312, 328); 12 of them pass policy positionally. Plus 1 production call site (orchestrator.py:1276) and 2 docstring mentions (experiment_loop.py:36, 163). Least-invasive: append `*, ledger: LedgerFile | None = None` \u2014 keyword-only, defaulted, so all 16 tests and the one caller are untouched; pass `load_ledger(memory_root / LEDGER_FILENAME)` at orchestrator.py:1276 (already fail-open, ledger.py:93). A thin adapter is strictly more code for no benefit.\n\n(c) GIT CHECKPOINT. Existing reusable git code: src/zo/surrogate.py:97 `_git` (the only `subprocess.run(['git', ...])` in ZO), surrogate.py:360 `commit_worktree(path, *, message)` (add -A + commit, returns True only if a commit was made), surrogate.py:379 `merge_branch`, surrogate.py:355 `remove_worktree`. Precedent caller: consolidate.py:174-182. Confirmed by grep that orchestrator.py, cli.py, scaffold.py, snapshots.py contain NO git-commit code (scaffold.py:227 is a `.gitignore` literal; scaffold.py:244 is an apt package list). `SessionState.git_head` (_memory_models.py:40) + `MemoryManager._get_git_head` (memory.py:303) already exist to record/verify the head. Fit: call `commit_worktree(delivery_repo, message=...)` in the driver right after `advance_phase` returns ITERATE (i.e. after orchestrator.py:1312's `reset_phase`), then update `git_head`.\n\n(d) CAPS. `LoopPolicy` fields at experiment_loop.py:113-118 (`max_iterations=10`, `plateau_epsilon=0.01`, `plateau_runs=3`, `stop_on_tier='must_pass'`, `dead_end_threshold=0.9`, `low_token=False`). Low-token clamps at experiment_loop.py:130-133 (`max_iterations=2`, `stop_on_tier='could_pass'`), applied BEFORE plan overrides in `resolve_policy` (166-175), with CLI `--max-iterations` winning last (176-177); documented precedence CLI > plan > clamp > default (149-150). The CLI-side preset lives separately at cli.py:262-269 and threads through cli.py:1118-1119. Budget is enforced at experiment_loop.py:261 by counting COMPLETE experiments in the phase across the whole registry \u2014 durable on disk, so a restart cannot reset it. The fresh loop must ALSO enforce max_iterations as a driver-side session counter, because a session that never reaches a gate never consults the evaluator at all.\n\n(e) PROMPT TEXT. `.claude/agents/lead-orchestrator.md` has NO Phase-4 iteration content whatsoever (grep for iterat|loop|fresh|restart|phase_4 hits only line 23 roster blurb, line 175 STATE.md example, line 216 session-end, line 230 checklist). It needs: a one-iteration-per-session rule, the specs/watchdog.md:77 liveness-by-evidence standing rule, and a STATE.md -> plan-ledger.json swap at lines 101/213/230. `.claude/agents/model-builder.md:86-110` already specifies a fully disk-based auto-proposer protocol (read parent result.md/diagnosis.md/next.md, write hypothesis.md, do not ask the human, do not decide when iteration ends) \u2014 it is already fresh-context-compatible and needs only a note that no prior-iteration context survives. One conflict to reconcile: model-builder.md:290 tells the builder to escalate on a 2-iteration plateau, while plateau ownership belongs to experiment_loop.py:274-296. The orchestrator-side mirror of this text is `_render_loop_briefing` (orchestrator.py:1456-1519) and `_prompt_experiment_context` (1378-1454)." + }, + { + "surface": "Session-state restore: STATE.md (`SessionState.phase_states` / `completed_subtasks_by_phase`) \u2192 plan-ledger.json cutover, preserving PR-036 (parse-time status validation) and PR-037 (GATED > ACTIVE > PENDING resume precedence). Read: src/zo/ledger.py (all 254 lines), src/zo/orchestrator.py (start_session/decompose_plan/_restore_phase_states/get_current_phase/advance_phase/apply_human_decision/_auto_iterate_if_needed/end_session), src/zo/memory.py, src/zo/_memory_formats.py, src/zo/_memory_models.py, src/zo/_orchestrator_models.py, src/zo/hookkit.py (sealed paths), src/zo/cli.py (status + build/continue), src/zo/preflight.py, tests/unit/test_ledger.py, tests/unit/test_orchestrator.py, tests/unit/test_memory.py, memory/zo-platform/PRIORS.md PR-036/PR-037.", + "integration_points": [ + { + "ref": "src/zo/ledger.py:67-74", + "what": "(a) On-disk schema of plan-ledger.json: `class LedgerFile: version:int=1; project:str=\"\"; generated_at:datetime; phase_status: dict[str,str] = Field(default_factory=dict); entries: list[LedgerEntry]`. phase_status is a FREE-FORM str map \u2014 no enum, no validator, no closed value set. It is the only phase-level state in the file.", + "action": "Cutover: replace `dict[str,str]` with a validated type (Literal/enum mirroring PhaseStatus, or a pydantic field_validator) so PR-036's parse-time validation moves to the ledger boundary; add a drift-guard test mirroring tests/unit/test_memory.py:156 `test_valid_phase_statuses_match_enum` against the ledger's allowlist." + }, + { + "ref": "src/zo/ledger.py:54-64", + "what": "(a) `LedgerEntry: subtask_id, phase_id, description, acceptance_criteria, verification, passes:bool=False, attempts:int=0, last_failure:str|None`. There is NO per-subtask `completed` flag: `passes` is flipped only wholesale per phase by mark_phase_passed (ledger.py:193-198).", + "action": "GAP \u2014 add a per-subtask completion signal (e.g. `completed:bool` set by mark_subtask_complete, distinct from oracle-owned `passes`), otherwise phase.completed_subtasks cannot be restored from the ledger." + }, + { + "ref": "src/zo/ledger.py:227-236", + "what": "(a) `record_attempt` \u2014 the only mutator wired to subtask completion (orchestrator.py:858) \u2014 increments `attempts` and explicitly does NOT touch passes. attempts>=1 means 'was worked on', not 'is complete'.", + "action": "Do not use attempts>0 as a completed_subtasks proxy in the cutover; it would resurrect abandoned subtasks. Add the explicit completed flag instead." + }, + { + "ref": "src/zo/ledger.py:144-148", + "what": "(a) emit_ledger seeds `phase_status[phase.phase_id] = prev_status.get(phase.phase_id, str(phase.status))` \u2014 the previous ledger value ALWAYS wins over the in-memory (STATE.md-restored) status on every regeneration. First emission is the only time workflow status seeds the ledger.", + "action": "This is the divergence root: once seeded, a hand-edited STATE.md can never correct the ledger. Cutover must decide precedence explicitly and (if ledger-primary) stop restoring from STATE.md rather than leaving two writable sources." + }, + { + "ref": "src/zo/ledger.py:184-200", + "what": "(d)/(b) `mark_phase_passed` sets every entry of the phase to passes=True, clears last_failure, and sets `phase_status[phase_id]=\"completed\"`. Docstring states it is ORACLE-OWNED and reachable only from orchestrator verified paths.", + "action": "Read-only dependency for restore; keep as the single completed-writer." + }, + { + "ref": "src/zo/ledger.py:203-213", + "what": "(b) `reset_phase(memory_root, phase_id, reason)` clears passes, writes last_failure[:500], sets phase_status=\"active\". This is the ONLY writer of the 'active' literal.", + "action": "Read-only dependency \u2014 this is what makes PR-037 ACTIVE-resume representable in the ledger." + }, + { + "ref": "src/zo/ledger.py:239-245", + "what": "(b) `set_phase_status(memory_root, phase_id, status)` takes an unvalidated str; orchestrator calls it with only \"gated\" (orchestrator.py:792) and \"blocked\" (orchestrator.py:960).", + "action": "Add value validation here (raise/normalize) so a typo can't poison the control plane the way `prep_complete` poisoned STATE.md (PRIORS.md:1046)." + }, + { + "ref": "src/zo/ledger.py:93-98", + "what": "(a)/(c) `load_ledger` returns None on OSError OR ValueError \u2014 fail-open by design. A corrupt or schema-invalid ledger is indistinguishable from an absent one at the call site.", + "action": "For restore, None must NOT silently mean 'all phases pending'. Distinguish absent (bootstrap \u2192 fall back to STATE.md) from corrupt (raise/refuse, mirroring PR-036's fail-loud stance) \u2014 see hazards." + }, + { + "ref": "src/zo/ledger.py:81-90", + "what": "(d) `_atomic_write` \u2014 mkdir + tempfile.mkstemp in the target dir + os.replace. All ledger writes are in-process Python file writes.", + "action": "Read-only dependency; confirms orchestrator writes bypass the Claude tool surface entirely." + }, + { + "ref": "src/zo/ledger.py:173-181", + "what": "`_mutate` load-modify-write; returns False if the ledger is absent or corrupt \u2014 every mutator silently no-ops in that case.", + "action": "During cutover, a no-op mutator becomes a lost state transition (today it only loses telemetry). Surface the False return at the orchestrator call site (orchestrator.py:398-401 currently discards it)." + }, + { + "ref": "src/zo/orchestrator.py:237-256", + "what": "(c) `start_session`: L239 `state = self._memory.recover_session()` (the STATE.md read), L240-244 mode is derived from STATE.md existence + `state.phase == \"init\"`, L246-247 caches and immediately rewrites STATE.md.", + "action": "Cutover point 1: after L239, load the ledger and overlay phase state; keep `state.phase`/mode derivation as-is initially (the ledger has no session-phase pointer or mode field \u2014 see gaps)." + }, + { + "ref": "src/zo/orchestrator.py:468-481", + "what": "(c) `_restore_phase_states` \u2014 the entire STATE.md\u2192workflow restore: L472 `saved_states = self._session_state.phase_states`, L473 `saved_subtasks = self._session_state.completed_subtasks_by_phase`, L474-475 early-return when empty, L477-478 `phase.status = PhaseStatus(saved_states[phase.phase_id])`, L479-481 `phase.completed_subtasks = list(saved_subtasks.get(...))`.", + "action": "Cutover point 2 (primary): make this read the ledger first \u2014 `doc = zo_ledger.load_ledger(memory_root/LEDGER_FILENAME)`; for each phase, status = doc.phase_status.get(pid) if present else STATE.md value else leave PENDING; completed_subtasks from ledger entries' new completed flag, else STATE.md. L478 is the exact line in the PR-036 traceback (PRIORS.md:1049-1053) \u2014 keep the coercion but validate before it." + }, + { + "ref": "src/zo/orchestrator.py:350-353", + "what": "(c) decompose_plan ordering: L350 `_restore_phase_states()` \u2192 L351 `_consume_gate_decision()` \u2192 L352 `_emit_contracts_file()` \u2192 L353 `_emit_plan_ledger()`. Restore happens BEFORE the ledger is (re)written, which is what makes the ledger seed correct on first run.", + "action": "Preserve this order. If restore becomes ledger-primary, emit_ledger at L353 must not clobber restored-and-then-mutated status \u2014 today prev_status wins (ledger.py:146), so it happens to be safe, but the invariant needs a test." + }, + { + "ref": "src/zo/orchestrator.py:359-360", + "what": "(c) `if self._session_state is not None and not self._session_state.phase_states: self._session_state.phase = phases[0].phase_id` \u2014 the session's current-phase pointer is gated on STATE.md phase_states being empty.", + "action": "Cutover must also consult the ledger here, otherwise a ledger-only project (no ## Phases in STATE.md) resets state.phase to phase_1 on every decompose." + }, + { + "ref": "src/zo/orchestrator.py:293-303", + "what": "(c) `_capture_phase_states` (called from end_session L284) writes phase.status and completed_subtasks back into SessionState \u2192 STATE.md. This is the sole producer of the ## Phases section.", + "action": "Keep writing STATE.md as the human projection (plan L103 'STATE.md becomes a projection for humans'), but stop treating it as the parse target; add a ledger write-through here for completed_subtasks so the two agree at session end." + }, + { + "ref": "src/zo/orchestrator.py:786-792", + "what": "(b) writer 1: `phase.status = PhaseStatus.GATED` (L787) + nonce mint + `zo_contracts.set_active_phase` + `self._ledger_safe(\"set_phase_status\", phase_id, \"gated\")` (L792). WRITE-THROUGH PRESENT.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/orchestrator.py:825-829", + "what": "(b) writer 2: `phase.status = PhaseStatus.COMPLETED` (L825) + `self._ledger_safe(\"mark_phase_passed\", phase_id)` (L829, sets phase_status=completed). WRITE-THROUGH PRESENT.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/orchestrator.py:943-946", + "what": "(b) writer 3: apply_human_decision PROCEED \u2192 `phase.status = PhaseStatus.COMPLETED` (L944) + `mark_phase_passed` (L946). WRITE-THROUGH PRESENT.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/orchestrator.py:951-956", + "what": "(b) writer 4: ITERATE \u2192 `phase.status = PhaseStatus.ACTIVE` (L952), `phase.completed_subtasks.clear()` (L953), `reset_phase(..., notes)` (L954-956, sets phase_status=active). WRITE-THROUGH PRESENT \u2014 note the ledger clears `passes` but there is no per-subtask completed flag to clear, so the L953 clear has no ledger analogue.", + "action": "When adding the per-subtask completed flag, reset_phase must clear it too or ITERATE will resume with stale completions." + }, + { + "ref": "src/zo/orchestrator.py:958-960", + "what": "(b) writer 5: ESCALATE \u2192 `phase.status = PhaseStatus.BLOCKED` (L959) + `set_phase_status(..., \"blocked\")` (L960). WRITE-THROUGH PRESENT.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/orchestrator.py:961-962", + "what": "(b) writer 6: the else/HOLD branch \u2014 `phase.status = PhaseStatus.GATED` (L962) with NO _ledger_safe call. MISSING WRITE-THROUGH.", + "action": "Add `self._ledger_safe(\"set_phase_status\", phase_id, \"gated\")` after L962. Today it is masked because HOLD is only reached from an already-GATED phase, but a ledger-primary restore makes any missing write-through a silent state loss." + }, + { + "ref": "src/zo/orchestrator.py:1310-1315", + "what": "(b) writer 7: autonomous loop CONTINUE \u2192 `phase.status = PhaseStatus.ACTIVE` (L1310), `completed_subtasks.clear()` (L1311), `reset_phase(..., f\"loop CONTINUE: {reason}\")` (L1312-1315). WRITE-THROUGH PRESENT. This is one of the two PR-037 ACTIVE producers (PRIORS.md:1116-1117).", + "action": "read-only dependency; this is exactly the state the Phase-3 fresh-context loop must re-derive from the ledger." + }, + { + "ref": "src/zo/orchestrator.py:476-478", + "what": "(b) writer 8: `_restore_phase_states` assigns phase.status from STATE.md with NO ledger write-back. MISSING WRITE-THROUGH \u2014 restore is the one status writer that never reaches the control plane.", + "action": "After cutover this either disappears (ledger-primary) or must write through; leaving it as-is guarantees STATE.md/ledger divergence." + }, + { + "ref": "src/zo/orchestrator.py:687-727", + "what": "(c) `get_current_phase` \u2014 PR-037 precedence lives here and is source-agnostic: L709-712 GATED loop, L713-716 ACTIVE loop, L717-726 PENDING-with-deps-met, BLOCKED intentionally never returned (L704-705 docstring). It reads only `phase.status` on the in-memory PhaseDefinition.", + "action": "NO CHANGE NEEDED for the cutover \u2014 precedence is preserved automatically as long as _restore_phase_states populates the same field. The whole cutover risk is upstream of L707." + }, + { + "ref": "src/zo/orchestrator.py:398-401", + "what": "(d)/(b) `_ledger_safe(fn, *args)` \u2014 `with contextlib.suppress(OSError): getattr(zo_ledger, fn)(self._memory.memory_root, *args)`. Runs as in-process platform code; the bool return (False = ledger absent/corrupt) is discarded.", + "action": "Reuse as the single write-through helper for new sites; consider logging when it returns False (comms.log_error) once the ledger is authoritative." + }, + { + "ref": "src/zo/orchestrator.py:363-396", + "what": "(c) `_emit_plan_ledger` \u2014 fail-open emission (OSError logged as ledger_emission_failed warning, never raised) driven by workflow + oracle threshold.", + "action": "read-only dependency; the bootstrap path that materializes a ledger for pre-Phase-2 projects on their first Phase-3 session." + }, + { + "ref": "src/zo/_memory_formats.py:30-32", + "what": "(c) PR-036's allowlist: `_VALID_PHASE_STATUSES: frozenset[str] = frozenset({\"pending\",\"active\",\"gated\",\"blocked\",\"completed\",\"skipped\"})`, deliberately a string set to avoid an orchestrator import, with a drift-guard test referenced in the comment (L27-29).", + "action": "Reuse verbatim as the ledger's allowlist \u2014 import it into zo.ledger (no cycle: _memory_formats imports nothing from ledger/orchestrator) rather than defining a third copy." + }, + { + "ref": "src/zo/_memory_formats.py:154-168", + "what": "(c) PR-036 enforcement point: `_PHASE_LINE_RE` match at L155, `if status not in _VALID_PHASE_STATUSES: raise ValueError(...)` at L158-165 naming the bad value, the phase id, STATE.md, and the sorted valid list.", + "action": "Mirror this error shape in the ledger loader so operators who hand-edit plan-ledger.json get the same quality of message." + }, + { + "ref": "src/zo/memory.py:131-140", + "what": "(c)/HAZARD: `read_state` catches `(ValueError, KeyError)` at L139 and returns a default `SessionState()`. PR-036's carefully worded ValueError from parse_state NEVER reaches an operator through this path \u2014 an invalid STATE.md silently restores as 'no phase_states', i.e. every phase back to PENDING.", + "action": "Verify before cutover: tests/unit/test_memory.py:174 calls `_parse_state` directly, so the swallow is untested. If STATE.md becomes the fallback source, the swallow must not turn a corrupt fallback into a clean-slate restore." + }, + { + "ref": "src/zo/memory.py:142-161", + "what": "(c) `write_state` \u2014 atomic (.STATE.md.tmp + os.replace), preserving agent-written sections after ## Phases via render_state(preserve_from=...).", + "action": "read-only dependency; keeps STATE.md viable as the human projection post-cutover." + }, + { + "ref": "src/zo/memory.py:279-301", + "what": "(c) `recover_session` \u2014 read_state + git-HEAD reconciliation; L290-293 treats a missing STATE.md as fresh BUILD; L295-300 pushes a git_head mismatch into active_blockers.", + "action": "Cutover point 3: a ledger-primary world still needs this git reconciliation; keep it, add ledger presence to the fresh-vs-resume decision at L290." + }, + { + "ref": "src/zo/_memory_models.py:42-43", + "what": "(a) The two STATE.md fields the ledger must replace: `phase_states: dict[str,str]` and `completed_subtasks_by_phase: dict[str, list[str]]`. The ledger has an equivalent for the first (phase_status) and NO equivalent for the second.", + "action": "Ledger schema addition required (per-subtask completed flag), else tests/unit/test_orchestrator.py:1049 test_partial_progress_restored regresses." + }, + { + "ref": "src/zo/_orchestrator_models.py:28-36", + "what": "(a) `PhaseStatus(StrEnum)`: PENDING/ACTIVE/GATED/BLOCKED/COMPLETED/SKIPPED. StrEnum means `str(phase.status)` at ledger.py:147 yields the bare value, so all six are representable in phase_status.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/hookkit.py:47-50", + "what": "(d) `_SEALED_DEFAULTS = (\"gate_mode\",\"gate_nonce\",\"gate_decision\", CONTRACTS_FILENAME, \"plan-ledger.json\", \"sealed_paths\")` \u2014 the ledger is sealed at hookkit.py:49.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/hookkit.py:314-348", + "what": "(d) `_sealed_prefixes` resolves each name under memory_root (L317) plus the optional sealed_paths file (L318-324); `_handle_sealed_paths` (L328) reads `tool_input.file_path`/`path` and denies on prefix match (L340-348).", + "action": "read-only dependency \u2014 the handler is driven entirely by a tool_input payload, i.e. it can only fire on tool calls." + }, + { + "ref": ".claude/settings.json:63-71", + "what": "(d) The sealed-paths hook is registered under PreToolUse with `\"matcher\": \"Write|Edit\"` only. Orchestrator/ledger writes go through zo.ledger._atomic_write in-process (no Write/Edit tool), so they correctly bypass the hook \u2014 confirmed by tests/unit/test_ledger.py:159-165 (oracle flip lands) vs 140-157 (builder Write denied).", + "action": "read-only dependency. Note it is registered ONLY in the ZO platform repo; nothing in src/zo/scaffold.py|target.py|environment.py installs it into a delivery repo \u2014 sealing relies on the session running with cwd=zo_root plus ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT set at cli.py:1076-1077." + }, + { + "ref": "src/zo/cli.py:1081-1082", + "what": "(e) STATE.md consumer making a control decision: `state_check = memory.read_state()` then `detected_mode = \"build\" if state_check.phase == \"init\" else \"continue\"` \u2014 drives the banner (L1088) and the plan-edit re-decompose branch (L1125).", + "action": "Cutover must not break this: the ledger has no `phase`/`mode` field, so either keep STATE.md for the session pointer or add one to LedgerFile." + }, + { + "ref": "src/zo/cli.py:2303-2311", + "what": "(e) `zo status` hard-requires STATE.md: `if not state_path.exists(): print('No STATE.md found...'); raise SystemExit(1)` \u2014 before the ledger is even loaded.", + "action": "Relax to 'STATE.md or plan-ledger.json exists' so a ledger-primary project is inspectable." + }, + { + "ref": "src/zo/cli.py:2313-2327", + "what": "(e) Existing ledger-first rendering precedent: loads the ledger (L2317) and overrides `current_phase` with the first non-'completed' key of `phase_status` (L2321-2327), falling back to `state.phase`. Note this ignores dependency order and treats blocked/skipped as current.", + "action": "Reuse the ledger-first-then-STATE.md fallback shape for _restore_phase_states, but do NOT reuse this 'first non-completed' heuristic \u2014 get_current_phase's GATED>ACTIVE>PENDING+deps logic is the contract (PRIORS.md:1133)." + }, + { + "ref": "src/zo/cli.py:2328-2348", + "what": "(f-adjacent) status ledger table: per-phase status string, passed/total from summarize (ledger.py:248-254), summed attempts, last failure truncated to 60 chars.", + "action": "read-only dependency" + }, + { + "ref": "src/zo/hookkit.py:233-254", + "what": "(e) `_handle_precompact` gates on `(memory_root/'STATE.md').exists()` (L236) and does a read_state\u2192write_state flush (L242-244). A project without STATE.md gets no pre-compaction checkpoint.", + "action": "Keep STATE.md written by end_session/_capture_phase_states so this hook keeps working, or extend the gate to the ledger." + }, + { + "ref": "src/zo/hookkit.py:257-261", + "what": "(e) `_handle_session_end` has the same `STATE.md exists` gate (L260).", + "action": "Same as precompact \u2014 do not delete STATE.md in the cutover." + }, + { + "ref": "src/zo/preflight.py:174-182", + "what": "(e) preflight memory round-trip asserts write_state/read_state on a temp MemoryManager (`loaded.phase != 'preflight-check'` \u2192 fail).", + "action": "read-only dependency; add a parallel ledger round-trip check if the ledger becomes the control-plane source." + }, + { + "ref": "tests/unit/test_orchestrator.py:460-566", + "what": "(c) `TestGetCurrentPhase` \u2014 PR-037 precedence lock-in: test_returns_first_pending:463, test_skips_blocked_dependency:472, test_advances_after_completion:483, test_returns_none_when_all_done:493, test_returns_none_without_decomposition:502, test_returns_active_phase_on_resume:508, test_gated_takes_priority_over_active:527, test_active_takes_priority_over_pending:539, test_blocked_phase_not_returned:551.", + "action": "All nine set `decomp.phases[i].status` directly, so they are source-agnostic and must stay green unchanged \u2014 that is the cutover's regression oracle." + }, + { + "ref": "tests/unit/test_orchestrator.py:568-605", + "what": "(c) `test_real_resume_via_state_md_round_trip` \u2014 the ONLY test coupling restore to STATE.md: writes `SessionState(phase_states={phase_1:completed, phase_2:active, phase_3:pending})` (L585-595), then start_session\u2192decompose_plan\u2192get_current_phase must return phase_2 ACTIVE (L596-605).", + "action": "Keep as the STATE.md-fallback test and add a ledger twin (`phase_status={...}` written via emit_ledger+set_phase_status) plus a conflict test asserting ledger wins." + }, + { + "ref": "tests/unit/test_orchestrator.py:938-1086", + "what": "(c) `TestPhasePersistence`: test_completed_phases_restored_after_redecompose:940 (two orchestrators sharing one MemoryManager, FULL_AUTO), test_no_phase_states_backward_compat:989 (old-format STATE.md with no ## Phases), test_partial_progress_restored:1049 (asserts `first_sub in restored_p1.completed_subtasks`).", + "action": "L1049 is the test that fails the moment restore stops reading completed_subtasks_by_phase \u2014 it is the concrete driver for the per-subtask completed flag." + }, + { + "ref": "tests/unit/test_memory.py:147-206", + "what": "(c) `TestPhaseStatusValidation` \u2014 PR-036 lock-in: test_valid_phase_statuses_match_enum:156 (drift guard `{s.value for s in PhaseStatus} == _VALID_PHASE_STATUSES`), test_unknown_status_raises_with_clear_message:163 (asserts 'prep_complete','phase_3','STATE.md','completed','active' all appear in the message), test_known_statuses_accepted:184 (all six round-trip).", + "action": "Mirror all three against the ledger loader; the drift guard at :156 is the one that must be duplicated or generalized so a new PhaseStatus member can't be added without updating both parsers." + }, + { + "ref": "tests/unit/test_ledger.py:36-52", + "what": "(f) Fixture pattern: no pytest fixtures at all \u2014 a module-level `_workflow()` helper builds a 2-phase WorkflowDecomposition (phase_1 BLOCKING with 2 subtasks + 1 required artifact, phase_4 AUTOMATED with 1 subtask) and every test passes pytest's `tmp_path` directly as memory_root.", + "action": "Reuse `_workflow()` + tmp_path-as-memory_root for new restore tests; note test_orchestrator.py instead uses `plan`/`orchestrator` fixtures (tests/unit/test_orchestrator.py:53-72) with a real MemoryManager \u2014 cutover tests spanning both layers should follow the orchestrator fixture." + }, + { + "ref": "tests/unit/test_ledger.py:140-157", + "what": "(d)/(f) The hook-interaction test pattern: monkeypatch.setenv ZO_REPO_ROOT/ZO_MEMORY_ROOT (L146-147), monkeypatch sys.stdin to io.StringIO(json.dumps({agent_name, tool_input.file_path})) (L148-154), assert hookkit.main([\"sealed-paths\"]) == 0 and parse capsys JSON for permissionDecision == 'deny' (L155-157).", + "action": "Reuse verbatim for any Phase-3 heartbeat/watchdog file that must also be sealed." + } + ], + "hazards": [ + { + "ref": "src/zo/ledger.py:63 + src/zo/orchestrator.py:858", + "hazard": "The ledger cannot express partial subtask progress. mark_subtask_complete only calls record_attempt (attempts += 1); `passes` is phase-wide. A pure ledger-primary restore loses phase.completed_subtasks, which advance_phase compares against phase.subtasks at orchestrator.py:766 (`all_done = set(phase.subtasks) == set(phase.completed_subtasks)`) \u2014 every resumed phase would re-run all subtasks and never reach its gate.", + "mitigation": "Add `completed: bool` to LedgerEntry set by a new mark_subtask_complete write-through, cleared by reset_phase (ledger.py:206-212); until then keep completed_subtasks_by_phase sourced from STATE.md. Guard with tests/unit/test_orchestrator.py:1049." + }, + { + "ref": "src/zo/orchestrator.py:478", + "hazard": "`PhaseStatus(saved_states[phase.phase_id])` is the exact line in the PR-036 crash trace (PRIORS.md:1049-1053). If the ledger becomes the source and `phase_status` stays an unvalidated dict[str,str] (ledger.py:73), a hand-edited or externally written plan-ledger.json reintroduces the identical bare `ValueError: 'x' is not a valid PhaseStatus` from enum.py \u2014 with no file name in the message, and one layer further from the operator than STATE.md was.", + "mitigation": "Validate at the ledger boundary (load_ledger / a pydantic field_validator on phase_status) using the existing _VALID_PHASE_STATUSES frozenset, raising a message that names plan-ledger.json, the phase id, the bad value, and the sorted valid list \u2014 the PR-036 shape from _memory_formats.py:158-165." + }, + { + "ref": "src/zo/ledger.py:93-98", + "hazard": "load_ledger is fail-open: OSError and ValueError both return None, so 'corrupt/invalid ledger' is indistinguishable from 'no ledger yet'. If restore treats None as 'nothing to restore', a truncated ledger silently resets every phase to PENDING and the next session redoes completed work \u2014 the loud failure PR-036 was written to prevent becomes a silent one.", + "mitigation": "Split the paths: `path.exists()` + parse failure \u2192 raise/abort with an operator-facing message; `not path.exists()` \u2192 fall back to STATE.md. Do not reuse the fail-open loader (which the hook path legitimately needs) for the restore path." + }, + { + "ref": "src/zo/memory.py:139", + "hazard": "`except (ValueError, KeyError): return SessionState()` swallows parse_state's PR-036 ValueError in every production caller (orchestrator.py:232, orchestrator.py:239 via recover_session, cli.py:1081, cli.py:2311, hookkit.py:242, preflight.py:179). The PR-036 tests only exercise _parse_state directly (tests/unit/test_memory.py:174), so this swallow is untested. An operator with an invalid STATE.md today sees a clean-slate restore, not the helpful error.", + "mitigation": "Pre-existing bug worth fixing in the same PR: let the ValueError propagate (or re-raise with the path) at least on the restore path, and add a test asserting read_state surfaces it. If STATE.md becomes the fallback, this swallow silently converts fallback into data loss." + }, + { + "ref": "src/zo/ledger.py:146-147", + "hazard": "emit_ledger always prefers the previous ledger's phase_status over the freshly restored in-memory status. Combined with _restore_phase_states writing STATE.md \u2192 workflow but never workflow \u2192 ledger (orchestrator.py:476-481), STATE.md and plan-ledger.json can diverge permanently and neither converges. Today the drift only misrenders `zo status`; after cutover it decides what work resumes.", + "mitigation": "Pick one writer. Recommended: ledger-primary read + write-through at every status site, and on first-run bootstrap (no ledger) seed from STATE.md then never read STATE.md phase_states again except as a diagnostic mismatch warning." + }, + { + "ref": "src/zo/orchestrator.py:962", + "hazard": "The HOLD branch of apply_human_decision sets PhaseStatus.GATED with no _ledger_safe call \u2014 the only status writer besides restore that lacks write-through. Under ledger-primary restore, a HOLD applied to a phase whose ledger row says 'active' (e.g. gated after an ITERATE cycle) resumes as ACTIVE and skips the human gate, breaking PR-037's GATED>ACTIVE precedence at the persistence layer rather than in get_current_phase.", + "mitigation": "Add `self._ledger_safe(\"set_phase_status\", phase_id, \"gated\")` after L962, and add a coverage test per PR-037 rule 1 (PRIORS.md:1125): every PhaseStatus a producer writes must have a proven write-through and a proven read-back." + }, + { + "ref": "src/zo/orchestrator.py:351 + 353", + "hazard": "_consume_gate_decision (L351) runs BEFORE _emit_plan_ledger (L353). On a project's first decompose the ledger file does not exist, so every _ledger_safe call inside apply_human_decision hits `_mutate`'s `doc is None` early return (ledger.py:176-178) and is silently dropped. It currently self-heals only because emit_ledger then seeds phase_status from the in-memory status \u2014 an accident the cutover must not rely on.", + "mitigation": "Move _emit_plan_ledger before _consume_gate_decision, or have _consume_gate_decision ensure the ledger exists first; add a test that a first-run gate decision lands in plan-ledger.json." + }, + { + "ref": "src/zo/cli.py:2303-2309", + "hazard": "`zo status` exits 1 when STATE.md is missing, before reading the ledger. A Phase-3 fresh-context iteration that only ever writes the ledger (plan L112) would make the project un-inspectable.", + "mitigation": "Change the guard to accept either file; render the ledger table when STATE.md is absent." + }, + { + "ref": "src/zo/ledger.py:153 + 77-78", + "hazard": "subtask_id = f\"{phase_id}:{_slug(subtask)}\" with `re.sub(r\"[^a-z0-9]+\", \"-\", ...)`. Two subtasks in one phase differing only in punctuation/case collide into one entry \u2014 silently merging their attempts and (post-cutover) their completion state. This is PR-036 rule 3 (silent structured-line non-match, PRIORS.md:1070) in a new place.", + "mitigation": "Detect collisions in emit_ledger and fail loudly or disambiguate with an index suffix; the original text is preserved in `description` (ledger.py:159) so restore can key on description instead." + }, + { + "ref": "src/zo/_orchestrator_models.py:36 + src/zo/ledger.py:198,211,239", + "hazard": "PhaseStatus.SKIPPED is never written by any orchestrator site (no `= PhaseStatus.SKIPPED` exists in orchestrator.py) and 'pending' reaches the ledger only via the emit seed (ledger.py:147) \u2014 no mutator ever returns a phase to pending. A ledger-primary restore therefore has no way to represent 'reverted to pending', and PR-037 rule 2's 'one test per non-terminal status the producer can write' is only satisfiable for gated/active/completed/blocked.", + "mitigation": "Either drop SKIPPED from the enum or give it a writer; add a set_phase_status('pending') path if any Phase-3 restart flow needs to rewind a phase." + }, + { + "ref": ".claude/settings.json:64 (vs src/zo/scaffold.py)", + "hazard": "The sealed-paths PreToolUse hook exists only in the ZO platform repo's settings; nothing in scaffold.py/target.py/environment.py installs it into a delivery repo. Sealing of plan-ledger.json depends entirely on the agent session running with the platform repo's .claude/ config plus ZO_MEMORY_ROOT (cli.py:1076). Any Phase-3 flow that launches a builder with a different cwd (fresh-context loop) loses ledger sealing.", + "mitigation": "When the fresh-context loop spawns builders, assert ZO_REPO_ROOT/ZO_MEMORY_ROOT are exported and that the hook config is in scope; add a hookkit test covering the delivery-repo cwd case." + } + ], + "reusable": [ + { + "ref": "src/zo/_memory_formats.py:30-32", + "what": "_VALID_PHASE_STATUSES frozenset + the drift-guard convention (comment at L27-29, test at tests/unit/test_memory.py:156). Import it into zo.ledger rather than writing a third copy of the status allowlist." + }, + { + "ref": "src/zo/_memory_formats.py:158-165", + "what": "The PR-036 error-message template (bad value, phase id, file name, sorted valid list, pointer to the PhaseStatus enum). Copy this shape for the ledger validator." + }, + { + "ref": "src/zo/ledger.py:81-90", + "what": "_atomic_write (mkstemp in target dir + os.replace + unlink-on-failure) \u2014 the atomic-write primitive for any Phase-3 heartbeat/state file; do not hand-roll another." + }, + { + "ref": "src/zo/ledger.py:173-181", + "what": "_mutate load-modify-write wrapper \u2014 the pattern every new ledger mutator should use so writes stay atomic and absent-file-safe." + }, + { + "ref": "src/zo/orchestrator.py:398-401", + "what": "_ledger_safe \u2014 the single write-through call site helper; new status writers should route through it instead of importing zo_ledger directly." + }, + { + "ref": "src/zo/ledger.py:136-148", + "what": "emit_ledger's merge-preserving regeneration (prev_entries by subtask_id, prev_status by phase_id) \u2014 the mechanism that lets a fresh-context session re-decompose the plan without erasing verified progress; the Phase-3 fresh loop should re-derive state through this rather than persisting its own." + }, + { + "ref": "src/zo/cli.py:2313-2327", + "what": "The existing ledger-first-with-STATE.md-fallback rendering precedent in `zo status`, including the `from zo.ledger import ...` local import to avoid an import cycle." + }, + { + "ref": "src/zo/ledger.py:248-254", + "what": "summarize(doc) \u2192 per-phase (passed, total); reuse for any Phase-3 progress/heartbeat reporting instead of recounting entries." + }, + { + "ref": "tests/unit/test_ledger.py:36-52", + "what": "_workflow() helper + tmp_path-as-memory_root: the zero-fixture test pattern for ledger-level tests." + }, + { + "ref": "tests/unit/test_ledger.py:140-157", + "what": "The monkeypatch(setenv ZO_REPO_ROOT/ZO_MEMORY_ROOT) + monkeypatch(sys.stdin=io.StringIO(json)) + capsys-JSON pattern for asserting a hook's permissionDecision \u2014 reuse for sealing any new Phase-3 control file." + }, + { + "ref": "tests/unit/test_orchestrator.py:53-72", + "what": "`plan` (parses the fixture plan) and `orchestrator` (real Target/MemoryManager/CommsLogger/SemanticIndex under tmp_path) fixtures \u2014 the harness for end-to-end restore tests spanning memory + ledger + orchestrator." + }, + { + "ref": "tests/unit/test_orchestrator.py:940-987", + "what": "The two-orchestrators-sharing-one-MemoryManager session-boundary pattern (orch1 works and end_session()s, orch2 start_session+decompose_plan and asserts restored state) \u2014 the exact shape needed for the ledger-primary restore tests and for the Phase-3 fresh-context per-iteration re-derivation test." + } + ], + "open_questions": [ + "Precedence on conflict: when plan-ledger.json says phase_2=active and STATE.md says phase_2=completed, does the ledger win silently, win with a logged warning, or refuse to start? Recommendation: ledger wins + comms.log_error mismatch, since STATE.md is documented as hand-editable (PRIORS.md:1066) and the plan calls it a human projection (plans/zo-v2-rearchitecture.md:103) \u2014 but this changes observable behavior for anyone who currently fixes a stuck run by editing STATE.md, which is exactly the prod-001 workflow in PR-036/PR-037.", + "Does the ledger gain a session pointer? cli.py:1082 derives build-vs-continue from `state.phase == \"init\"` and orchestrator.py:359-360 sets state.phase from phase_states emptiness. LedgerFile (ledger.py:67-74) has no `phase`, `mode`, or `last_completed_subtask`. Either those stay STATE.md-owned (partial cutover) or LedgerFile grows three fields.", + "Should `passes` and a new per-subtask `completed` be separate? `passes` is oracle-owned and sealed (ledger.py:184-191); subtask completion is builder-reported via mark_subtask_complete (orchestrator.py:848-858). Merging them would let a builder's self-report flip an oracle flag \u2014 the exact anti-Goodhart property WS-B was built for.", + "Migration for existing projects: on the first Phase-3 session a project may have a populated STATE.md ## Phases and either no ledger or a ledger seeded before the phases progressed. Is there a one-shot backfill (STATE.md \u2192 ledger) and is it gated on the operator, or does emit_ledger's prev_status-wins rule (ledger.py:146) mean a stale ledger permanently shadows a correct STATE.md?", + "Does `evaluate_loop_state` reading the ledger (the folded-in deferral) change its signature? Today it takes only (registry, phase, policy) at experiment_loop.py:205-208 and is called at orchestrator.py:1276; adding a ledger read means either passing memory_root through or passing a pre-loaded LedgerFile. Sixteen call sites in tests/unit/test_experiment_loop.py pass positionally, so a keyword-only optional param is the low-churn option.", + "Where do the Phase-3 heartbeat JSON files live relative to plan-ledger.json, and do they belong in _SEALED_DEFAULTS (hookkit.py:47-50)? Sealing them prevents an agent from faking liveness, but the wrapper writes them out-of-process \u2014 confirm the writer is platform code (like ledger._atomic_write) and not an agent tool call before sealing.", + "Is anything expected to read plan-ledger.json from outside the orchestrator process (a watchdog checker in the LifecycleWrapper poll loop)? If so, concurrent load-modify-write via _mutate (ledger.py:173-181) is last-writer-wins across processes \u2014 os.replace is atomic per write but there is no lock, so a watchdog-driven status write could clobber a concurrent orchestrator gate write." + ], + "notes": "CUTOVER PROPOSAL (answer to (c), concrete):\n\n1. Add to LedgerFile (src/zo/ledger.py:67-74): validated `phase_status` (reuse _VALID_PHASE_STATUSES from src/zo/_memory_formats.py:30-32) and to LedgerEntry (ledger.py:54-64) a `completed: bool = False`.\n2. New mutator `mark_subtask_completed(memory_root, phase_id, subtask)` alongside record_attempt (ledger.py:227-236); call it from mark_subtask_complete at src/zo/orchestrator.py:858 next to the existing record_attempt write-through. reset_phase (ledger.py:206-212) must clear it, mirroring `phase.completed_subtasks.clear()` at orchestrator.py:953 and 1311.\n3. Rewrite _restore_phase_states (src/zo/orchestrator.py:468-481) as: load ledger (strict variant that raises on parse failure of an existing file); for each workflow phase \u2014 status = ledger.phase_status[pid] if present, else STATE.md saved_states[pid] if present, else leave PhaseStatus.PENDING; completed_subtasks = [e.description for e in entries if e.phase_id == pid and e.completed] if the ledger has any entry for that phase, else STATE.md saved_subtasks. Keep the `PhaseStatus(...)` coercion at L478 but only after boundary validation, so the PR-036 traceback (PRIORS.md:1049-1053) cannot recur.\n4. Precedence is ledger > STATE.md, per-phase, presence-based (not whole-file), so a project with a partial ledger still restores. Log a comms warning on any per-phase mismatch rather than silently discarding the STATE.md value.\n5. PR-037 (GATED > ACTIVE > PENDING) needs NO change: get_current_phase (orchestrator.py:709-726) reads only phase.status, and all nine TestGetCurrentPhase tests (tests/unit/test_orchestrator.py:463-566, notably test_gated_takes_priority_over_active:527 and test_active_takes_priority_over_pending:539) set status directly on the decomposition, so they are source-agnostic and remain the regression oracle. The precedence is only preserved end-to-end if the ledger can actually hold 'gated' and 'active' \u2014 it can: 'gated' via set_phase_status (orchestrator.py:792) and 'active' via reset_phase (ledger.py:211, from orchestrator.py:954 and 1312).\n6. PR-036 is preserved by MOVING the validation, not dropping it: keep _memory_formats.py:158-165 for the STATE.md fallback and add the same check to the ledger loader, with tests/unit/test_memory.py:156's drift guard extended (or duplicated) to cover the ledger allowlist.\n7. Close the two write-through gaps before flipping the source: orchestrator.py:962 (HOLD \u2192 GATED, no ledger write) and orchestrator.py:476-481 (restore, no ledger write-back). All six other status writers already write through (792, 829, 946, 954, 960, 1312).\n8. STATE.md keeps being written (_capture_phase_states at orchestrator.py:293-303 \u2192 write_state at 285) as the human projection \u2014 required by hookkit.py:236 and hookkit.py:260, which both gate on STATE.md existing, and by cli.py:2303-2309.\n\n(f) tests/unit/test_ledger.py: 12 tests across 3 classes \u2014 TestGeneration (line 55, 3 tests), TestMutators (line 87, 6 tests), TestOracleOwnership (line 137, 2 tests). No pytest fixtures; a module-level `_workflow()` builder at line 36 plus pytest's tmp_path used directly as memory_root, and monkeypatch/capsys only in the hook-interaction test at line 140.\n\n(d) Sealed status confirmed: \"plan-ledger.json\" is in `_SEALED_DEFAULTS` at src/zo/hookkit.py:47-50 (the tuple lives in hookkit.py, not contracts.py), resolved under memory_root at hookkit.py:317. The hook is a PreToolUse handler registered with matcher \"Write|Edit\" at .claude/settings.json:64-68 and keyed entirely off `tool_input.file_path` (hookkit.py:329-331), so orchestrator writes via ledger._atomic_write (ledger.py:81-90) never touch it \u2014 the intended asymmetry, locked by tests/unit/test_ledger.py:140-157 (builder Write denied) vs :159-165 (oracle flip lands)." + }, + { + "surface": "CLI launch/lifecycle orchestration across sessions: src/zo/cli.py (build, continue, _launch_and_monitor, status), src/zo/wrapper.py (LifecycleWrapper poll loops + rate-limit path), src/zo/surrogate.py (per-PID liveness), src/zo/preflight.py, src/zo/project_config.py, src/zo/experiment_loop.py, src/zo/orchestrator.py (phase resume/persist), src/zo/ledger.py.\n\n(a) LIFECYCLE \u2014 one lead session per CLI invocation, resolving exactly ONE phase. `zo build` is straight-line; no loop, no relaunch. cli.py:1134 `phase = orchestrator.get_current_phase()`; cli.py:1135-1137 `if phase is None: ... raise SystemExit(0)`; cli.py:1140-1141 one-shot `_show_phase_review` + `_ask_additional_instructions`; cli.py:1150-1166 a single `_launch_and_monitor(...)` call, after which build returns and the process exits. `zo continue` (cli.py:1206) is pure delegation \u2014 cli.py:1274 `click_ctx.invoke(build, plan_path=plan_path, ...)`. Phase advancement happens INSIDE the agent session (orchestrator.py:752 advance_phase) and is persisted to STATE.md (orchestrator.py:861, orchestrator.py:285); the CLI never advances phases or relaunches \u2014 the next phase needs a fresh `zo continue`. Gate handling is out-of-band, not a CLI loop: `zo gates approve` (cli.py:2497) -> `_apply_gate_decision_cli` (cli.py:2426) writes DECISION_LOG/comms/gate decision and prints at cli.py:2490-2493 \"A running session picks this up at its next gate poll; otherwise it applies on zo continue\". The wrapper's per-poll gate check (wrapper.py:648-670) only logs a checkpoint, it takes no action.\n\n(b) RATE-LIMITED \u2014 never reaches the CLI in the default tmux path, and the CLI has no handler. RATE_LIMITED is only ever set in `_wait_headless` (wrapper.py:795); `_wait_tmux` (wrapper.py:672-761) contains no rate-limit detection at all. Today's behaviour is a retry-loop, exactly what the spec wants replaced: wrapper.py:802-812 `time.sleep(wait_secs); retries += 1; continue` with `_backoff_wait` = 30 * 2^n + jitter (wrapper.py:898-900), capped by `max_retries=3` (wrapper.py:87), after which RATE_LIMITED is returned. In cli.py, `AgentStatus.RATE_LIMITED` is never referenced; the status lands in the generic else at cli.py:924-928 (\"Session ended with status: rate_limited\") and then teardown proceeds (cli.py:930-960). A wait-and-resume state must surface at cli.py:920 (the single `wait_for_completion` call site) and be handled before the teardown block at cli.py:930.\n\n(c) WATCHDOG CONFIG THREADING \u2014 ProjectConfig (project_config.py:28-51) is a plain pydantic BaseModel with no `model_config`, so pydantic v2 default `extra='ignore'` silently drops a `watchdog:` block today, and `save_project_config` (project_config.py:132-152) round-trips `model_dump()` so unknown keys are lost on any save. Add a nested `watchdog: WatchdogConfig = WatchdogConfig()` field there (spec asks for this at specs/watchdog.md:78, block shape at specs/watchdog.md:62-68). Threading path: build resolves context at cli.py:1071 and calls `ctx.make_target()` at cli.py:1072 \u2014 `make_target` (cli.py:75-87) loads the ProjectConfig internally and discards it, so build never holds one; either load it explicitly next to cli.py:1071 or add a `make_project_config()` to ProjectContext (cli.py:51-87). Then pass through `_launch_and_monitor` (new kwarg alongside cli.py:709-729) into the `wait_for_completion` call at cli.py:920-923, which today passes only on_status/gate_mode_file/project_name/delivery_repo \u2014 note `poll_interval` is NOT passed, so it silently uses the 10.0s default at wrapper.py:551, and `timeout` (wrapper.py:552) is never set by any caller.\n\n(d) HEADLESS / LINUX DEMO BOX \u2014 `--no-tmux` is documented as NOT supported for full builds: docs/cli/build.mdx:133 \"multi-phase orchestration requires the interactive session; `--no-tmux` is for `init` and `draft`, not full builds\". No doc says the Linux demo box uses headless; every demo/CI headless reference is `zo init --no-tmux` (docs/SAMPLE_PROJECT.md:19, docs/SAMPLE_PROJECT.md:71, docs/reference/cost-benchmark.mdx:173, docs/reference/cost-benchmark.mdx:177), and docs/TROUBLESHOOTING.md:40-42 uses `zo build --no-tmux` only as a diagnostic. Existing `claude -p` invocations in src/: (1) `_launch_headless` wrapper.py:414-461 builds `[claude, --print, --output-format, json, --model, M, --max-turns, N, --add-dir, cwd]` + optional `--dangerously-skip-permissions` + one `--add-dir` per extra dir + `[\"-p\", prompt]`, env = `os.environ.copy()` then `env.update(extra_env)` (wrapper.py:456-459), Popen with stdout/stderr redirected to log files (wrapper.py:461-463) \u2014 this is the model to reuse for a fresh-context builder spawn; (2) the end-of-session Haiku summary at cli.py:688-694: `[\"claude\", \"-p\", \"--model\", \"haiku\", ]` with `capture_output=True, timeout=20`, no env, no cwd, bare `claude` (not `_resolve_claude_bin`), wrapped in `except Exception: pass` (cli.py:695-696) \u2014 do NOT copy this pattern. PRIORS PR-001 (memory/zo-platform/PRIORS.md:16-23) confirms `--dangerously-skip-permissions` only works with `--print`, so a per-iteration fresh builder must be headless.\n\n(e) STATUS / WATCH DATA SOURCES \u2014 `zo status` (cli.py:2275) reads STATE.md via `memory.read_state()` (cli.py:2305) and, when present, `plan-ledger.json` becomes the phase source (cli.py:2317-2352, WS-B already landed this), plus recent session summaries (cli.py:2368). There is NO `zo watch` CLI command \u2014 the only watch command is `zo watch-training` (cli.py:2912), which reads training metrics from the active experiment dir (cli.py:2955-2960). The `/watch` slash command (.claude/commands/observe/watch.md:12-20) does ad-hoc liveness via `ps aux | grep claude`, `~/.claude/teams/`, `~/.claude/tasks/`, and `logs/comms/*.jsonl` \u2014 precisely what a heartbeat file replaces. Live in-session status flows through `wrapper.monitor_team` (wrapper.py:481, reads `~/.claude/teams/{team}/config.json` at wrapper.py:1013) and `read_task_list` (wrapper.py:499, `~/.claude/tasks/{team}`), rendered by the `_print_status` callback (cli.py:831) on every poll cycle.\n\nPREFLIGHT (PR-046): an explicit claude-CLI check ALREADY exists \u2014 `_check_claude_cli` (preflight.py:84-89), invoked first at preflight.py:70. It is only `shutil.which(\"claude\")`; it does not verify version, auth, or that `claude -p` actually runs.", + "integration_points": [ + { + "ref": "src/zo/cli.py:920", + "what": "The single `wrapper.wait_for_completion(...)` call in the whole build/continue path. Passes on_status/gate_mode_file/project_name/delivery_repo only \u2014 no poll_interval, no timeout. This is the one place a watchdog config and a rate-limit-pause signal can enter the poll loop.", + "action": "Add watchdog kwargs (enabled, poll_interval, stall_threshold, nudge budget) and a wait-and-resume mode flag here; thread from a ProjectConfig loaded in build." + }, + { + "ref": "src/zo/wrapper.py:547", + "what": "`wait_for_completion` signature \u2014 today `poll_interval: float = 10.0, timeout: float|None = None, on_status, gate_mode_file, project_name, delivery_repo`. Stashes per-run state on self at wrapper.py:576-580 (`self._gate_mode_file`, `self._training_pane_id`, ...).", + "action": "Extend with a `watchdog: WatchdogConfig|None = None` kwarg and follow the existing self._ stash convention so both _wait_tmux and _wait_headless can read it." + }, + { + "ref": "src/zo/wrapper.py:672", + "what": "`_wait_tmux` \u2014 the default poll loop for `zo build` inside tmux. Has gate-mode check + training-pane check + liveness debounce, but NO rate-limit detection and NO heartbeat check.", + "action": "Insert the external heartbeat/stall checker next to `self._check_gate_mode_change()` (wrapper.py:723) and add rate-limit detection via pane capture (`_capture_tmux_pane`, used at wrapper.py:750)." + }, + { + "ref": "src/zo/wrapper.py:793", + "what": "`if self._detect_rate_limit(output):` in `_wait_headless` \u2014 the only rate-limit code path. Retry-loop at wrapper.py:802-812 (sleep backoff, retries += 1, continue), terminal RATE_LIMITED at wrapper.py:795.", + "action": "Replace the retry-loop with wait-and-resume: parse the reset time, emit a PAUSED status through on_status, sleep until reset, resume without consuming a retry. Oracle check #12 targets this line." + }, + { + "ref": "src/zo/cli.py:924", + "what": "`else: console.print(f\"[red bold]Session ended with status:[/] {process.status}\")` \u2014 the generic branch that swallows RATE_LIMITED. `AgentStatus.RATE_LIMITED` appears nowhere in cli.py.", + "action": "Add an explicit RATE_LIMITED / PAUSED branch here that either resumes or prints a resume-at time, before the teardown block at cli.py:930-960." + }, + { + "ref": "src/zo/cli.py:709", + "what": "`_launch_and_monitor` signature (kwargs through cli.py:729). Straight-line: register surrogate (cli.py:789) -> launch (cli.py:804) -> wait (cli.py:920) -> summary/next-steps (cli.py:930-944) -> end_session (cli.py:947) -> deregister + consolidate (cli.py:953-957).", + "action": "This is where an iteration-restart loop must wrap the launch+wait pair. Note register/deregister must be re-run per restart, and end_session/consolidate must fire only after the final iteration." + }, + { + "ref": "src/zo/cli.py:1076", + "what": "extra_env assembly: ZO_MEMORY_ROOT = memory.memory_root (cli.py:1076), ZO_DELIVERY_ROOT (cli.py:1077), ZO_CONTRACTS_PATH (cli.py:1078); low-token adds CLAUDE_AUTOCOMPACT_PCT_OVERRIDE at cli.py:1056.", + "action": "Add ZO_HEARTBEAT_PATH (or derive from ZO_MEMORY_ROOT) here so hooks/agents can write heartbeats. hookkit.py:64-73 already resolves memory root from ZO_MEMORY_ROOT with a repo fallback \u2014 reuse that resolver." + }, + { + "ref": "src/zo/surrogate.py:279", + "what": "`_pid_alive(pid)` \u2014 `os.kill(pid, 0)` only; ProcessLookupError=>False, PermissionError=>True, OSError=>False. No process-start-time comparison.", + "action": "Extend with a start-time check (psutil-free: `ps -o lstart= -p PID` on darwin/linux) and store the start time in the lock file so recycled PIDs are never treated as live." + }, + { + "ref": "src/zo/surrogate.py:254", + "what": "`register_session` lock payload writes pid, role, surrogate_id, worktree, started_at \u2014 where started_at is the REGISTRATION wall-clock (datetime.now(UTC)), not the OS process start time.", + "action": "Add a `proc_start` field capturing the OS process start time; this is the identity token Phase 3's watchdog needs." + }, + { + "ref": "src/zo/wrapper.py:277", + "what": "tmux LeadProcess is constructed with `pid=None, ... tmux_pane_id=pane_id` \u2014 the interactive lead session has NO pid recorded. Headless does (wrapper.py:466 `pid=proc.pid`).", + "action": "Capture the real Claude PID for tmux launches (`tmux display-message -p '#{pane_pid}'` then descend, or record pane_pid) so PID+start-time identity works in the default path." + }, + { + "ref": "src/zo/project_config.py:28", + "what": "`class ProjectConfig(BaseModel)` \u2014 no model_config, so extra keys are silently ignored; `save_project_config` (project_config.py:132) writes `config.model_dump()`, losing any unknown keys on round-trip.", + "action": "Add a nested `watchdog: WatchdogConfig = WatchdogConfig()` model (enabled, poll_interval_sec, stall_threshold_min, stall_ticks_hard, stall_ticks_soft, nudge_budget) matching specs/watchdog.md:62-68." + }, + { + "ref": "src/zo/cli.py:75", + "what": "`ProjectContext.make_target` loads the ProjectConfig via `load_project_config(self.delivery_repo)` and immediately discards it, returning only a TargetConfig. Legacy layout (cli.py:88) never has a ProjectConfig at all.", + "action": "Add `make_project_config()` (returning None for legacy) so build can read the watchdog block without a second disk read, and define the legacy fallback default." + }, + { + "ref": "src/zo/experiment_loop.py:205", + "what": "`evaluate_loop_state(registry, phase, policy)` \u2014 reads only the ExperimentRegistry. The deferral says it should read the ledger. Only caller is orchestrator.py:1276.", + "action": "Add a `ledger: LedgerFile | None = None` (or memory_root) parameter; consume `LedgerFile.phase_status` / entry pass flags (ledger.py:67-75) so the loop verdict is control-plane-derived." + }, + { + "ref": "src/zo/experiment_loop.py:56", + "what": "`__all__` \u2014 every export is a pure decision function (evaluate_loop_state, resolve_policy, tier_meets, check_dead_end). There is NO spawn/subprocess machinery anywhere in this module.", + "action": "Phase 3's fresh-context per-iteration spawner is net-new code here; model the subprocess construction on wrapper.py:414-461, not on the Haiku call at cli.py:688." + }, + { + "ref": "src/zo/orchestrator.py:468", + "what": "`_restore_phase_states` \u2014 the STATE.md -> PhaseStatus parse target: `phase.status = PhaseStatus(saved_states[phase.phase_id])` at orchestrator.py:478. Called from orchestrator.py:350 during decompose_plan.", + "action": "Cutover point for session-restore: source `saved_states` from plan-ledger.json phase_status while keeping PhaseStatus coercion and the PR-036 parse-time validation." + }, + { + "ref": "src/zo/orchestrator.py:706", + "what": "`get_current_phase` resume precedence: GATED (orchestrator.py:709-711) > ACTIVE (orchestrator.py:713-715) > PENDING-with-deps-met (orchestrator.py:717-726). BLOCKED intentionally excluded. This is the PR-036/PR-037 contract (memory/zo-platform/PRIORS.md:1042-1152).", + "action": "Read-only dependency \u2014 the ledger cutover must produce identical ordering; PRIORS PR-037 rule 3 requires explicit GATED-vs-ACTIVE and ACTIVE-vs-PENDING tests." + }, + { + "ref": "src/zo/preflight.py:84", + "what": "`_check_claude_cli` already exists and runs first (preflight.py:70). It is only `shutil.which(\"claude\")` \u2014 no version, no auth, no `claude -p` smoke test.", + "action": "PR-046 is already partly satisfied; strengthen with a `claude --version` probe (mirroring _check_docker at preflight.py:192-199) rather than adding a new check." + }, + { + "ref": "src/zo/ledger.py:67", + "what": "`LedgerFile.phase_status: dict[str, str]` \u2014 free-form strings, no enum validation; `load_ledger` (ledger.py:93-98) is fail-open, returning None on any OSError/ValueError.", + "action": "Before cutting session-restore over to the ledger, constrain phase_status values (mirror PhaseStatus) and decide whether fail-open is acceptable for a control decision (it is not for resume)." + }, + { + "ref": "src/zo/wrapper.py:481", + "what": "`monitor_team` reads ~/.claude/teams/{team}/config.json (wrapper.py:1013); `read_task_list` (wrapper.py:499) reads ~/.claude/tasks/{team}. Both feed the `on_status` callback each poll cycle.", + "action": "Read-only dependency \u2014 this is the WS-E seam where heartbeat freshness should later join TeamStatus for `zo status`/HUD." + } + ], + "hazards": [ + { + "ref": "src/zo/wrapper.py:277", + "hazard": "In tmux mode (the default for `zo build`) LeadProcess is created with `pid=None` \u2014 only `tmux_pane_id` is recorded. Phase 3's \"PID + process-start-time identity\" therefore has no PID to work with in the primary path; only the headless path (wrapper.py:466) carries a real pid.", + "mitigation": "Capture the pane's process pid at launch (tmux display-message -p '#{pane_pid}', then resolve the claude child) and store it plus its OS start time on LeadProcess before any watchdog logic is written." + }, + { + "ref": "src/zo/surrogate.py:279", + "hazard": "`_pid_alive` is `os.kill(pid, 0)` with no start-time comparison, and `register_session` (surrogate.py:254-267) records `started_at` as the registration timestamp, not the process start time. A recycled PID reads as live, which suppresses stale-overlay cleanup (cli.py:772-780) and blocks auto-consolidation (cli.py:953-957) forever.", + "mitigation": "Record OS process start time in the lock JSON and require (pid, start_time) to match before treating a lock as live; sweep_locks (surrogate.py:293) is the single chokepoint to change." + }, + { + "ref": "src/zo/wrapper.py:672", + "hazard": "`_wait_tmux` has no rate-limit detection whatsoever \u2014 the entire rate-limit code path (wrapper.py:793-812) lives only in `_wait_headless`. A rate-limited interactive build is invisible to the wrapper, so a naive watchdog would classify it as a stall and nudge it, violating the never-block taxonomy and failing oracle check #11.", + "mitigation": "Implement never-block classification from pane capture (wrapper.py:750 already captures 5 lines) BEFORE any nudge, and gate the nudge on a positive not-rate-limited/not-context-limit/not-auth-error verdict." + }, + { + "ref": "src/zo/wrapper.py:53", + "hazard": "`_RATE_LIMIT_PATTERNS` includes a bare `re.compile(r\"429\", re.IGNORECASE)` matched against a raw stdout tail (wrapper.py:792 `_read_tail`). Any output containing the substring 429 (a loss value 0.429, a byte count, a file path, a line number) triggers a false rate-limit, currently costing a 30s+ backoff and after 3 hits a terminal RATE_LIMITED.", + "mitigation": "Tighten to structured signals (HTTP status field in the --output-format json stream, or 'status 429'/'429 Too Many Requests' with word boundaries) before the never-block taxonomy is built on top of this predicate." + }, + { + "ref": "src/zo/cli.py:920", + "hazard": "`_launch_and_monitor` is straight-line: one launch (cli.py:804), one wait (cli.py:920), then unconditional teardown \u2014 session summary (cli.py:934), next steps (cli.py:941), orchestrator.end_session (cli.py:947), surrogate deregister + consolidate (cli.py:953-957). There is no restart seam, so \"escalate to iteration restart\" cannot be added without restructuring this function.", + "mitigation": "Extract launch+wait into an inner function and wrap it in a bounded restart loop, keeping register/deregister paired per attempt and end_session/consolidate outside the loop." + }, + { + "ref": "src/zo/project_config.py:28", + "hazard": "ProjectConfig has no `model_config`, so pydantic v2 ignores unknown keys: a hand-added `watchdog:` block in .zo/config.yaml is silently dropped today with no warning, and `save_project_config` (project_config.py:144) rewrites the file from `model_dump()`, permanently deleting it. Operators who follow specs/watchdog.md:62-68 before the code lands will silently lose config.", + "mitigation": "Land the WatchdogConfig model in the same PR as any doc that advertises the block; consider `model_config = ConfigDict(extra='forbid')` or an explicit unknown-key warning so silent drops become visible." + }, + { + "ref": "src/zo/cli.py:3491", + "hazard": "Only `zo build` assembles extra_env (cli.py:1056-1078). The report launch (cli.py:3491), the init-architect launch (cli.py:2002) and the draft launch (cli.py:3168) all call `_launch_and_monitor` WITHOUT `extra_env`, so ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT are unset in those sessions. A heartbeat writer keyed on ZO_MEMORY_ROOT would be blind for every non-build session, and hookkit falls back to the repo default (hookkit.py:64-73).", + "mitigation": "Either set the env inside `_launch_and_monitor` from `delivery_repo`, or accept that heartbeats only cover build/continue sessions and document it; do not assume ZO_MEMORY_ROOT is always present." + }, + { + "ref": "src/zo/ledger.py:93", + "hazard": "`load_ledger` is fail-open \u2014 any OSError or malformed JSON returns None. `zo status` degrades gracefully (cli.py:2320 `if ledger_doc is not None`), but if session-restore is cut over to the ledger, a corrupt ledger would silently fall back to \"no saved phase states\" and re-run completed phases. `phase_status` is also an unvalidated `dict[str, str]` (ledger.py:74), so it cannot deliver PR-036's parse-time validation.", + "mitigation": "Add a strict-load variant for control decisions that raises with the file path, offending field, bad value, and valid options (PR-036 rule 1, memory/zo-platform/PRIORS.md:1052), and constrain phase_status values to the PhaseStatus enum with a drift-guard test." + }, + { + "ref": "docs/cli/build.mdx:133", + "hazard": "Documented: \"multi-phase orchestration requires the interactive session; --no-tmux is for init and draft, not full builds\". No doc anywhere shows the Linux demo box running a headless build \u2014 every headless demo reference is `zo init --no-tmux` (docs/SAMPLE_PROJECT.md:19, docs/reference/cost-benchmark.mdx:173). Oracle check #13 on a Linux box therefore has an unvalidated assumption about which mode it runs in.", + "mitigation": "Decide explicitly whether the demo box runs tmux (then the fresh-context builder is a headless CHILD of an interactive lead) or headless end-to-end (then this doc claim must be retested and fixed); do not let the choice be implicit." + }, + { + "ref": "src/zo/cli.py:695", + "hazard": "`_generate_session_summary` uses bare `\"claude\"` (not `_resolve_claude_bin`, wrapper.py:904-915), passes no env and no cwd, and swallows every exception with `except Exception: pass`. Copying this shape into the fresh-context per-iteration spawner would make iteration failures completely silent.", + "mitigation": "Base the fresh-context spawner on `_launch_headless` (wrapper.py:414-461) \u2014 resolved binary, explicit env, stdout/stderr to log files, comms checkpoint on launch \u2014 and log every non-zero exit." + }, + { + "ref": "src/zo/wrapper.py:551", + "hazard": "`poll_interval` defaults to 10.0 and `timeout` defaults to None; the CLI never passes either (cli.py:920-923). So there is no global run timeout today, and a watchdog poll cycle is hard-wired to 10s. The 38-hour silent stall in specs/watchdog.md:5 is consistent with a poll loop that runs forever and never evaluates progress.", + "mitigation": "Thread both from the watchdog config, and be aware that the debounce constants (_STARTUP_GRACE_POLLS=2, _DEAD_CONFIRM_POLLS=2, _DEAD_RECHECK_INTERVAL=2.0 at wrapper.py:72-81) are expressed in poll counts \u2014 changing poll_interval silently changes the grace window." + }, + { + "ref": "src/zo/wrapper.py:647", + "hazard": "`wait_for_completion` stashes per-run state as ad-hoc attributes on self (self._gate_mode_file, self._last_gate_mode, self._training_pane_id, self._project_name, self._delivery_repo at wrapper.py:576-580) rather than in a state object, and `_check_gate_mode_change` reads them via getattr with defaults. Adding watchdog counters (tick count, nudges spent, pause deadline) the same way makes a restart loop error-prone, since stale counters persist across a re-entry.", + "mitigation": "Introduce a small per-run state dataclass reset at the top of wait_for_completion so a restart begins with a fresh nudge budget and tick counter." + } + ], + "reusable": [ + { + "ref": "src/zo/wrapper.py:414", + "what": "`_launch_headless` \u2014 the canonical headless `claude -p` spawn: cmd assembly (wrapper.py:427-441: --print, --output-format json, --model, --max-turns, --add-dir, optional --dangerously-skip-permissions, then -p prompt), env via os.environ.copy() + extra_env (wrapper.py:456-459), Popen with stdout/stderr file handles (wrapper.py:461), LeadProcess with real pid + started_at (wrapper.py:465-469). Reuse verbatim for the fresh-context per-iteration builder." + }, + { + "ref": "src/zo/wrapper.py:904", + "what": "`_resolve_claude_bin` \u2014 resolves the absolute claude path via `which`, with graceful fallback. Any new spawn must use this rather than a bare \"claude\" string." + }, + { + "ref": "src/zo/wrapper.py:648", + "what": "`_check_gate_mode_change` \u2014 the existing pattern for a cheap file-mtime/content poll inside the wait loop that only speaks on change. The heartbeat freshness checker should be structurally identical (specs/watchdog.md:33 'cheap + quiet')." + }, + { + "ref": "src/zo/wrapper.py:592", + "what": "`_maybe_open_training_pane` \u2014 an existing 'external checker in the poll loop that fires at most once and degrades silently' pattern, including the `self._training_pane_id = \"\"` sentinel for 'attempted, do not retry'. Good template for a one-shot escalation." + }, + { + "ref": "src/zo/ledger.py:81", + "what": "`_atomic_write` (mkstemp + os.replace in the target dir) and the `_mutate` read-modify-write wrapper (ledger.py:173). Heartbeat JSON writes should reuse this so a partially written heartbeat can never be read as stale/corrupt." + }, + { + "ref": "src/zo/memory.py:158", + "what": "MemoryManager's tmp-then-replace STATE.md write (`.STATE.md.tmp`) and `_append_locked` (memory.py:178) for concurrent appends \u2014 the existing precedent for multi-writer safety under the project memory root, which is where heartbeats will live." + }, + { + "ref": "src/zo/surrogate.py:240", + "what": "`register_session` / `deregister_session` / `sweep_locks` / `live_sessions` \u2014 a complete per-PID JSON lock registry under /.zo/surrogates/locks/. Heartbeats are the same shape (per-agent JSON under the memory root); extend this module rather than inventing a parallel registry." + }, + { + "ref": "src/zo/cli.py:772", + "what": "The `_peers_live` guard pattern: bookkeeping wrapped in try/except so it 'can never break a run' (cli.py:747-749, cli.py:786-796). Watchdog bookkeeping must adopt the same fail-open posture." + }, + { + "ref": "src/zo/preflight.py:187", + "what": "`_check_docker` \u2014 the shutil.which + subprocess `--version` + timeout pattern, including warning-only CheckResult. Template for strengthening `_check_claude_cli` (preflight.py:84) per PR-046." + }, + { + "ref": "src/zo/experiment_loop.py:126", + "what": "`resolve_policy` \u2014 the documented precedence merge (CLI override > plan spec > low_token clamp > base default) over a pydantic model dump. Reuse this exact shape for watchdog config precedence (CLI flag > project_config watchdog block > defaults); the CLI-side twin is `_resolve_gate_mode` / `_resolve_lead_model` at cli.py:272-298." + }, + { + "ref": "src/zo/comms.py:1", + "what": "CommsLogger (log_checkpoint / log_error / log_decision / log_gate) is already the audit bus that both the wrapper (wrapper.py:756) and _print_status (cli.py:869-913) read; specs/watchdog.md:76 explicitly routes nudges through it. No new event channel is needed." + }, + { + "ref": "src/zo/hookkit.py:64", + "what": "`_memory_root(repo_root)` \u2014 resolves ZO_MEMORY_ROOT with a repo-relative fallback. Any hook-side heartbeat writer should call this rather than re-deriving the path." + }, + { + "ref": "src/zo/cli.py:2317", + "what": "The `zo status` ledger-rendering block (Rich table over ledger_doc.phase_status + summarize counts, with `if ledger_doc is not None` degradation). WS-E heartbeat rendering should extend this table rather than adding a separate view." + } + ], + "open_questions": [ + "Which process does the watchdog track as 'the agent' in tmux mode? LeadProcess.pid is None there (wrapper.py:277) and only the pane id exists; is the target the pane's shell pid, the claude child, or per-subagent pids that ZO never sees at all? The team/task state ZO reads (~/.claude/teams, wrapper.py:1013) contains no pids.", + "Who writes the per-agent heartbeats? Nothing in src/ writes one today. Options: a Claude Code hook (hookkit.py already has PostToolUse/Stop/SubagentStop handlers at hookkit.py:120/196/287 and the shim at .claude/hooks/zo-hookkit.sh routes events), or the lead agent via an explicit instruction. Hook-based writing is the only 'evidence not self-report' option, but .claude/hooks/zo-hookkit.sh:19 restricts hooks to the ZO platform repo \u2014 delivery repos carry no ZO hooks, so a delivery-repo session would emit no heartbeats.", + "Does the never-block taxonomy get its evidence from the same source in both modes? Headless has a JSON stdout log (wrapper.py:428 --output-format json, tail read at wrapper.py:792); tmux only has a 5-line pane capture (wrapper.py:750). Context-limit / auth-error / user-abort detection may not be reliably derivable from a 5-line pane snapshot.", + "Is `zo build --no-tmux` actually expected to work for a full multi-phase build on the Linux demo box? docs/cli/build.mdx:133 says no. If oracle check #13 runs headless, that doc claim needs retesting; if it runs under tmux, the fresh-context builder is a headless child of an interactive lead and needs its own liveness identity separate from the lead's pane.", + "Where does the fresh-context loop live \u2014 inside the agent session (lead spawns children) or above it (CLI/experiment_loop spawns a fresh `claude -p` per iteration)? The plan says 'experiment_loop.py spawns a fresh builder per iteration' (plans/zo-v2-rearchitecture.md:111-113), but experiment_loop.py is a pure decision module today (experiment_loop.py:56-66) and the CLI's single-session shape (cli.py:1150) means nothing above the session currently loops.", + "Should a watchdog-triggered iteration restart re-enter `get_current_phase` (orchestrator.py:687) \u2014 i.e. re-derive the phase from disk \u2014 or re-use the in-memory phase? Re-deriving is the fresh-context principle but requires the ledger cutover to have landed first, since STATE.md is written only at end_session (cli.py:947 -> orchestrator.py:285) and on subtask completion (orchestrator.py:861).", + "What is the intended interaction between the watchdog's rate-limit pause and the existing max_retries=3 / exponential backoff (wrapper.py:87, wrapper.py:898)? Does wait-and-resume replace the retry counter entirely, or does it apply only when a reset timestamp can be parsed, falling back to backoff otherwise?", + "Legacy-layout projects have no .zo/config.yaml (cli.py:88 falls through to parse_target) and therefore no place for a watchdog block. Is the watchdog default-on with hardcoded defaults for legacy projects, or off?" + ], + "notes": "Read-only recon; no files modified. Branch claude/v2-phase3-substrate at 1faf53a (Phase 2 merge).\n\nThree structural facts that most constrain Phase 3:\n\n1. The CLI has no loop anywhere. One `zo build`/`zo continue` = one phase = one lead session (cli.py:1134 -> cli.py:1150), and `_launch_and_monitor` is straight-line launch->wait->teardown (cli.py:804 / cli.py:920 / cli.py:930-957). Both watchdog escalation-to-restart and the fresh-context per-iteration loop need a caller-level loop that does not exist today.\n\n2. The rate-limit machinery is in the wrong branch. It lives entirely in `_wait_headless` (wrapper.py:793-812) while the default build path is `_wait_tmux` (wrapper.py:672-761), which has none. So oracle check #11 (\"a rate-limited session is NOT nudged\") currently has no signal to test against in the mode it will run in, and the never-block taxonomy must be built from pane capture, not from the existing `_detect_rate_limit` (which is also fired by a bare `429` substring, wrapper.py:53).\n\n3. Identity is not durable. tmux LeadProcess carries `pid=None` (wrapper.py:277); the surrogate lock file records a registration timestamp rather than a process start time (surrogate.py:261) and liveness is a bare `os.kill(pid, 0)` (surrogate.py:279-290). The \"recycled PIDs are never acted on\" requirement needs both a real pid for the tmux lead and a start-time field in the lock schema.\n\nTwo smaller items worth flagging: PR-046's explicit claude-CLI preflight check already exists (preflight.py:84, wired at preflight.py:70) and only needs strengthening beyond `shutil.which`; and `zo watch` does not exist as a CLI command at all \u2014 only `zo watch-training` (cli.py:2912) plus the `/watch` slash command (.claude/commands/observe/watch.md), which does exactly the `ps aux | grep claude` liveness guessing the heartbeat is meant to replace." + }, + { + "surface": "Test + docs conventions Phase 3 (WS-C execution substrate: watchdog + fresh-context loop) must follow in /Users/sam101fe4x/Documents/code/zero-operators on branch claude/v2-phase3-substrate. Covers: pytest fixture/mocking conventions, the seeded-failure oracle-check test pattern established by Phases 1-2, the CI + validate-docs gates, the docs cascade, the RFC-vs-plan design divergence in specs/watchdog.md, and the memory-protocol files to update.", + "integration_points": [ + { + "ref": "tests/conftest.py:24-69", + "what": "Only 4 shared fixtures exist repo-wide: sample_plan_path, sample_target_path, tmp_project_dir (copies tests/fixtures/test-project/, creates logs/comms + memory), comms_logger (CommsLogger to tmp_path). There is NO shared tmp_path-based memory_root or registry fixture \u2014 every test module defines its own locally.", + "action": "Read-only dependency. Define watchdog fixtures locally in tests/unit/test_watchdog.py (module-scope, like test_wrapper.py) rather than widening conftest.py; only promote to conftest.py if both the unit and integration watchdog tests need the identical heartbeat-root fixture." + }, + { + "ref": "tests/unit/test_wrapper.py:28-46", + "what": "The wrapper test fixture triad Phase 3 must reuse verbatim: tmp_log_dir (tmp_path/logs/wrapper), comms (CommsLogger), wrapper (LifecycleWrapper(comms, log_dir=tmp_log_dir)). All watchdog tests that drive the poll loop need these three.", + "action": "Reuse these three fixtures as-is in the new watchdog test module. Do not re-invent a LifecycleWrapper constructor fixture." + }, + { + "ref": "tests/unit/test_wrapper.py:190-191", + "what": "The canonical mocking idiom: @mock.patch(\"zo.wrapper.time.sleep\") stacked with @mock.patch(\"zo.wrapper.subprocess.run\") / (\"zo.wrapper.subprocess.Popen\"). Patch target is always the module-qualified name zo.wrapper.X, never the global. os.kill is patched as \"zo.wrapper.os.kill\" (test_wrapper.py:649), atexit as \"zo.wrapper.atexit.register\" (test_wrapper.py:242).", + "action": "Mock the watchdog poll clock the same way. A 10-minute stall (oracle check 11) must be simulated by controlling mocked time, never by real sleeping \u2014 patch zo.wrapper.time.sleep AND inject monotonic/heartbeat mtimes, so the whole test runs in milliseconds." + }, + { + "ref": "tests/unit/test_wrapper.py:559-574", + "what": "Precedent for testing a poll-loop counter: mock.patch.object(LifecycleWrapper, \"_tmux_pane_alive\", return_value=False) then assert alive_mock.call_count == 4 to prove the grace/confirm poll budget. Class-level constants _STARTUP_GRACE_POLLS=2 / _DEAD_CONFIRM_POLLS=2 are asserted through observed call counts.", + "action": "Model the bounded nudge budget the same way: expose it as a LifecycleWrapper (or Watchdog) class constant and assert the exact nudge call_count. This is the established way this repo proves a budget is bounded." + }, + { + "ref": "tests/unit/test_wrapper.py:594-609", + "what": "The negative-assertion idiom for 'must NOT act': drive the loop with side_effect=[False] + [True]*50 and timeout=-1, then assert the TIMED_OUT branch was reached \u2014 proving the intervention did NOT fire. Docstring names the regression it locks.", + "action": "Use exactly this shape for the never-block half of oracle check 11 ('a rate-limited session is NOT nudged'): let the loop run to a terminal non-nudge state and assert nudge mock was never called plus the terminal status." + }, + { + "ref": "src/zo/wrapper.py:763-822", + "what": "_wait_headless is the ONLY poll loop with rate-limit handling. while True at :776; _detect_rate_limit called at :793 (the sole call site \u2014 verified by grep, :793 and the def at :894); retries/_max_retries branch at :794; _backoff_wait at :801; time.sleep(wait_secs) at :806; RATE_LIMITED set at :795.", + "action": "This is where the never-block taxonomy and wait-and-resume must land. Replace the retry-loop (:801-808) with pause-until-reset semantics for the rate-limit case, keeping the existing _detect_rate_limit patterns as the classifier input." + }, + { + "ref": "src/zo/wrapper.py:669-761", + "what": "_wait_tmux \u2014 the default zo build path \u2014 has while True at :709 and time.sleep(poll_interval) at :761, but ZERO rate-limit handling (grep for _detect_rate_limit returns only :793 and the def at :894; an awk scan of 669-761 for /rate|_detect|429/ returns nothing).", + "action": "Add the external heartbeat checker to BOTH poll loops, and add rate-limit classification to _wait_tmux. If the checker only lands in _wait_headless, oracle checks 11-12 pass in tests while the default operator path stays unprotected." + }, + { + "ref": "src/zo/wrapper.py:74,78", + "what": "_STARTUP_GRACE_POLLS = 2 and _DEAD_CONFIRM_POLLS = 2 \u2014 the existing class-constant convention for poll-loop tuning knobs, with _max_retries set from a constructor arg at :96.", + "action": "Add watchdog knobs (stall_threshold, nudge budget, poll cadence) as sibling class constants here so tests can override them per-test the way test_wrapper.py:528 does `wrapper._max_retries = 2`." + }, + { + "ref": "src/zo/experiment_loop.py:205-209", + "what": "evaluate_loop_state(registry: ExperimentRegistry, phase: str, policy: LoopPolicy | None = None) -> LoopDecision. Takes the experiment registry ONLY \u2014 no ledger parameter. This is the signature the folded-in deferral must change.", + "action": "Add the ledger as an optional keyword-only param with a None default (e.g. ledger: LedgerFile | None = None) so all 16 existing positional call sites keep compiling. A required positional would force edits to every one of them." + }, + { + "ref": "tests/unit/test_experiment_loop.py:102", + "what": "First of exactly 16 evaluate_loop_state(...) call sites in this file (verified: grep -c 'evaluate_loop_state(' = 16, at lines 102,111,122,137,145,150,159,176,185,196,218,232,243,259,312,328). All are positional: evaluate_loop_state(reg, \"phase_4\", policy).", + "action": "Count these 16 when scoping the ledger-read deferral. A keyword-only optional ledger param leaves all 16 untouched; any positional insertion breaks all 16." + }, + { + "ref": "tests/unit/test_experiment_loop.py:62-92", + "what": "The registry 'fixture' is NOT a pytest fixture \u2014 it is two module-level builder helpers: _exp(exp_id, *, phase, parent_id, tier, metric_value, delta_vs_parent, status, minutes_ago) at :62 and _registry(*exps) -> ExperimentRegistry(project=\"demo\", ...) at :91.", + "action": "Reuse _exp/_registry for any fresh-context-loop test needing lineage. If ledger state is also needed, add a parallel module-level _ledger(...) builder in the same style rather than a pytest fixture." + }, + { + "ref": "src/zo/orchestrator.py:1276", + "what": "The single production call site: decision = evaluate_loop_state(registry, phase.phase_id, policy), inside a lazy import block at :1265-1269 that also pulls LoopVerdict and resolve_policy. Registry is loaded at :1270 via load_registry(exp_dir).", + "action": "Load the ledger next to load_registry here and pass it through. Keep the lazy-import block pattern \u2014 it is deliberate (avoids import cycles)." + }, + { + "ref": "tests/integration/test_hooks_shim.py:113-146", + "what": "class TestSettingsWiring with the docstring 'The settings.json must actually reference every new hook (unwired mechanisms are the #1 anti-pattern from the v2 review)'. test_all_ws_a_events_wired at :117 loads .claude/settings.json and asserts each event's command list contains 'zo-hookkit.sh'.", + "action": "Phase 3 MUST add the analogous wiring assertion \u2014 a test proving the watchdog checker is actually invoked from the LifecycleWrapper poll loop, not merely defined. plans/zo-v2-rearchitecture.md:74 makes this a merge gate." + }, + { + "ref": "tests/integration/test_hooks_shim.py:29-38", + "what": "_run_shim helper: subprocess.run([\"bash\", str(SHIM), event], input=json.dumps(payload), capture_output=True, text=True, timeout=30, env={**os.environ, **overrides}, cwd=str(REPO_ROOT), check=False) with SHIM resolved at :27 via Path(__file__).resolve().parents[2].", + "action": "Reuse this REPO_ROOT/parents[2] + subprocess-with-timeout shape for any Phase 3 integration test that must drive a real external process (stall harness, git-commit checkpoint)." + }, + { + "ref": "src/zo/contracts.py:151-161", + "what": "The atomic-write pattern used for all control-plane JSON: mkdir(parents=True, exist_ok=True), tempfile.mkstemp(dir=str(memory_root), suffix=\".tmp\"), os.fdopen write, os.replace(tmp, path), finally unlink. Docstring at :180-183 states the rationale: 'a torn read in the fail-open hook layer would silently disable contract enforcement.'", + "action": "Write heartbeat JSON with exactly this pattern. Heartbeats are read by an external checker concurrently with agent writes \u2014 a torn read is the same hazard class and would produce false stall detections." + }, + { + "ref": "src/zo/contracts.py:164-169", + "what": "load_contracts: returns None on any OSError/ValueError \u2014 the fail-open read convention paired with the atomic write.", + "action": "Mirror this for load_heartbeat. A missing/corrupt heartbeat must be distinguishable from a stale one: returning None (unknown) must NOT be treated as 'stalled' or the never-block taxonomy is bypassed on first read." + }, + { + "ref": "src/zo/hookkit.py:47-50", + "what": "_SEALED_DEFAULTS = (\"gate_mode\", \"gate_nonce\", \"gate_decision\", CONTRACTS_FILENAME, \"plan-ledger.json\", \"sealed_paths\") \u2014 named files under memory_root only. Prefix match at :340-342 is `resolved == anchor or resolved.startswith(anchor.rstrip(\"/\") + \"/\")`.", + "action": "Heartbeat files under memory_root are NOT sealed today. Decide explicitly: add the heartbeat dir name to this tuple so agents cannot forge their own liveness via the Write tool (the wrapper/hook writes bypass PreToolUse, so sealing is safe), or document why forgery is acceptable." + }, + { + "ref": "src/zo/cli.py:1076-1077", + "what": "extra_env[\"ZO_MEMORY_ROOT\"] = str(memory.memory_root); extra_env[\"ZO_DELIVERY_ROOT\"] = str(target.target_repo) \u2014 the env contract between the CLI and hook/subprocess layer, consumed at hookkit.py:65 and :135.", + "action": "The watchdog checker runs in-process in the wrapper, so resolve the heartbeat root from the same memory_root object rather than re-reading ZO_MEMORY_ROOT. If a subprocess needs it, extend this block (it is the single source of the env contract)." + }, + { + "ref": "src/zo/hookkit.py:64-70", + "what": "_memory_root(repo_root): ZO_MEMORY_ROOT env first, else repo_root/\"memory\"/\"zo-platform\" if it is_dir(), else None. Delivery projects use delivery_repo/.zo/memory (cli.py:69, consolidate.py:77).", + "action": "Read-only dependency \u2014 reuse this resolution order for the heartbeat root so hooks and the wrapper agree on where heartbeats live. Two different roots would make the checker read an empty dir and report a false stall." + }, + { + "ref": "src/zo/comms.py:31-38", + "what": "EventType StrEnum with exactly five canonical members (MESSAGE/DECISION/GATE/ERROR/CHECKPOINT), docstring 'The five canonical event types in the ZO audit trail'. Dispatch table at :166-170; models at :101-160.", + "action": "Do NOT add a sixth event type for watchdog nudges. Emit stalls via log_error (wrapper.py:796 precedent: error_type=\"rate_limit\", severity=\"blocking\") and nudges/resumes via log_checkpoint (wrapper.py:802 precedent, subtask=\"rate-limit-backoff\"). A new member forces edits to comms.py, the dispatch table, and specs/comms.md:48." + }, + { + "ref": "specs/comms.md:36-48", + "what": "'## JSONL Audit Schema' / '### Base Schema' with the literal line \"event_type\": \"message | decision | gate | error | checkpoint\" at :48. Per-type sections at :54 (Message), :74 (Decision), :94 (Gate), :121 (Error), :141 (Checkpoint).", + "action": "Only touch this if a new event type is added (recommended: don't). If watchdog reuses error+checkpoint, no spec edit is needed here." + }, + { + "ref": "specs/memory.md:279-301", + "what": "'### Session Recovery' section. :291 is the '**Recovery mechanism (implemented, v2 WS-A3)**' block listing the four hooks. :283-288 numbers the recovery steps; step 4 at :286 reads 'New session reads STATE.md and picks up from last_completed_subtask'.", + "action": "Edit :286 for the session-restore cutover \u2014 STATE.md is no longer the control-decision parse target (plans/zo-v2-rearchitecture.md:103-105 already says STATE.md becomes 'a projection for humans, never the parse target'). Add a watchdog bullet to the :291 mechanism list." + }, + { + "ref": "specs/workflow.md:511", + "what": "'## Phase 4: Training and Iteration'. Subtask 4.1 Baseline Training Run at :524, 4.2 Training Diagnostics at :535, 4.3 Iteration Protocol at :547, 4.4 Cross-Validation at :573, 4.5 Ensemble at :584, Gate 4 Human Model Approval at :758.", + "action": "Subtask 4.3 'Iteration Protocol' (:547) is the exact anchor for the fresh-context loop \u2014 that section currently describes iteration without stating that each iteration is a fresh session. Update it, not the Phase 4 header." + }, + { + "ref": "docs/reference/v2-rearchitecture.mdx:65", + "what": "Feature #2 Watchdog row. Table header at :63 is `| # | Feature | From | Priority |` \u2014 there is NO Status column anywhere in this file. The row carries Priority 'P0' and provenance 'oh-my-claudecode + ruflo'; it carries no status text at all.", + "action": "Answers the question directly: there is no per-row status to update. To reflect Phase 3 shipping you must either add a Status column to all five workstream tables (:44, :62-ish, :63, :78, :92) or update the prose in '## How it ships' (:87-103). Adding a column to only WS-C would break table consistency." + }, + { + "ref": "docs/reference/v2-rearchitecture.mdx:66", + "what": "Feature #6 Fresh-context-per-subtask row, Priority P1, same statusless table.", + "action": "Same as #2 \u2014 no status field exists. Note the row already claims 'a new agent per iteration re-derives state from the ledger, experiment lineage, and a curated priors digest', which is exactly the Phase 3 deliverable; the text needs no change, only a status mechanism." + }, + { + "ref": "docs/reference/v2-rearchitecture.mdx:99", + "what": "'3. **Test** \u2014 the 854-test platform suite stays green on Python 3.11 and 3.12'. The literal 854 is hardcoded in prose here, and also at plans/zo-v2-rearchitecture.md:67 (oracle check 20) and :132 (Phase 6).", + "action": "Phase 3 adds tests, so 854 becomes wrong in three places. Update all three together, or rephrase to 'the platform suite'. Note validate-docs does NOT check these prose numbers \u2014 only the README badge." + }, + { + "ref": "README.md:13", + "what": "[![Tests](https://img.shields.io/badge/tests-854_passing-...)] \u2014 the badge validate-docs Check 6 parses via 'tests-[0-9]+'.", + "action": "Update to the post-Phase-3 count. It is already stale (see hazards) so Phase 3 should correct it regardless." + }, + { + "ref": "README.md:529", + "what": "'780 platform tests. ruff clean (`src/zo/`). 21 agents. 24 slash commands.' \u2014 a second, different, hand-maintained test count in the Status section.", + "action": "Update alongside :13. These two numbers already disagree with each other (780 vs 854) and both disagree with reality (917)." + }, + { + "ref": "memory/zo-platform/STATE.md:65", + "what": "'- [x] Slash commands: 24 commands across 8 categories' \u2014 the exact line validate-docs Check 3 greps via 'commands across' (validate-docs.sh:123) and compares to find .claude/commands -name '*.md' | wc -l (currently 24). This is a HARD FAIL check.", + "action": "Only touch if Phase 3 adds a .claude/commands/*.md slash command. A `zo watchdog` CLI subcommand does NOT count \u2014 Check 3 counts .claude/commands/ files only." + }, + { + "ref": "memory/zo-platform/STATE.md:9-11", + "what": "'## Current Position' at :9, then the top entry at :11 beginning '**Session 040 (current) \u2014 pick up here.**' \u2014 one dense bolded paragraph carrying shipped artifacts, branch names, PR numbers, and verification caveats. Front-matter at :3-7 is project/mode/phase/iteration/status.", + "action": "Prepend a 'Session 041 (current) \u2014 pick up here.' paragraph in this exact style and demote 040 (strip '(current) \u2014 pick up here'). Follow the existing convention of naming the branch and PR number inline." + }, + { + "ref": "memory/zo-platform/DECISION_LOG.md:1-5", + "what": "Header '# DECISION_LOG \u2014 Zero Operators Platform Build' + 'Append-only. Every orchestration decision with timestamp, rationale, and outcome.' Entry format (first entry at :7-14): '## Decision: ' then bolded **Type:** / **Title:** / **Decision:** / **Rationale:** / **Alternatives considered:** / **Outcome:**, separated by '---'.", + "action": "Append (never edit) at least two entries: the watchdog RFC-vs-plan design divergence (cron-tick vs wrapper-poll-loop) and the evaluate_loop_state ledger-signature choice. Type: ARCHITECTURE. File is 1279 lines \u2014 append at EOF." + }, + { + "ref": "memory/zo-platform/PRIORS.md:1057-1101", + "what": "PR-036 entry: '## PR-036: STATE.md Schema Validation Belongs at the Parser, Not the Consumer', with sections **Source:** / **Root cause category:** / **Failure:** / '### Rules' (numbered, each with **Why:** and **How to apply:**) / '### Verified Solution' / closing test-count line.", + "action": "Read-only dependency for the session-restore cutover \u2014 Rule 1 (:1073) mandates parse-time validation at the memory boundary and Rule 2 (:1076) mandates the parse-time validator + drift-guard test ship in the SAME PR. Any ledger-backed restore must carry its own parse-time validator." + }, + { + "ref": "memory/zo-platform/PRIORS.md:1103-1152", + "what": "PR-037 entry: get_current_phase must return ACTIVE on resume. Rule 3 at :1148 states 'Priority order between status values is part of the contract. GATED > ACTIVE > PENDING isn't arbitrary'. Rule 2 at :1143 requires 'at least one test per non-terminal status the producer can write'. Names the five locking tests at :1150-ish including test_real_resume_via_state_md_round_trip.", + "action": "The ledger cutover MUST preserve GATED > ACTIVE > PENDING and MUST port all five named tests to the ledger-backed path. Losing test_real_resume_via_state_md_round_trip would silently drop the prod-001 regression guard." + }, + { + "ref": "memory/zo-platform/sessions/session-040-2026-08-12.md:1-3", + "what": "Last session file. Format: '# Session 040 \u2014 2026-08-12 \u2014 ' then '**Type:** Research session (no platform code changes, no commits)'. Headings: '## What happened' (:5), '## Headline conclusions' (:22), '## Decision (same session)' (:37), '## Phase 1 build (same session, part 2)' (:60), '## Live verification + merges + Phase 2 (same session, parts 3-4)' (:78), '## Next session \u2014 pick up here' (:92). 109 lines.", + "action": "Create memory/zo-platform/sessions/session-041-<YYYY-MM-DD>.md in this exact shape. The '## Next session \u2014 pick up here' closing section is mandatory \u2014 it is how session 041 was expected to bootstrap. Note the numbering has gaps (009-040, no 041 yet) so 041 is correct." + }, + { + "ref": "tests/unit/test_cli.py:28-38", + "what": "test_cli_group_has_all_commands: `expected = {...13 names...}` then `assert expected <= actual` (subset, not equality) with a comment at :34-37 explaining CLI plugins legitimately add commands.", + "action": "Direct answer: adding `zo watchdog` will NOT break this test \u2014 the subset assertion tolerates new commands. No edit required. If you want the command covered, add it to the `expected` set deliberately." + }, + { + "ref": "docs/COMMANDS.md:13-17", + "what": "'## CLI Commands' at :13, then one '### zo <name>' subsection per command (zo build :17, zo continue :37, zo draft :53, zo init :84, zo status :119, zo migrate :131, zo preflight :143, zo gates set :151, zo status (control plane) :161, zo gates approve/reject :169, zo watch-training :189, zo experiments :199, zo learnings promote :213, zo report :223, zo consolidate :235). '## Slash Commands' at :247 with category headings.", + "action": "Add a '### zo watchdog' subsection under '## CLI Commands' if a CLI surface ships. Verified read-only: no test and no validate-docs check parses COMMANDS.md, so this is convention-only \u2014 but every existing CLI command has an entry." + }, + { + "ref": ".github/workflows/ci.yml:26-28", + "what": "matrix.python-version = [\"3.11\", \"3.12\"], fail-fast: false, runs-on ubuntu-latest.", + "action": "Read-only dependency. Watchdog code must be 3.11-compatible (pyproject requires-python >=3.11, ruff target-version py311 at pyproject.toml:73). Process-start-time reading must work on Linux CI and macOS dev." + }, + { + "ref": ".github/workflows/ci.yml:41", + "what": "'run: uv sync --extra dev' \u2014 dependencies resolve from the committed uv.lock (303KB, present at repo root, last touched 2026-08-06).", + "action": "If psutil is added to pyproject dependencies, uv.lock MUST be regenerated in the same PR or CI fails at the sync step before any test runs." + }, + { + "ref": ".github/workflows/ci.yml:44", + "what": "'run: uv run ruff check src/' \u2014 lint covers src/ ONLY, not tests/.", + "action": "New src/zo/watchdog.py must be ruff-clean under the pyproject.toml:78 rule set (E,F,I,N,W,UP,B,SIM,TCH). Test files are unlinted in CI, matching the existing noqa: SLF001 usage at test_wrapper.py:275." + }, + { + "ref": "scripts/validate-docs.sh:192-208", + "what": "Check 6, test count badge. Parses 'tests-[0-9]+' from README.md (:193), counts `grep -r \"def test_\" tests/ | wc -l` (:200), and warns (NOT fails) when abs diff > 5 (:203-207).", + "action": "Warn-only, so it cannot fail CI \u2014 but it is ALREADY tripping (see hazards). Update README.md:13 to the real post-Phase-3 count to clear it." + }, + { + "ref": "scripts/validate-docs.sh:110-129", + "what": "Check 3, HARD FAIL. ACTUAL_COMMANDS = find .claude/commands -name '*.md' | wc -l (:111, currently 24); compared against README.md 'N slash commands' (:114, README.md:307) and STATE.md 'N commands across' (:123, STATE.md:65).", + "action": "Direct answer to 'would a check fire if we add a CLI command like zo watchdog': NO for a CLI subcommand, YES for a .claude/commands/*.md slash command \u2014 and it is a hard fail requiring BOTH README.md:307 and STATE.md:65 updated in the same commit." + }, + { + "ref": "scripts/validate-docs.sh:46-74", + "what": "Check 1, HARD FAIL, three-way agent count: README 'agents-[0-9]+' badge (:50, README.md:14), setup.sh 'AGENT_COUNT -eq N' (:59), and .claude/agents/lead-orchestrator.md 'N pre-defined agents' (:68). Actual = 21. Check 7 (:215-222) adds a fourth: setup.sh 'All N agent'.", + "action": "Direct answer: only fires if Phase 3 adds a .claude/agents/*.md file. The plan explicitly does NOT add a monitor agent (external checker, not an agent) \u2014 so this stays green. If a watchdog agent is ever added, FOUR files must change together." + }, + { + "ref": "scripts/validate-docs.sh:81-103", + "what": "Check 2, HARD FAIL: agent filenames must exactly match setup.sh's EXPECTED_AGENTS array, compared bidirectionally with comm -23 / comm -13.", + "action": "Same trigger as Check 1 \u2014 no-op for Phase 3 unless an agent file is added." + }, + { + "ref": "scripts/validate-docs.sh:136-148", + "what": "Check 4, HARD FAIL: version must match across pyproject.toml `version = `, src/zo/__init__.py `__version__ = `, src/zo/cli.py `_VERSION = `. Currently 1.0.2 (pyproject.toml:3).", + "action": "If Phase 3 bumps the version, all three must move together in one commit." + }, + { + "ref": "scripts/validate-docs.sh:231-261", + "what": "Check 8, HARD FAIL: client confidentiality. Greps all tracked .md/.py/.yaml/.yml/.json/.sh files against scripts/.client-blocklist (gitignored). Currently warns 'Skipped \u2014 no blocklist file' locally (:244), meaning it is effectively inert in this working copy.", + "action": "Read-only. Do not put prod-001's real client name into any Phase 3 test fixture, spec, or session note \u2014 the check may be armed in another environment even though it is inert here." + }, + { + "ref": "scripts/validate-docs.sh:155-185", + "what": "Check 5, WARN-ONLY: agent model tier in .claude/agents/*.md frontmatter vs specs/agents.md '### ... model tier'.", + "action": "No-op for Phase 3 (no new agents). Listed for completeness \u2014 these 8 checks are the complete validate-docs surface." + }, + { + "ref": ".github/workflows/validate-docs.yml:16-18", + "what": "The validate-docs job is a bare `run: ./scripts/validate-docs.sh` with no setup steps; it fails the build on exit 1 (validate-docs.sh:273-277 exits 1 iff FAIL_COUNT > 0). Warnings never fail.", + "action": "Read-only dependency. Confirms: only the five HARD-FAIL checks (1,2,3,4,7,8) gate CI; the test badge and tier checks cannot block Phase 3." + }, + { + "ref": "specs/watchdog.md:3", + "what": "'**Status:** design spec (RFC). Implementation tracked as the follow-up to this PR.'", + "action": "Flip to implemented and record which sections were superseded (see the divergence hazard). This one line is the single most important doc edit in Phase 3 \u2014 leaving it as RFC after shipping is exactly the 'dead code presented as capability' anti-pattern the plan bans." + }, + { + "ref": "specs/watchdog.md:83-88", + "what": "'## 5. Acceptance / tests' \u2014 five bullets: stall-detection predicate unit test, remediation escalation state machine, config parse + re-arm idempotency, integration all-idle-vs-live-but-slow, regression on failure-signature filter.", + "action": "Bullet 3 ('re-arm idempotency (arming twice doesn't double-schedule)') is cron-specific and becomes moot under the poll-loop design. Rewrite this section to the checks 11-12 acceptance criteria; do not leave tests specified that will never exist." + } + ], + "hazards": [ + { + "ref": "specs/watchdog.md:40", + "hazard": "DESIGN DIVERGENCE (primary). The RFC's \u00a73.1 specifies 'A recurring, off-minute schedule (default `*/17 * * * *` ...) enqueues a **watchdog tick** prompt to the orchestrator' \u2014 a cron-driven, orchestrator-owned, LLM-in-the-loop tick. The Phase 3 plan at plans/zo-v2-rearchitecture.md:108-109 specifies the opposite: 'heartbeat JSON per agent + external checker in the LifecycleWrapper poll loop'. These are incompatible mechanisms (LLM self-invoke on a wall clock vs. deterministic Python in an existing poll loop), and the RFC never mentions the never-block taxonomy, bounded nudge budgets, PID+start-time identity, or rate-limit wait-and-resume that the plan makes central.", + "mitigation": "Before writing code, record the divergence as a DECISION_LOG entry (memory/zo-platform/DECISION_LOG.md format at :7-14) and rewrite specs/watchdog.md \u00a73.1 (:39-40), \u00a73.4 (:53-58), \u00a74 (:74-79) and \u00a75 (:83-88) to the poll-loop design, marking the cron/orchestrator-owned text as superseded rather than deleting it. Shipping the plan's design while the spec still describes cron is a documentation-reality split of exactly the kind validate-docs exists to prevent." + }, + { + "ref": "specs/watchdog.md:75", + "hazard": "The RFC's integration points name `orchestrator.py` ('arm the heartbeat cron on run/session start ... own the watchdog-tick handler') and `project_config.py` (:78, a `watchdog:` config block). The plan puts the checker in wrapper.py instead. A builder following the spec will wire the watchdog into the orchestrator and produce a mechanism that never runs during `zo build` \u2014 the LifecycleWrapper poll loop is what is actually live during a run.", + "mitigation": "Rewrite specs/watchdog.md:74-79 to name src/zo/wrapper.py's _wait_tmux (:669) and _wait_headless (:763) as the integration points. Then add the TestSettingsWiring-style runtime-caller test (tests/integration/test_hooks_shim.py:113) proving the checker is invoked from the poll loop." + }, + { + "ref": "specs/watchdog.md:79", + "hazard": "RFC line 79 says 'A dedicated monitor agent (if any) becomes a *helper*'. The Phase 3 brief explicitly says the checker is NOT a monitor agent. Leaving this line invites a future builder to add a .claude/agents/*.md watchdog agent, which would trip validate-docs Checks 1, 2, and 7 simultaneously (all HARD FAIL) across README.md:14, setup.sh (two literals), and .claude/agents/lead-orchestrator.md.", + "mitigation": "Delete or explicitly supersede :79. If an agent is ever added, update all four count sites in the same commit." + }, + { + "ref": "src/zo/wrapper.py:801", + "hazard": "The current rate-limit path is a retry-loop with exponential backoff (_backoff_wait at :801, time.sleep at :806, retries++ at :807, giving up into RATE_LIMITED at :794-799) \u2014 precisely the 'retry-loop' behaviour the Phase 3 brief says to replace with 'wait-and-resume'. Oracle check 12 ('a rate-limit pause auto-resumes on reset') cannot pass against this code, and worse, the retry-loop consumes the nudge budget's cousin: it will keep poking a rate-limited session, violating the never-block taxonomy that check 11 tests.", + "mitigation": "Rewrite :793-808 to parse the reset time, pause without retrying, and resume on reset. Keep tests/unit/test_wrapper.py:504-541 (TestWaitForCompletion.test_detects_rate_limit_and_backs_off, test_rate_limit_exhausts_retries) working or consciously replace them \u2014 they currently assert the retry semantics and will lock in the wrong behaviour if left untouched." + }, + { + "ref": "src/zo/wrapper.py:709", + "hazard": "_wait_tmux (the default zo build path) has no rate-limit detection at all \u2014 verified: the only _detect_rate_limit call site is :793 inside _wait_headless. If the watchdog checker is added to _wait_tmux without the taxonomy, a rate-limited tmux session presents as 'no heartbeat progress' and will be nudged \u2014 the exact failure oracle check 11 forbids. Tests that exercise only the headless path would pass while the default operator path misbehaves.", + "mitigation": "Add classification to BOTH loops and write the check-11 never-nudge test against the tmux path specifically, using the mock.patch.object(LifecycleWrapper, ...) side_effect idiom at test_wrapper.py:594-601." + }, + { + "ref": "pyproject.toml:34", + "hazard": "psutil is NOT a dependency (verified: grep found no psutil in pyproject.toml, uv.lock, or src/zo/). Deps are pydantic>=2.0, pyyaml, click, rich, nbformat. The plan requires 'PID + process-start-time identity so recycled PIDs are never acted on', which is the natural psutil use case. Adding it means a new runtime dep on a C-extension package, plus regenerating the 303KB uv.lock or CI fails at 'uv sync --extra dev' (.github/workflows/ci.yml:41) before any test runs.", + "mitigation": "Prefer stdlib: derive start-time from /proc/<pid>/stat on Linux and `ps -o lstart=` on macOS, both already reachable via the existing subprocess usage. If psutil is genuinely needed, add it to pyproject.toml dependencies AND regenerate uv.lock in the same commit, and confirm it resolves on both 3.11 and 3.12." + }, + { + "ref": "README.md:13", + "hazard": "The test-count cascade is already three-way inconsistent BEFORE Phase 3: README badge says 854 (README.md:13), README Status prose says 780 (README.md:529), and the actual grep count is 917 (`grep -r \"def test_\" tests/ | wc -l`). validate-docs Check 6 (scripts/validate-docs.sh:203) tolerates only \u00b15, so it is currently emitting a warning with diff 63. A Phase 3 builder who trusts the badge will write '854 + N' and be wrong by ~63.", + "mitigation": "Recompute from the filesystem (`grep -r \"def test_\" tests/ | wc -l`) rather than incrementing the badge. Update README.md:13 AND README.md:529 AND the three prose copies at docs/reference/v2-rearchitecture.mdx:99, plans/zo-v2-rearchitecture.md:67, plans/zo-v2-rearchitecture.md:132." + }, + { + "ref": "memory/zo-platform/PRIORS.md:1152", + "hazard": "PRIORS entries record test counts as '735 \u2192 738 + 7 skipped' / '738 \u2192 743 + 7 skipped'. The '7 skipped' is a real, stable fact: it is the single @needs_fastembed-decorated class TestSemanticEmbeddings at tests/unit/test_semantic.py:451, which contains exactly 7 test methods, gated by the skipif at tests/unit/test_semantic.py:37 (`not (_has_fastembed and _has_numpy)`). IMPORTANT: the 7 skipped are NOT e2e tests. tests/e2e/ contains zero Python files \u2014 only tests/e2e/mnist-project/source-docs/project-brief.md. Any Phase 3 assumption that e2e tests exist and are skipped by default is false.", + "mitigation": "When writing the Phase 3 PRIORS/session entries, keep the '+ 7 skipped' suffix and do not attribute it to e2e. If Phase 3 needs an end-to-end harness for the induced-stall test, it must be built as a new tests/integration/ file \u2014 there is no e2e pytest scaffolding to extend." + }, + { + "ref": "tests/unit/test_experiment_loop.py:102", + "hazard": "All 16 evaluate_loop_state call sites in the test file pass arguments positionally as (reg, \"phase_4\", policy). Inserting a ledger parameter anywhere before `policy` in the signature at src/zo/experiment_loop.py:205-209 silently changes what `policy` binds to at every one of those 16 sites plus src/zo/orchestrator.py:1276 \u2014 a type error at best, a silently-wrong policy at worst.", + "mitigation": "Make the new parameter keyword-only with a None default, appended after `policy`. Add one test asserting the ledger-absent path still returns the pre-Phase-3 verdict, so the 16 legacy call sites are proven unaffected." + }, + { + "ref": "src/zo/hookkit.py:47", + "hazard": "Heartbeat files written under memory_root are NOT covered by _SEALED_DEFAULTS (which lists only gate_mode, gate_nonce, gate_decision, contracts.json, plan-ledger.json, sealed_paths). An agent can therefore Write its own heartbeat file and forge liveness, defeating the watchdog \u2014 the same anti-Goodhart hole that oracle checks 7 and 9 exist to close for the ledger.", + "mitigation": "Add the heartbeat directory name to _SEALED_DEFAULTS at src/zo/hookkit.py:47-50 and add a seeded-forgery test in the style of tests/unit/test_ledger.py:143 (test_builder_direct_write_denied_by_sealed_paths). Sealing is safe: the wrapper writes heartbeats in-process, and the PreToolUse sealed-paths guard only intercepts agent Write/Edit tool calls." + }, + { + "ref": "src/zo/contracts.py:167", + "hazard": "The fail-open read convention (return None on any error) is correct for enforcement hooks but INVERTS for a watchdog: if load_heartbeat returns None for both 'file missing because the agent never started' and 'file corrupt', and the checker treats None as stalled, it will nudge healthy sessions; if it treats None as healthy, it will never detect a\u771f stall on first read.", + "mitigation": "Make the heartbeat loader return a three-state result (fresh / stale / unknown) rather than Optional. Route 'unknown' into the never-block taxonomy (do not nudge) and require at least one successful prior read before any staleness verdict, so a missing file cannot be read as a stall." + }, + { + "ref": "docs/reference/v2-rearchitecture.mdx:63", + "hazard": "None of the five workstream tables in this file has a Status column (headers are uniformly `| # | Feature | From | Priority |`). There is no per-feature status text to update for features #2 and #6, so a Phase 3 builder told to 'update the status' will either invent a column in one table (breaking consistency with the other four at :44, :63, :78, :92) or silently skip the doc entirely.", + "mitigation": "Decide once: either add a Status column to ALL five tables in the same edit, or leave the tables alone and reflect progress only in '## How it ships' prose (:87-103) plus docs/roadmap.mdx:30. Record the choice in DECISION_LOG so Phases 4-6 follow it." + }, + { + "ref": "docs/roadmap.mdx:30", + "hazard": "roadmap.mdx:30 describes the watchdog in the future tense as part of a v2 pillar ('with a watchdog built from battle-tested parts (heartbeats, never-fight-these-stops taxonomy, rate-limit auto-resume)'). It is a second, independent copy of the watchdog claim outside v2-rearchitecture.mdx and is easy to miss in the cascade.", + "mitigation": "Include docs/roadmap.mdx:30 in the Phase 3 docs-cascade checklist alongside the mdx and README." + }, + { + "ref": "tests/unit/test_wrapper.py:504", + "hazard": "TestWaitForCompletion currently contains two tests that assert the retry-loop rate-limit semantics (test_detects_rate_limit_and_backs_off at :504 asserts mock_sleep.called; test_rate_limit_exhausts_retries at :525 asserts RATE_LIMITED after _max_retries). Under wait-and-resume these assertions encode the behaviour Phase 3 is removing, so a green suite would be evidence of the OLD design surviving.", + "mitigation": "Explicitly rewrite both tests in the Phase 3 PR rather than leaving them passing. Per plans/zo-v2-rearchitecture.md:74, a merge requires a seeded-failure test \u2014 the check-12 test must fail against the current retry-loop code before the fix lands." + }, + { + "ref": "plans/zo-v2-rearchitecture.md:60", + "hazard": "Oracle check 13 ('Phase 4 on the fresh-context loop completes demo-cifar10 with results >= v1 baseline (91.62%) and total cost <= 1.15x v1 baseline') is the gate for the fresh-context half of Phase 3 (:114-115: 'Check 13 is the go/no-go'), but it requires a GPU demo run and per the brief runs later on a Linux box. Phases 11-12 can be verified in-repo; 13 cannot.", + "mitigation": "Land the fresh-context loop behind an explicit off-by-default flag and do not update any doc to claim check 13 status until the Linux demo runs. Record in STATE.md that Phase 3's gate is partially satisfied (11-12 green, 13 pending) rather than marking Phase 3 complete." + }, + { + "ref": "tests/conftest.py:41", + "hazard": "tmp_project_dir creates only logs/comms and memory/ \u2014 it does NOT create a .zo/ tree, an experiments registry, or a plan-ledger.json. Phase 3 tests that need ledger + lineage + heartbeat state together will be tempted to widen this shared fixture, which is used across the whole suite and would perturb unrelated tests.", + "mitigation": "Build Phase 3 state with module-local helpers (the tests/unit/test_experiment_loop.py:62-92 _exp/_registry pattern) instead of extending conftest.py." + } + ], + "reusable": [ + { + "ref": "tests/unit/test_contracts.py:1-4", + "what": "The canonical seeded-failure docstring, and the answer to (a): \"Seeded-failure pattern per plans/zo-v2-rearchitecture.md: every enforcement mechanism must catch a deliberately planted violation (oracle check 1).\" Class TestSeeded... at :93 is labelled 'Seeded-violation checks \u2014 the heart of oracle check 1.' Copy this module-docstring shape for tests/unit/test_watchdog.py, naming oracle checks 11-12." + }, + { + "ref": "tests/unit/test_hookkit.py:1-8", + "what": "The two-test-per-mechanism rule, stated explicitly: 'Each handler gets: (a) a seeded-violation test proving the mechanism catches a planted problem (plan oracle checks 1-4, 7), and (b) a fail-open test proving infrastructure problems never block a session.' Phase 3's analogue: (a) induced stall IS escalated, (b) rate-limited/context-limited/auth-error/user-abort sessions are NOT nudged." + }, + { + "ref": "tests/unit/test_hookkit.py:61", + "what": "The section-comment convention that ties tests to oracle checks: '# ---- subagent-stop (oracle check 1) ----------------------------------------', repeated at :108 (check 2), :186 (check 3), :245 (check 4), :274 (check 7). Phase 3 must use '# ---- stall detection (oracle check 11) ----' and '# ---- rate-limit resume (oracle check 12) ----' so the check-to-test mapping stays greppable." + }, + { + "ref": "tests/unit/test_ledger.py:1-6", + "what": "The most recent (Phase 2) module docstring, closest precedent for Phase 3: 'Tests for zo.ledger \u2014 the WS-B control plane (plan oracle checks 8-9).' followed by 'Seeded-failure pattern: builders' direct ledger writes are denied by the...'. Phase 3's should read 'Tests for zo.watchdog \u2014 the WS-C execution substrate (plan oracle checks 11-12).'" + }, + { + "ref": "tests/unit/test_ledger.py:137-138", + "what": "The oracle-check class-docstring convention: `class TestOracleOwnership:` with docstring '\"\"\"Plan oracle check 9: builder flip blocked, oracle flip lands.\"\"\"' \u2014 a one-line statement of both the positive and negative halves. Mirror as 'Plan oracle check 11: induced stall escalated, rate-limited session not nudged.'" + }, + { + "ref": "tests/unit/test_plan.py:685,741,751", + "what": "The seeded-failure pattern for a non-hook mechanism (the closest analogue to a pure-Python watchdog checker): a section comment '# Stories & sizing lint (v2 WS-B3, plan oracle check 10)' at :685, a class docstring 'Plan oracle check 10: non-verifiable stories are rejected.' at :741, and a test docstring 'The seeded violation: criteria with no threshold/path/command.' at :751. This is the exact three-level labelling to copy for src/zo/watchdog.py tests." + }, + { + "ref": "tests/unit/test_gate_nonce.py:1-3,75", + "what": "Module docstring naming the workstream, feature id, and check ('Tests for nonce-verified gate approvals (v2 WS-A5, plan oracle check 5).' + 'Seeded forgery: an approval WITHOUT the minted nonce must be rejected;') plus the test docstring 'The seeded forgery of oracle check 5.' at :75. Use 'v2 WS-C1' / 'v2 WS-C2' as the Phase 3 feature ids." + }, + { + "ref": "tests/unit/test_hookkit.py:31-36", + "what": "_run(event, payload, monkeypatch, capsys) \u2014 the module-local driver helper that patches sys.stdin with io.StringIO(json.dumps(payload)), asserts the return code, and parses stdout. Pattern to copy: one small module-local driver that exercises the real public entry point, so every test is a one-liner against it." + }, + { + "ref": "tests/unit/test_hookkit.py:39-58", + "what": "_emit_demo_contracts(mem) \u2014 a module-local builder that constructs a full WorkflowDecomposition (PhaseDefinition + AgentContract) and calls the real emit_contracts. Precedent for building real on-disk state via the production writer rather than hand-writing JSON, which keeps tests honest about the schema." + }, + { + "ref": "tests/integration/test_hooks_shim.py:41-66", + "what": "The contracts_env pytest fixture returning a dict of env overrides {ZO_CONTRACTS_PATH, ZO_DELIVERY_ROOT, ZO_REPO_ROOT} built from tmp_path. Reuse this shape for a watchdog integration fixture supplying ZO_MEMORY_ROOT + a seeded heartbeat directory." + }, + { + "ref": "tests/integration/test_hooks_shim.py:113-146", + "what": "class TestSettingsWiring \u2014 the 'nothing ships unwired' guard. Two tests read the real .claude/settings.json and assert the mechanism is actually referenced by a runtime caller. This is the single most important pattern for Phase 3 to replicate (as a test that the watchdog checker is invoked from the wrapper poll loop), because plans/zo-v2-rearchitecture.md:74 makes a runtime caller a merge requirement." + }, + { + "ref": "src/zo/contracts.py:151-161", + "what": "Atomic JSON write: mkdir(exist_ok=True) \u2192 tempfile.mkstemp(dir=..., suffix='.tmp') \u2192 os.fdopen write \u2192 os.replace \u2192 finally unlink. Use verbatim for heartbeat writes; it is the repo's only durable-write idiom and its docstring (:180-183) already states the torn-read rationale." + }, + { + "ref": "src/zo/contracts.py:164-169", + "what": "load_contracts: `try: return Model.model_validate_json(path.read_text(...)) except (OSError, ValueError): return None`. The pydantic-v2 parse-and-fail-open reader. Reuse the model_validate_json form (pydantic>=2.0 per pyproject.toml:35) for heartbeat parsing \u2014 but see the hazard about three-state results." + }, + { + "ref": "src/zo/experiment_loop.py:183-202", + "what": "LoopDecision(BaseModel) \u2014 the verdict-object convention: a pydantic model carrying verdict + human-readable `reason` (documented at :190 as 'Human-readable justification logged to DECISION_LOG') + evidence fields + evaluated_at timestamp, with model_config = {'use_enum_values': True} at :202. Model the watchdog's StallVerdict on this exactly, including the reason-for-DECISION_LOG field." + }, + { + "ref": "src/zo/experiment_loop.py:227-283", + "what": "The priority-ordered verdict cascade with numbered comments ('# Priority 1: target tier hit.' :245, '# Priority 2: budget exhausted.' :262, '# Priority 3: plateau detection.' :274), each returning a LoopDecision with an f-string reason citing the concrete numbers. Reuse this shape for the never-block taxonomy, which is inherently a priority cascade evaluated BEFORE any intervention." + }, + { + "ref": "src/zo/orchestrator.py:1279-1288", + "what": "The verdict\u2192DECISION_LOG bridge: self._memory.append_decision(DecisionEntry(title=f\"Loop verdict for {phase.phase_id}: {decision.verdict}\", context=..., decision=..., rationale=decision.reason, outcome=...)). Reuse verbatim for watchdog escalations so stalls land in the audit trail the same way loop verdicts do." + }, + { + "ref": "src/zo/orchestrator.py:1294-1302", + "what": "self._record_learning(title=..., root_cause=..., rule_gap=...) invoked for the failure-ish verdicts (DEAD_END, PLATEAU). Reuse for watchdog escalation-to-restart so repeated stalls become durable priors \u2014 this is the existing hook into the PRIORS pipeline." + }, + { + "ref": "src/zo/wrapper.py:790-792", + "what": "self._comms.log_checkpoint(agent=\"wrapper\", phase=\"lifecycle\", subtask=\"completion\", progress=...) and log_error(agent=\"wrapper\", error_type=\"rate_limit\", severity=\"blocking\", description=...) at :796-799. The wrapper's existing comms vocabulary \u2014 reuse agent=\"wrapper\", phase=\"lifecycle\", with new subtask values ('stall-detected', 'nudge', 'rate-limit-pause', 'rate-limit-resume') instead of adding an EventType." + }, + { + "ref": "src/zo/wrapper.py:894-901", + "what": "@staticmethod _detect_rate_limit(output: str) -> bool at :894 and _backoff_wait(self, attempt) at :898. Static, pure, individually testable classifiers \u2014 tests/unit/test_wrapper.py:729-744 exercises _detect_rate_limit via @pytest.mark.parametrize over five pattern strings without any wrapper instance. Build the never-block taxonomy as sibling static classifiers so each stop reason (context-limit / rate-limit / auth / user-abort) gets a cheap parametrized test." + }, + { + "ref": "tests/unit/test_wrapper.py:729-744", + "what": "The parametrize idiom for a pure classifier: @pytest.mark.parametrize(\"text\", [...5 strings...]) + a separate test_returns_false_for_normal_output. Exactly how each never-block taxonomy category should be tested \u2014 one parametrized positive list per category plus a negative." + }, + { + "ref": "tests/unit/test_memory.py (TestPhaseStatusValidation, described at memory/zo-platform/PRIORS.md:1092-1099)", + "what": "The drift-guard test convention mandated by PR-036 Rule 1: when a parser mirrors an enum's values in a module-local frozenset, ship a test asserting `{s.value for s in Enum} == _MODULE_ALLOWLIST`. Required if the session-restore cutover introduces any ledger-side status allowlist." + }, + { + "ref": "tests/unit/test_orchestrator.py TestGetCurrentPhase (five tests named at memory/zo-platform/PRIORS.md:1150)", + "what": "The resume-precedence lock: test_returns_active_phase_on_resume, test_gated_takes_priority_over_active, test_active_takes_priority_over_pending, test_blocked_phase_not_returned, test_real_resume_via_state_md_round_trip. These five must be ported (not deleted) when the restore path moves from STATE.md to the ledger \u2014 PR-037 Rule 3 (PRIORS.md:1148) makes GATED > ACTIVE > PENDING a contract, and PR-037 Rule 2 (:1143) requires one test per non-terminal status." + }, + { + "ref": "memory/zo-platform/sessions/session-040-2026-08-12.md:1-3,92", + "what": "Session file template: H1 '# Session NNN \u2014 YYYY-MM-DD \u2014 <Title>', then '**Type:** <kind>' on line 3, then '## What happened', topic sections, and a mandatory closing '## Next session \u2014 pick up here' (:92). 109 lines is representative length. Phase 3 writes session-041-<date>.md in this shape." + }, + { + "ref": "memory/zo-platform/DECISION_LOG.md:7-14", + "what": "DECISION_LOG entry template: '## Decision: <ISO8601 Z>' / '**Type:** ARCHITECTURE|SCOPE|...' / '**Title:**' / '**Decision:**' / '**Rationale:**' / '**Alternatives considered:**' (numbered inline) / '**Outcome:**', entries separated by '---'. Append-only per the header note at :3." + }, + { + "ref": "memory/zo-platform/PRIORS.md:1057-1101", + "what": "PRIORS entry template (PR-NNN): H2 '## PR-NNN: <Rule as a sentence>' / '**Source:** Session NNN (date), <context>' / '**Root cause category:** missing_rule|...' / '**Failure:** <narrative with code block>' / '### Rules' (numbered; each with a **Why:** and a **How to apply:**) / '### Verified Solution' (files touched + tests added, named individually) / closing 'Test count N \u2192 M + 7 skipped. ruff <paths> clean. validate-docs X passed / Y failed / Z warnings.' Only add a PRIORS entry if Phase 3 hits a real failure." + } + ], + "open_questions": [ + "docs/reference/v2-rearchitecture.mdx has no Status column in any of its five workstream tables (headers uniformly `| # | Feature | From | Priority |` at :63 and siblings). The task brief asked 'what status text they carry so we can update' \u2014 the answer is none. Does Phase 3 (a) add a Status column to all five tables, (b) update only the '## How it ships' prose at :87-103, or (c) leave the mdx alone? This choice binds Phases 4-6 too and should be recorded in DECISION_LOG.", + "Should the watchdog get a CLI surface (`zo watchdog`)? Nothing in plans/zo-v2-rearchitecture.md:107-115 requires one \u2014 the checker runs inside the LifecycleWrapper poll loop. If added: tests/unit/test_cli.py:38 tolerates it (subset assertion) and validate-docs Check 3 does not count CLI subcommands (only .claude/commands/*.md), so it is cheap; but docs/COMMANDS.md would need a '### zo watchdog' section under '## CLI Commands' (:13) by convention. Confirm whether a CLI surface is in scope.", + "Where exactly do heartbeat JSON files live? The brief says 'under the project memory root'. hookkit._memory_root (src/zo/hookkit.py:64-70) resolves ZO_MEMORY_ROOT, else repo_root/memory/zo-platform; delivery projects use delivery_repo/.zo/memory (src/zo/cli.py:69). Confirm one subdirectory name (e.g. <memory_root>/heartbeats/<agent>.json) and whether it goes into _SEALED_DEFAULTS (src/zo/hookkit.py:47) to prevent agent self-forgery.", + "How is process start time obtained without psutil (not a dependency \u2014 verified absent from pyproject.toml:34-40 and uv.lock)? /proc/<pid>/stat on Linux vs `ps -o lstart=` on macOS diverge, and CI is ubuntu-latest (.github/workflows/ci.yml:24) while development is darwin. Confirm the approach and whether the macOS path needs to be test-mocked rather than exercised.", + "Oracle check 11 says the stall must be 'detected and escalated within one poll cycle', but the default poll_interval is 10.0s (src/zo/wrapper.py:551) while the stall threshold is minutes. Is 'one poll cycle' measured in poll iterations (satisfiable with mocked time) or wall-clock seconds? This determines whether the check-11 test can run in the unit suite or needs a longer-running integration harness.", + "Does the fresh-context loop's 'git commit as checkpoint' run against the delivery repo or the ZO repo? No existing git-commit helper was found in src/zo/ during this sweep. If Phase 3 introduces one, confirm whether it must respect the sealed-paths guard and how it behaves when the delivery repo has no git remote or a dirty tree.", + "For the session-restore cutover: does the ledger fully subsume STATE.md's phase_states, or does STATE.md remain the write-side projection (plans/zo-v2-rearchitecture.md:103-105 says STATE.md becomes 'a projection for humans, never the parse target for control decisions')? If both are written, which wins on disagreement during resume \u2014 and does the GATED > ACTIVE > PENDING precedence (PRIORS.md:1148) get evaluated against the ledger, STATE.md, or a merge?", + "specs/watchdog.md \u00a73.5 (:60-70) defines a `watchdog:` block in project_config with tick_cron, stall_ticks_hard/soft, and remediation: [nudge, respawn, reroute]. Under the poll-loop design, tick_cron is meaningless and 'reroute' (route the task to a different agent role) is a far larger scope than the plan's 'nudge \u2192 iteration restart'. Confirm which config keys survive, and whether 'reroute' is deferred out of Phase 3." + ], + "notes": "SCOPE OF VERIFICATION. Every line number below was read directly from source (Read or grep -n), not inferred. Counts were computed live: 917 `def test_` occurrences across tests/; 21 files in .claude/agents/; 24 files in .claude/commands/; 16 `evaluate_loop_state(` call sites in tests/unit/test_experiment_loop.py.\n\nDIRECT ANSWERS TO THE FOUR NUMBERED QUESTIONS.\n\n(a) SEEDED-FAILURE TEST PATTERN FOR ORACLE CHECKS 1-10 \u2014 three-level labelling, consistent across all five Phase 1-2 test modules:\n 1. Module docstring names module + workstream + check numbers + the seeded-failure contract. Best templates: tests/unit/test_ledger.py:1-6 (Phase 2, most recent), tests/unit/test_contracts.py:1-4, tests/unit/test_hookkit.py:1-8, tests/unit/test_gate_nonce.py:1-3.\n 2. Section comment or class docstring ties a group to one check: `# ---- sealed-paths (oracle check 7) ----` (tests/unit/test_hookkit.py:274, also :61 :108 :186 :245); `class TestOracleOwnership:` + `\"\"\"Plan oracle check 9: builder flip blocked, oracle flip lands.\"\"\"` (tests/unit/test_ledger.py:137-138); `# Stories & sizing lint (v2 WS-B3, plan oracle check 10)` (tests/unit/test_plan.py:685).\n 3. Test name and docstring carry the word \"seeded\": test_seeded_missing_deliverable_blocks (test_hookkit.py:65), test_seeded_claim_plus_stub_blocks (:139), test_seeded_write_to_sealed_control_file_denied (:278), test_seeded_violation_emits_block_json (tests/integration/test_hooks_shim.py:70), test_builder_direct_write_denied_by_sealed_paths (test_ledger.py:143 \u2014 docstring \"A builder agent's Write to plan-ledger.json is denied (seeded)\"), test_plan.py:751 (\"The seeded violation: criteria with no threshold/path/command.\").\n PAIRING RULE, stated verbatim at tests/unit/test_hookkit.py:3-5: every mechanism gets (a) a seeded-violation test AND (b) a fail-open/negative test. For checks 11-12 the negative half IS the never-block taxonomy \u2014 the \"rate-limited session is NOT nudged\" assertion is not optional colour, it is the required second test.\n Recommended Phase 3 files: tests/unit/test_watchdog.py (module docstring: \"Tests for zo.watchdog \u2014 the WS-C execution substrate (plan oracle checks 11-12).\"), plus a tests/integration/ file for the induced-stall harness and the wiring guard.\n\n(b) DOCS CASCADE FOR PHASE 3, in dependency order:\n MUST (spec-reality):\n specs/watchdog.md:3 \u2014 flip \"**Status:** design spec (RFC)\"; specs/watchdog.md:39-40 (\u00a73.1 cron tick \u2192 poll loop), :53-58 (\u00a73.4 remediation), :60-70 (\u00a73.5 config), :74-79 (\u00a74 integration points \u2192 wrapper.py not orchestrator.py; delete :79 monitor-agent line), :83-88 (\u00a75 acceptance \u2192 checks 11-12), :92-93 (\u00a76 \"this PR is spec-only\" is now false).\n specs/workflow.md:547 \u2014 Subtask 4.3 \"Iteration Protocol\" gains the fresh-context-per-iteration semantics.\n specs/memory.md:286 (recovery step 4, \"New session reads STATE.md and picks up from last_completed_subtask\") and :291 (the \"Recovery mechanism (implemented, v2 WS-A3)\" hook list) \u2014 session-restore cutover.\n MUST (counts, validate-docs Check 6 is warn-only but already tripping):\n README.md:13 (tests-854 badge) and README.md:529 (\"780 platform tests\") \u2014 both wrong today; real count is 917.\n docs/reference/v2-rearchitecture.mdx:99, plans/zo-v2-rearchitecture.md:67, plans/zo-v2-rearchitecture.md:132 \u2014 three prose copies of \"854\".\n SHOULD:\n docs/reference/v2-rearchitecture.mdx:65 (feature #2) and :66 (feature #6) \u2014 see open question, there is no Status column.\n docs/roadmap.mdx:30 \u2014 second independent copy of the watchdog claim, future tense.\n docs/COMMANDS.md:13 \u2014 add \"### zo watchdog\" only if a CLI surface ships.\n NOT TRIGGERED (verified): specs/comms.md:48 event-type list (only if a sixth EventType is added \u2014 recommend not); specs/agents.md (only if an agent file is added); setup.sh and .claude/agents/lead-orchestrator.md agent counts (no new agent).\n\n(c) VALIDATE-DOCS RULES THAT WILL BITE. Eight checks total in scripts/validate-docs.sh; six are HARD FAIL (1,2,3,4,7,8), two are warn-only (5,6). The job is a bare script invocation (.github/workflows/validate-docs.yml:16-18) exiting 1 iff FAIL_COUNT>0 (validate-docs.sh:273-277).\n WILL BITE: Check 6 (:192-208) is ALREADY warning \u2014 README badge 854 vs grep count 917, diff 63, tolerance \u00b15 (:203). Warn-only, so it cannot fail CI, but Phase 3 should clear it.\n WILL BITE ONLY IF: Check 3 (:110-129, HARD FAIL) fires if a .claude/commands/*.md slash command is added \u2014 requires README.md:307 AND memory/zo-platform/STATE.md:65 updated in the same commit. Checks 1/2/7 (:46-103, :215-222, HARD FAIL) fire if a .claude/agents/*.md is added \u2014 four literals across README.md:14, setup.sh (AGENT_COUNT and \"All N agent\"), and .claude/agents/lead-orchestrator.md. Check 4 (:136-148, HARD FAIL) fires on any version bump \u2014 three files.\n WILL NOT BITE: adding a `zo watchdog` CLI subcommand (Check 3 counts .claude/commands/ files only, and tests/unit/test_cli.py:38 uses a subset assertion `expected <= actual` with an explicit comment at :34-37 permitting extra commands). Adding or editing a spec file \u2014 no check reads specs/ except Check 5's agent-tier warn (:173).\n Check 8 (:231-261) is inert locally (no scripts/.client-blocklist, warns at :244) but may be armed elsewhere \u2014 keep client identifiers out of Phase 3 fixtures.\n\n(d) MEMORY PROTOCOL FILES + LAST SESSION FORMAT.\n memory/zo-platform/STATE.md \u2014 prepend a new top entry under \"## Current Position\" (:9) in the style of :11 (\"**Session 040 (current) \u2014 pick up here.** ...\"), one dense bolded paragraph naming branch, PR number, shipped artifacts, and any verification caveat; demote 040. Front-matter at :3-7 (project/mode/phase/iteration/status) may need phase/status updated. Do NOT disturb :65 (\"24 commands across\") unless a slash command is added \u2014 validate-docs Check 3 reads it.\n memory/zo-platform/DECISION_LOG.md \u2014 append-only (header note at :3); file is 1279 lines, append at EOF using the :7-14 template. At minimum: the watchdog RFC-vs-plan divergence and the evaluate_loop_state signature choice.\n memory/zo-platform/PRIORS.md \u2014 only on a real failure; template at :1057-1101 (PR-036). Note the closing-line convention \"Test count N \u2192 M + 7 skipped. ruff <paths> clean. validate-docs X passed / Y failed / Z warnings.\" Read PR-036 (:1057) and PR-037 (:1103) before touching the restore path \u2014 they are binding constraints, not background.\n memory/zo-platform/sessions/session-041-<YYYY-MM-DD>.md \u2014 new file. Last session file is session-040-2026-08-12.md (109 lines); numbering has gaps (009,010,011,013,014,016,017,022,023,024,027,034,035,038,039,040) so 041 is next. Format: H1 \"# Session 040 \u2014 2026-08-12 \u2014 Repo Deep-Dive Research (pre-rearchitecture)\" (:1), \"**Type:** Research session (no platform code changes, no commits)\" (:3), \"## What happened\" (:5), topic H2s (:22, :37, :60, :78), and a mandatory closing \"## Next session \u2014 pick up here\" (:92). Note 040 appended multiple work phases as \"(same session, part 2)\" / \"(parts 3-4)\" H2s rather than creating new files \u2014 a valid pattern if Phase 3 lands across one long session.\n\nTEST INFRASTRUCTURE FACTS worth stating plainly because they contradict common assumptions:\n - tests/e2e/ contains ZERO Python files \u2014 only tests/e2e/mnist-project/source-docs/project-brief.md. There is no e2e pytest suite to extend or skip.\n - The \"7 skipped\" referenced throughout PRIORS is the single class TestSemanticEmbeddings (tests/unit/test_semantic.py:451) with 7 methods, gated by the one and only skipif in the repo (tests/unit/test_semantic.py:37, `not (_has_fastembed and _has_numpy)`). It is unrelated to e2e.\n - pyproject.toml [tool.pytest.ini_options] (:80-82) defines ONLY testpaths=[\"tests\"] and pythonpath=[\"src\"] \u2014 no addopts, no custom markers, no strict-markers. Any new pytest marker would be unregistered (warning, not error).\n - tests/integration/ has 18 files covering: hook shim end-to-end (test_hooks_shim.py), auto-iteration (test_auto_iteration.py), experiment flow and CLI (test_experiment_flow.py, test_experiments_cli.py), full pipeline (test_full_pipeline.py), init lifecycle (test_init_lifecycle.py), low-token mode (test_low_token.py), migration (test_migrate.py), phase snapshots (test_phase_snapshots.py), plan+comms (test_plan_comms_integration.py), preflight (test_preflight.py), report CLI (test_report_cli.py), surrogate consolidate/edge (test_surrogate_consolidate.py, test_surrogate_edge.py), .zo layout (test_zo_dir_layout.py), dynamic/adapted agents (test_dynamic_agents_e2e.py, test_agent_adaptations_e2e.py). test_auto_iteration.py is the closest existing analogue for a fresh-context-loop integration test.\n - ruff (pyproject.toml:72-78): target py311, line-length 100, select E,F,I,N,W,UP,B,SIM,TCH \u2014 enforced in CI on src/ only (.github/workflows/ci.yml:44), so test files may use noqa sparingly as at tests/unit/test_wrapper.py:275." + }, + { + "surface": "Phase 3 (WS-C execution substrate) recon over the three MIT reference repos, read at source (not digests), cross-mapped to the ZO v2 branch's actual Phase-1/2 code. Covers: (a) OMC per-worker heartbeat file schema/write-trigger/freshness/readers; (b) OMC idle-nudge budget + ready-AND-no-active-task predicate + thinking-only-streak transcript detector; (c) OMC never-block stop taxonomy (context-limit #213, rate-limit #777, auth #1308, user abort, /cancel, scheduled wakeup, oversize-tool-result) with exact pattern tables; (d) OMC rate-limit-wait auto-resume daemon (OAuth usage-API reset times, edge-triggered resume, tmux send-keys); (e) cross-platform PID+process-start-time identity \u2014 found in OMC (src/team/team-owner-epoch.ts), NOT in ruflo; ruflo contributes signal-0 liveness + 24h TTL orphan reaping + supervisor stale-heartbeat election; (f) ralph.sh 113-line fresh-context loop (spawn cmd, prd.json+progress.txt feed, sentinel completion, git-commit checkpoint, iteration cap); (g) OMC autopilot phase state \u2014 materialized typed JSON + Stop-hook continuation, explicitly NOT fresh-spawn-per-phase. Also mapped the ZO landing sites: LifecycleWrapper poll loops (src/zo/wrapper.py), ledger API (src/zo/ledger.py), contracts/hookkit control-file conventions, evaluate_loop_state + orchestrator call site, and the PR-036/PR-037 GATED>ACTIVE>PENDING resume precedence that the STATE.md->ledger cutover must preserve.", + "integration_points": [ + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/types.ts:103", + "what": "(a) HeartbeatData schema \u2014 the whole contract is 8 fields: {workerName, teamName, provider, pid, lastPollAt (ISO string), currentTaskId?, consecutiveErrors, status: 'ready'|'polling'|'executing'|'shutdown'|'quarantined'}. Note pid is present but there is NO process-start-time field, and no schema_version.", + "action": "ADAPT. Port the shape into a ZO HeartbeatFile pydantic model, but ADD three fields OMC lacks: schema_version:int, process_started_at:str (from the identity fn at team-owner-epoch.ts:69), and a monotonic progress marker (e.g. last_ledger_mutation / artifact digest) so 'fresh mtime but zero progress' is detectable. Keep status enum + consecutiveErrors verbatim \u2014 they map onto ZO AgentStatus." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/heartbeat.ts:19", + "what": "(a) Heartbeat file path: getOmcRoot(cwd)/state/team-bridge/{sanitize(team)}/{sanitize(worker)}.heartbeat.json \u2014 one file per agent, sanitized names, atomicWriteJson at :34. readHeartbeat (:38) and listHeartbeats (:54) swallow all parse errors and return null / skip malformed.", + "action": "PORT-VERBATIM (as Python). Layout maps 1:1 to ZO's memory_root convention already used by contracts.json (src/zo/contracts.py:48) and plan-ledger.json (src/zo/ledger.py:52): put heartbeats at memory_root/heartbeats/{agent}.json. Reuse ledger.py:81 _atomic_write rather than writing a new atomic writer." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/heartbeat.ts:81", + "what": "(a) isWorkerAlive(cwd, team, worker, maxAgeMs) \u2014 freshness ONLY: (Date.now() - Date.parse(lastPollAt)) < maxAgeMs; missing file = dead, NaN date = dead. It never checks the pid it stores.", + "action": "ADAPT \u2014 do NOT port as-is. ZO's oracle check #11 requires liveness = freshness AND process identity. Combine this with isProcessIdentityDead (team-owner-epoch.ts:138) into a single ZO predicate: stale = (age > threshold) AND (process positively dead OR no progress delta)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/worker-health.ts:47", + "what": "(a) Freshness thresholds, all hardcoded defaults, and inconsistent across call sites: worker-health.ts:47 and :125 use heartbeatMaxAgeMs = 30000; team-status.ts:98 uses 30000; unified-team.ts:68 hardcodes 60000. Write cadence is BridgeConfig.pollIntervalMs, documented default 3000 (src/team/types.ts:21).", + "action": "ADAPT. Take the ~10x (write cadence : staleness threshold) ratio as the design rule, not the absolute numbers \u2014 ZO's watchdog spec already picks stall_threshold_min: 20 (specs/watchdog.md:65). Put the threshold in project_config, single source, never per-call-site defaults." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/mcp-team-bridge.ts:700", + "what": "(a) Write trigger: heartbeat is written at the TOP of every poll cycle (buildHeartbeat at :186 stamps lastPollAt = now, pid = process.pid), plus on startup (:621, wrapped in try/except so startup IO failure cannot block loop entry), on quarantine (:691), on first-ready transition (:710), and once at task claim with status 'executing' (:746). There is NO heartbeat write while the spawned CLI child runs.", + "action": "ADAPT the write sites; FIX the gap. In ZO the heartbeat must be refreshed by the agent-side hook plane (hookkit PostToolUse / SubagentStop already fire per tool call \u2014 src/zo/hookkit.py:392 main dispatch) so 'agent is working hard on one long tool call' still refreshes. Copy the protected-startup-write pattern (:621) verbatim." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/idle-nudge.ts:29", + "what": "(b) Nudge budget constants: DEFAULT_NUDGE_CONFIG = {delayMs: 30_000 (idle dwell before first nudge), maxCount: 3 (max nudges per pane per wait call), message: 'Continue working on your assigned task and report concrete progress (not ACK-only).'}; scanIntervalMs = 5_000 throttle (:68, enforced at :91); budget check `if (state.nudgeCount >= maxCount) continue` at :107; dwell check at :123; on successful send the idle timer RESETS (firstIdleAt = null, :131) so each nudge costs a full fresh dwell.", + "action": "PORT-VERBATIM (constants + state machine). PaneNudgeState {nudgeCount, firstIdleAt, lastNudgeAt} (:58) is exactly ZO's per-agent nudge record. Two required deltas: (1) ZO's budget must be run-scoped and persisted (OMC's is in-memory per wait call, so a restart silently refills the budget); (2) ZO escalates to iteration restart at budget exhaustion, OMC just stops nudging (no escalation exists \u2014 that is ZO-new code)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/idle-nudge.ts:48", + "what": "(b) The idle predicate: isPaneIdle = paneLooksReady(captured) && !paneHasActiveTask(captured). Never nudges the leader pane (:98).", + "action": "ADAPT the LOGIC, SKIP the implementation. The AND-of-two-predicates shape is the transferable idea (ready-to-accept-input AND not-executing); the implementation is tmux-screen-scraping, which ZO should not inherit. ZO's equivalent: heartbeat.status in {ready, polling} AND no ledger attempt recorded since last tick." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/tmux-session.ts:1414", + "what": "(b) paneHasActiveTask(captured): tail-40-lines regexes \u2014 /\\b\\d+\\s+background terminal running\\b/i, /esc to interrupt/i, /\\bbackground terminal running\\b/i, plus a spinner-line regex /^[\u00b7\u273b]\\s+[A-Za-z]...(?:\u2026|\\.{3})$/u. paneLooksReady (:1432) requires a prompt glyph line /^\\s*(?:[\u2502\u2503\u2551\u258c\u2590\u258f\u2595\u254e\u2506\u250a]\\s*)?[\u203a>\u276f]\\s*/u (:1429), returns false while bootstrapping.", + "action": "SKIP for ZO's watchdog core (brittle TUI scraping, breaks on any Claude Code UI change). Keep only as an optional last-resort signal for the tmux-mode wrapper path (src/zo/wrapper.py:713 already calls _tmux_pane_alive/_tmux_claude_running)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/hooks/persistent-mode/index.ts:1479", + "what": "(b) Thinking-only streak detection: THINKING_ONLY_STREAK_MAX = 3, TTL 5 min (:1481); classifyLastAssistantTurn (:1513) walks the bounded transcript tail BACKWARD to the previous real user message, returns 'tool_use' on any tool_use block (:1546) or any user record carrying a tool_result (:1561 \u2014 proof the turn invoked a tool), 'thinking_only' if only thinking/redacted_thinking seen, else 'indeterminate'. Guard (:1584) fails OPEN: no transcript, unreadable, or indeterminate => keep the original decision; tool_use resets the counter to 0 (:1607).", + "action": "PORT-VERBATIM (algorithm + fail-open discipline + the 3/5-min constants). This is the single best non-wall-clock stall signal in all three repos and directly serves oracle check #11: a stalled agent that is 'thinking' has a fresh heartbeat but zero tool progress. ZO already reads transcripts (src/zo/hookkit.py:153 _last_assistant_text) \u2014 extend that reader rather than writing a second one." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/hooks/todo-continuation/index.ts:370", + "what": "(c) NEVER-NUDGE taxonomy, exact tables. isContextLimitStop (#213): patterns ['context_limit','context_window','context_exceeded','context_full','max_context','token_limit','max_tokens','conversation_too_long','input_too_long'] substring-matched against normalized stop-reason fields. isRateLimitStop (#777, :390): ['rate_limit','rate_limited','ratelimit','too_many_requests','429','quota_exceeded','quota_limit','quota_exhausted','request_limit','api_limit','overloaded','capacity']. AUTHENTICATION_ERROR_PATTERNS (#1308, :442, exactly 16): ['authentication_error','authentication_failed','auth_error','unauthorized','unauthorised','401','403','forbidden','invalid_token','token_invalid','token_expired','expired_token','oauth_expired','oauth_token_expired','invalid_grant','insufficient_scope']. isUserAbort (:295): user_requested||userRequested flag, EXACT-match ['aborted','abort','cancel'], SUBSTRING ['user_cancel','user_interrupt','ctrl_c','manual_stop'] \u2014 bare 'interrupt' deliberately excluded (issue #2478: it also means 'new user message arrived mid-turn'). Normalization at :141 getStopReasonFields: reads stop_reason|stopReason|end_turn_reason|endTurnReason|reason, lowercases, s/[\\s-]+/_/g.", + "action": "PORT-VERBATIM (the four pattern lists + the exact/substring split + the normalizer). These are literal production scar tissue with issue numbers; do not re-derive. ZO must apply the taxonomy BEFORE any nudge, and must also feed it from stderr/stdout tails since ZO has no Stop-hook stop_reason in headless mode (see the wrapper hazard below)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/hooks/persistent-mode/index.ts:2268", + "what": "(c) Bypass ORDER, which is itself load-bearing: critical-context (:2271) -> explicit /cancel (:2282) -> authenticated session cancel (:2292) -> user abort (:2303) -> rate limit (:2316, returns a '[RALPH PAUSED - RATE LIMITED]' message with mode 'none') -> auth error (:2328) -> scheduled wakeup (:2341) -> oversize tool-result redirect with a bounded consecutive window (:2356, counter reset on the else branch at :2373). Every branch returns shouldBlock:false; createHookOutput (:2556) maps shouldBlock -> {continue:false}.", + "action": "PORT-VERBATIM (order + the 'return a human-readable pause message, do not act' convention). Two additions for ZO: isCriticalContextStop (:1003) also triggers on estimated transcript context >= CRITICAL_CONTEXT_STOP_PERCENT even with no stop_reason \u2014 port that belt-and-braces check, since ZO's headless path often has no reason string at all." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/features/rate-limit-wait/rate-limit-monitor.ts:19", + "what": "(d) How the reset time is obtained: NOT parsed from the error message and NOT a fixed backoff \u2014 checkRateLimitStatus calls the OAuth usage API (getUsage from src/hud/usage-api.js) and reads structured fields fiveHourResetsAt / weeklyResetsAt / monthlyResetsAt as Date objects; limited = percent >= RATE_LIMIT_THRESHOLD (100, :12); nextResetAt = earliest reset among the limits actually hit (:53-58); timeUntilResetMs = max(0, nextResetAt - now).", + "action": "ADAPT / partially SKIP. ZO has no OAuth usage-API dependency and should not take one. Recommended ZO design: (1) primary = wait-and-poll on an observable predicate (re-probe until the session stops emitting rate-limit signatures), (2) secondary = parse a resets-at timestamp out of the message when present, (3) fallback = ZO's existing exponential backoff (src/zo/wrapper.py:898). Copy the 100%-threshold + earliest-of-many-windows arithmetic verbatim." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/features/rate-limit-wait/daemon.ts:305", + "what": "(d) Resume trigger is EDGE-triggered and conservative: shouldResumeBlockedPanesOnStatusChange = wasLimited && !isNowLimited && !isRateLimitStatusDegraded(next) \u2014 a degraded/stale usage-API 429 response is explicitly NOT treated as an all-clear (rate-limit-monitor.ts:161,171). Poll loop at :345 with pollIntervalMs default 60_000 (:46) and a 30s Promise.race timeout on the status check (:360) so the poll loop itself cannot stall.", + "action": "PORT-VERBATIM (the edge-trigger predicate, the 'degraded != all-clear' rule, and the timeout-wrapped status probe). This is exactly oracle check #12's 'auto-resume on reset'. The 30s race guard is also directly applicable to ZO's own poll loop, which currently has no per-probe timeout." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/features/rate-limit-wait/daemon.ts:405", + "what": "(d) Resume mechanics + cooldown: per-pane dedup via state.resumedPaneIds (skip at :406, push at :420), blockedPanes cleared after the attempt (:429), resumedPaneIds cleared only when not limited and nothing blocked (:433) \u2014 that list IS the cooldown, there is no timer. The actual resume is sendResumeSequence (tmux-detector.ts:414): validate pane id against /^%\\d+$/, then tmux send-keys '1' Enter \u2014 i.e. it answers the CLI's menu prompt; it does NOT relaunch the process or re-send the original prompt, and it returns true WITHOUT verifying the pane state changed (comment at :432-434).", + "action": "ADAPT the dedup/cooldown ledger; SKIP the send-keys mechanism. ZO's fresh-context loop makes resume trivially cleaner: on reset, spawn the next iteration (which re-derives state from the ledger) instead of poking a live TUI. Do add the verification OMC omits \u2014 after resume, require a heartbeat/ledger delta within N seconds or mark the resume failed." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/features/rate-limit-wait/tmux-detector.ts:34", + "what": "(d) Rate-limit text detection (message-based, complementary to the API): RATE_LIMIT_PATTERNS at :34 (/rate limit/i, /usage limit/i, /quota exceeded/i, /too many requests/i, /please wait/i, /try again later/i, /limit reached/i, /hit your limit/i, /hit .+ limit/i, /resets? .+ at/i, /5[- ]?hour/i, and a TIGHTENED weekly pattern /\\bweekly\\s+(?:usage\\s+)?(?:limit|quota|cap|allowance|allocation)\\b/i). False-positive suppression is the valuable part: stripGitOutputLines (:153) removes commit/diff lines before matching (:138 GIT_OUTPUT_LINE_PATTERNS) because commit messages like 'fix weekly report' were tripping it; SAVED_TRANSCRIPT_COMMAND_PATTERN/LABEL (:70,:74) reject `cat`ed transcripts of old rate-limit screens; scanForBlockedPanes uses a cursor so only NEW output since the last scan is analyzed (:376-389).", + "action": "PORT-VERBATIM the pattern list AND the anti-false-positive layer. ZO's current detector (src/zo/wrapper.py:51) is 4 naive regexes over a log tail with no cursor and no git-output stripping \u2014 it will fire on a `git log` in the agent's own stdout. The cursor-tracked 'only new lines' rule is the highest-value single fix." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/team-owner-epoch.ts:69", + "what": "(e) Cross-platform process-start identity \u2014 this lives in OMC, NOT ruflo. processStartIdentityForPlatform(pid): linux => read /proc/{pid}/stat, slice after the last ')', take field[19] (starttime ticks) => 'linux:{ticks}'; win32 => powershell (Get-Process -Id N).StartTime.ToUniversalTime().Ticks => 'win32:{ticks}'; darwin => sysctl -b kern.proc.pid.N parsed as struct kinfo_proc leading timeval p_starttime (darwinProcessStartFromKinfo :59, sanity-bounds seconds>946684800 and <=now+86400, micros<1e6) => 'darwin:{sec}:{usec}', with a portable fallback `ps -o lstart= -p N` under LC_ALL=C => 'darwin:{sec}:0'. isValidProcessStartIdentity (:112) is a per-platform regex allowlist capped at 1024 chars.", + "action": "PORT-VERBATIM (as Python). This is the only complete cross-platform implementation in the three repos and it directly satisfies Phase 3's 'PID + process-start-time identity so recycled PIDs are never acted on'. In Python: linux /proc field 22, darwin `ps -o lstart=` (psutil.Process(pid).create_time() is a cleaner one-liner if a dependency is acceptable). Keep the string-tagged format ('{platform}:{value}') and the validator." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/team-owner-epoch.ts:138", + "what": "(e) The death predicate, and its key safety rule: isProcessIdentityDead returns true ONLY on positive proof \u2014 process.kill(pid,0) throwing ESRCH, or a recorded start-identity that provably differs from the observed one. Unknown/malformed observed identity is explicitly 'never positive proof of death' (:146-147). processStartIdentitiesMayMatch (:128) tolerates the darwin coarse-vs-fine mismatch by treating usec=='0' on either side as a wildcard.", + "action": "PORT-VERBATIM including both comments. 'Positive-death-only' is the rule that prevents ZO from killing/restarting a healthy agent on a transient probe failure, and the darwin wildcard is a real cross-source-of-truth bug they already paid for." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ruflo/v3/@claude-flow/cli/src/mcp-tools/swarm-tools.ts:80", + "what": "(e) ruflo's liveness + TTL orphan reaping (issue #1799). isPidAlive (:85): process.kill(pid,0) -> true; on throw, `return code === 'EPERM'` \u2014 EPERM means alive-but-other-user, only ESRCH means dead. reconcileOrphanSwarms (:107): for status==='running', if pid recorded and !isPidAlive => terminate with reason `host process N exited`; if NO pid (legacy records) fall back to ORPHAN_TTL_MS = 24h against updatedAt (:106,:119-122). Called on EVERY store load (loadSwarmStore :148) and persisted only when something changed (:149), so readers never see ghost 'running' entries.", + "action": "PORT-VERBATIM the EPERM-is-alive rule and the reconcile-on-every-load pattern; ADAPT the TTL (24h is a legacy-record backstop, far too slow for a 20-min stall budget \u2014 ZO wants the TTL as a schema-migration fallback only). Note ruflo has NO process-start-time check anywhere (verified: no btime/lstart/starttime usage in v3/@claude-flow/cli/src) \u2014 take identity from OMC, take reaping hygiene from ruflo." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ruflo/v3/@claude-flow/cli/src/services/repo-supervisor.ts:20", + "what": "(e) Stale-owner takeover election, the closest analog to 'external checker owns the watch': SUPERVISOR_STALE_MS = 3 * 60 * 1000 (:39, explicitly chosen as 'longer than any plausible heartbeat interval so a slow tick never triggers false takeover, short enough that a crashed supervisor yields within minutes'); takeover only when no record OR dead pid OR heartbeat older than stale-ms; a live supervisor with a fresh heartbeat is NEVER overwritten (:22-23). Lock is O_CREAT|O_EXCL with a 10s stale-lock break on mtime (:104-119); registry files are lstat-checked to reject symlinks (:68).", + "action": "ADAPT. ZO's watchdog runs inside the LifecycleWrapper poll loop (single owner), so full election is out of scope \u2014 but port the stale-threshold RATIONALE comment as the way to pick ZO's numbers, and port the symlink-refusal + O_EXCL stale-lock-break if the heartbeat dir is ever written by more than one process." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ralph/ralph.sh:84", + "what": "(f) The whole fresh-context loop, 113 lines. Spawn: line 95 `OUTPUT=$(claude --dangerously-skip-permissions --print < \"$SCRIPT_DIR/CLAUDE.md\" 2>&1 | tee /dev/stderr) || true` (amp variant at :92). Prompt is piped on STDIN, NOT passed as an argv prompt; `--print` = single non-interactive response; no --continue/--resume anywhere (statelessness is deliberate); `|| true` so an agent crash cannot kill the loop under `set -e` (:5); `tee /dev/stderr` gives live output while capturing. Cap: `for i in $(seq 1 $MAX_ITERATIONS)` (:84), MAX_ITERATIONS default 10 (:9), `sleep 2` between iterations (:107), exit 1 at cap (:113). Completion: `grep -q \"<promise>COMPLETE</promise>\"` on captured stdout (:99) => exit 0.", + "action": "ADAPT. Port the loop skeleton and the stdin-prompt spawn form into src/zo/experiment_loop.py (which today has NO spawn surface at all \u2014 verified: no subprocess/spawn in the file). Use ruflo's hardened spawn instead of shell: `spawn('claude', ['--print','--output-format','json'])` + `child.stdin.end(prompt)` (headless-worker-executor.ts:1397-1409) \u2014 ruflo documents (:1378-1381) that passing the prompt as an argument caused shell-tokenization corruption, and (:1382-1388, :1413) that `claude --print` spawns grandchildren that survive a plain kill, so it uses detached:true + process.kill(-pid, sig). SKIP `--dangerously-skip-permissions` \u2014 ZO has permissions_overlay.py and a sealed-paths hook; bypassing them would void the Phase-1 enforcement plane. SKIP the sentinel grep (see hazards)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ralph/CLAUDE.md:7", + "what": "(f) What the fresh session is fed \u2014 the entire re-derivation contract, 10 numbered steps: read prd.json; read progress.txt 'Codebase Patterns section first'; verify/checkout the branch from PRD branchName; pick highest-priority story with passes:false; implement ONLY that story; run quality checks; update nearby CLAUDE.md files with reusable patterns; commit ALL changes as `feat: [Story ID] - [Story Title]` ONLY if checks pass (:14, plus 'Do NOT commit broken code' :76); flip passes:true; append to progress.txt. Stop condition at :90-97; 'Work on ONE story per iteration / Commit frequently / Keep CI green' at :101-104.", + "action": "PORT-VERBATIM the CONTRACT SHAPE into ZO's per-iteration builder prompt, with ZO substitutions: prd.json -> plan-ledger.json (src/zo/ledger.py:52), progress.txt Codebase-Patterns -> the priors digest, story -> next ledger entry with passes:false, commit-only-if-green -> commit gated on the oracle. CRITICAL DELTA: in ralph the builder flips its own passes flag; ZO's ledger docstring already forbids that (src/zo/ledger.py:5-7 \u2014 only oracle-verified paths may flip, builders denied by the sealed-paths hook). Keep that inversion." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ralph/ralph.sh:42", + "what": "(f) Run-lifecycle hygiene: archive-on-branch-change. Reads branchName from prd.json via jq, compares to a .last-branch dotfile, and on change copies prd.json+progress.txt to archive/YYYY-MM-DD-<branch-minus-ralph/-prefix>/ and rewrites progress.txt with a fresh header (:49-64), then re-stamps .last-branch (:68-73).", + "action": "ADAPT (low priority). ZO already scopes memory per project; the transferable bit is 'reset the per-run scratch memory when the run identity changes so feature-A learnings do not pollute feature-B iterations' \u2014 applies to a per-phase priors digest." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/hooks/autopilot/types.ts:20", + "what": "(g) OMC's phase-state pattern: AutopilotPhase union ('expansion'|'planning'|'execution'|'ralplan'|'ralph'|'qa'|'validation'|'complete'|'failed') materialized in typed session-scoped JSON (AutopilotState carries phase, current_phase, phase_durations, total_agents_spawned), with per-phase caps in DEFAULT_CONFIG (:285): maxIterations 10, maxExpansionIterations 2, maxArchitectIterations 5, maxQaCycles 5, maxValidationRounds 3, parallelExecutors 5, validationArchitects ['functional','security','quality'].", + "action": "ADAPT the typed-state idea (ZO already has it in plan-ledger.json phase_status, src/zo/ledger.py:73). SKIP as a fresh-spawn reference: there is NO fresh-spawn-per-phase in OMC. Autopilot advances by blocking the Stop event in the SAME session and re-injecting a continuation prompt (createHookOutput -> {continue:false}, persistent-mode/index.ts:2556-2564); the only `claude` spawns in OMC are cross-provider team workers (src/team/model-contract.ts:190). Ralph (the bash repo) is the sole fresh-spawn precedent." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/lib/security-config.ts:44", + "what": "(g) Global backstop over every per-mode cap: hardMaxIterations 500 default (:44), 200 under OMC_SECURITY=strict (:54), read via getHardMaxIterations (:151); file overrides can only LOWER it (Math.min at :108).", + "action": "PORT-VERBATIM the layering rule (per-mode cap < global hard cap; overrides may only tighten). ZO's LoopPolicy.max_iterations (src/zo/experiment_loop.py:261) is currently the only bound \u2014 add an un-raisable global ceiling for the fresh-context loop." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/wrapper.py:709", + "what": "ZO LANDING SITE #1 \u2014 _wait_tmux poll loop. Per-cycle work is _check_gate_mode_change() + _maybe_open_training_pane() + tmux liveness, with _STARTUP_GRACE_POLLS=2 / _DEAD_CONFIR M_POLLS=2 / _DEAD_RECHECK_INTERVAL=2.0 (:74-81) and default poll_interval=10.0 (:551). This is exactly the 'external checker in the LifecycleWrapper poll loop' the plan calls for (plans/zo-v2-rearchitecture.md:108).", + "action": "Add a single `self._watchdog_tick()` call next to _check_gate_mode_change() at :710 (and the headless twin at :777). Reuse the existing startup-grace + consecutive-confirm debounce pattern for stall confirmation rather than inventing a second one \u2014 it is already the right shape and already justified in the docstring (:684-700)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/wrapper.py:792", + "what": "ZO LANDING SITE #2 \u2014 the ONLY existing rate-limit handling: _wait_headless reads a log tail, _detect_rate_limit (:894) matches 4 naive regexes (_RATE_LIMIT_PATTERNS :51: /429/, /rate.?limit/, /overloaded/, /too many requests/, all IGNORECASE) over the whole tail, then does a RETRY-LOOP with exponential backoff (_backoff_wait :898, base 30s * 2^attempt + jitter, max_retries default 3 :89) and gives up with AgentStatus.RATE_LIMITED. The tmux path (_wait_tmux) has NO rate-limit handling at all.", + "action": "REPLACE with wait-and-resume. This is precisely the #777 retry-loop antipattern OMC removed. Concretely: (1) add the git/transcript false-positive stripping and the new-lines-only cursor from tmux-detector.ts:153/:376; (2) on detection, enter a paused state instead of retrying; (3) resume on the edge-triggered all-clear predicate (daemon.ts:305); (4) extend coverage to the tmux path. Oracle check #12 tests exactly this function." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/ledger.py:125", + "what": "ZO LANDING SITE #3 \u2014 ledger read/write API the fresh-context loop must drive: emit_ledger (:125, regeneration merges preserve passes/attempts/last_failure/phase_status by subtask_id), load_ledger (:93, fail-open None), mark_phase_passed (:184), reset_phase (:203), record_phase_failure (:216), record_attempt (:227), set_phase_status (:239), summarize (:248). LedgerEntry (:54) already carries passes/attempts/last_failure \u2014 the exact fields ralph's prd.json lacks.", + "action": "Read-only dependency for the watchdog (progress delta = any ledger mutation since last tick). For the fresh-context loop: call record_attempt at iteration start and let ONLY the oracle path call mark_phase_passed, preserving the docstring's contract (:5-7). No new mutators needed \u2014 the API is complete." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/experiment_loop.py:205", + "what": "ZO LANDING SITE #4 \u2014 evaluate_loop_state reads ONLY the ExperimentRegistry (registry.experiments filtered by phase/status, :230-233), never the ledger. Verdict priority: TARGET_HIT (:245) > BUDGET_EXHAUSTED (:261) > PLATEAU (:274). The module has no spawn surface whatsoever (verified: no subprocess/spawn import) \u2014 the fresh-context spawner is entirely new code.", + "action": "Deferral #1 lands here: add a ledger read (load_ledger from src/zo/ledger.py:93) so the verdict can also account for ledger phase_status/passes, and keep the fail-open contract (load_ledger returns None on any problem \u2014 a ledger IO error must not change the verdict)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/orchestrator.py:1276", + "what": "ZO LANDING SITE #5 \u2014 the sole evaluate_loop_state call site: lazy-imports LoopVerdict/evaluate_loop_state/resolve_policy (:1265), loads the registry, resolves policy with low_token + max_iterations_override, logs EVERY verdict to DECISION_LOG (:1280), and records a learning on DEAD_END/PLATEAU (:1292).", + "action": "This is where a CONTINUE verdict must fan out into 'spawn a fresh builder session for the next iteration'. Preserve the log-every-verdict behaviour; add the git-commit checkpoint here so each iteration boundary is a recoverable point." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/orchestrator.py:687", + "what": "ZO LANDING SITE #6 \u2014 get_current_phase, the PR-036/PR-037 resume precedence that the STATE.md->ledger cutover must preserve verbatim: (1) GATED :711, (2) ACTIVE :715, (3) PENDING with all depends_on in COMPLETED :722-726; BLOCKED deliberately not returned (:704). Rationale and the five locking tests are recorded in memory/zo-platform/PRIORS.md PR-037 rule 3 ('Priority order between status values is part of the contract').", + "action": "Deferral #2 lands here: swap the source of phase status from STATE.md-parsed phase_states to LedgerFile.phase_status (src/zo/ledger.py:73) WITHOUT touching the three-branch order. Keep the parse-time validation invariant from PR-036 \u2014 port _VALID_PHASE_STATUSES-style closed-enum validation to the ledger loader, and add the same drift-guard test that asserts the allowlist equals PhaseStatus values." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/hookkit.py:64", + "what": "ZO LANDING SITE #7 \u2014 _memory_root(repo_root): ZO_MEMORY_ROOT env override, else repo_root/memory/zo-platform, else None; _repo_root (:60) honors ZO_REPO_ROOT. Hook dispatch main at :392; per-invocation JSONL observability trace at :81 (logs/hook-trace-{date}.jsonl, disable with ZO_HOOK_TRACE=0); deny output envelope at :373 (hookSpecificOutput.permissionDecision='deny').", + "action": "Use this exact resolver for the heartbeat directory so hooks (writers) and the wrapper (reader) agree on the root \u2014 do not add a second root-resolution path. The heartbeat write is a natural addition to the existing hook handlers." + } + ], + "hazards": [ + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/mcp-team-bridge.ts:746", + "hazard": "Heartbeat staleness during legitimate long work. OMC writes 'executing' once at task claim (:746) and then blocks in spawnCliProcess; there is NO heartbeat write while the child runs (verified: writeHeartbeat appears only at :622,:691,:701,:710,:746). With heartbeatMaxAgeMs = 30_000 (worker-health.ts:47), any task longer than 30s reads as DEAD. A naive ZO port would nudge/restart every agent doing a real 10-minute training step \u2014 the exact false positive oracle check #11 forbids.", + "mitigation": "Refresh the heartbeat from the agent side on every tool boundary (ZO already has PostToolUse/SubagentStop handlers in src/zo/hookkit.py:392), and require BOTH stale-heartbeat AND positive process death (or a zero progress-delta over N ticks) before any intervention \u2014 matching specs/watchdog.md:48's hard/soft stall split." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/heartbeat.ts:81", + "hazard": "PID recorded but never checked. HeartbeatData carries pid (types.ts:107) yet isWorkerAlive is pure mtime freshness; nothing correlates the file to a live process, and there is no process_started_at at all. A recycled PID or a stale file from a crashed run is indistinguishable from a live agent \u2014 the precise scenario Phase 3 says must never be acted on.", + "mitigation": "Do not port heartbeat.ts's liveness in isolation. Join it with team-owner-epoch.ts:138 isProcessIdentityDead and store process_started_at in the heartbeat record itself; treat 'unknown identity' as NOT dead (their :146 comment)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/heartbeat.ts:81", + "hazard": "Name collision that will silently mis-port: there are TWO different exported isWorkerAlive functions \u2014 heartbeat.ts:81 (cwd, team, worker, maxAgeMs -> file freshness) and tmux-session.ts:1729 (paneId -> tmux pane liveness), the latter imported by runtime.ts:10 and used at runtime.ts:491/:567. Reading the wrong one gives a completely wrong model of how OMC decides liveness.", + "mitigation": "When citing OMC liveness in the Phase 3 design doc, always qualify the module. In ZO use one unambiguous name (e.g. agent_is_stalled) with a single implementation." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/wrapper.py:794", + "hazard": "ZO's current rate-limit path is the #777 antipattern OMC had to remove: on detection it sleeps and RETRIES the same session up to _max_retries (default 3), then marks RATE_LIMITED. Worse, _detect_rate_limit (:894) scans the whole log tail with /rate.?limit/ and /429/ \u2014 the agent's own stdout containing a git log, a test name, or a quoted error will trip it, and the tmux path has no rate-limit handling at all so a rate-limited tmux session just looks idle to any new watchdog.", + "mitigation": "Convert to pause/wait-and-resume; add cursor-tracked 'new output only' scanning plus git-output stripping (tmux-detector.ts:153,:376); gate the watchdog's nudge path on the never-block taxonomy BEFORE any nudge so a rate-limited tmux session is classified, not poked." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/hooks/todo-continuation/index.ts:101", + "hazard": "The taxonomy's inputs are not guaranteed. The module header states 'the abort detection patterns below are ASSUMED' and :287-290 notes that per Anthropic docs the Stop hook does not even run on user interrupt \u2014 so isUserAbort may never fire in practice. All predicates read stop_reason/end_turn_reason fields that ZO's headless subprocess path does not receive at all (src/zo/wrapper.py:779 only has a return code and a log tail).", + "mitigation": "Feed ZO's classifier from the evidence it actually has: stdout/stderr tail + transcript tail (the pattern OMC itself falls back to in getOversizeStopEvidence, :226-248, and in isCriticalContextStop's transcript-percent estimate, :1008). Do not assume a structured stop reason exists; default to 'do not nudge' when classification is ambiguous." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ralph/ralph.sh:99", + "hazard": "Sentinel completion by grep is unsound in both directions: the literal <promise>COMPLETE</promise> appears verbatim in the prompt file that is piped to the model (ralph/CLAUDE.md:95), so any model that quotes its own stop condition terminates the loop early; and there is no per-story failure counter, so one impossible item silently burns every remaining iteration.", + "mitigation": "ZO must derive completion from the ledger predicate (all entries passes:true, oracle-flipped) rather than from model prose, and add a per-subtask attempt cap on top of the global iteration cap \u2014 LedgerEntry.attempts (src/zo/ledger.py:62) and record_attempt (:227) already exist for exactly this." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ralph/ralph.sh:95", + "hazard": "No subprocess timeout anywhere in the ralph loop: a hung `claude --print` blocks the for-loop forever, and the only watchdog is a human reading the tee'd output. Porting the loop shape without a timeout re-imports the stall the watchdog exists to kill. Additionally `--dangerously-skip-permissions` (:95) disables the permission plane wholesale.", + "mitigation": "Wrap the per-iteration spawn with a hard timeout + process-group kill (ruflo headless-worker-executor.ts:1413-1431 killTree with SIGTERM then SIGKILL after 5s, detached:true so grandchildren die too), and run ZO iterations under permissions_overlay.py rather than a blanket bypass." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/features/rate-limit-wait/tmux-detector.ts:432", + "hazard": "OMC's resume is fire-and-forget: sendResumeSequence returns true after `tmux send-keys 1 Enter` with an explicit in-code admission that it does not verify the pane state changed, and the pane is then removed from blockedPanes (daemon.ts:429) and added to resumedPaneIds (:420) \u2014 so a failed resume is recorded as success and never retried. Oracle check #12 ('a rate-limit pause auto-resumes on reset') would pass a port of this while the session stays dead.", + "mitigation": "Make ZO's resume verifiable: after resuming, require a heartbeat refresh or ledger delta within a bounded window; only then mark resumed. Assert that condition in the check-#12 harness, not merely that a resume was attempted." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/orchestrator.py:711", + "hazard": "The STATE.md->ledger cutover can silently regress PR-037. The GATED>ACTIVE>PENDING order is enforced only by tests (per PRIORS.md PR-037 'Verified Solution': test_gated_takes_priority_over_active, test_active_takes_priority_over_pending, test_real_resume_via_state_md_round_trip) \u2014 the last of which round-trips through STATE.md and will need rewriting against the ledger, which is exactly when the invariant can be lost. LedgerFile.phase_status is a plain dict[str,str] (src/zo/ledger.py:73) with no enum validation, so PR-036's parse-time guarantee does not currently exist on the ledger side.", + "mitigation": "Before the cutover: add closed-enum validation + a PhaseStatus drift-guard test to the ledger loader (mirroring _memory_formats.py's _VALID_PHASE_STATUSES), and keep both round-trip tests (STATE.md and ledger) green during the transition rather than replacing one with the other." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/idle-nudge.ts:66", + "hazard": "OMC's nudge budget lives in an in-memory Map scoped to a single wait call (`private readonly states = new Map()`), so it resets on every restart \u2014 a crash-looping supervisor can nudge without bound. ZO's LifecycleWrapper is long-lived but restartable, so an in-memory port inherits the same hole and can defeat the 'bounded nudge budget' requirement.", + "mitigation": "Persist the nudge record per agent per run (alongside the heartbeat under memory_root) and reload it on wrapper start, so the budget survives wrapper restarts; escalate to iteration restart on exhaustion rather than falling silent." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/features/rate-limit-wait/rate-limit-monitor.ts:21", + "hazard": "The reset-time source is an OAuth usage API (getUsage from src/hud/usage-api.js), not the error message. If ZO copies the design without the API, there is no reset timestamp at all and check #12's 'auto-resumes on reset' has nothing to key on; the daemon degrades to 'poll until not limited', which is fine in production but needs a deterministic clock to be testable.", + "mitigation": "Design the ZO resume around an injectable clock + an injectable 'is still limited' probe so the controlled test for check #12 can drive the reset deterministically; treat any parsed resets-at timestamp as an optimization, never the sole mechanism." + } + ], + "reusable": [ + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/ledger.py:81", + "what": "_atomic_write (mkstemp in the target dir + os.replace, with finally-unlink) \u2014 already the ZO atomic-write idiom, matches OMC's atomicWriteJson semantics. Use it for heartbeat writes; do not add a second writer." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/contracts.py:116", + "what": "emit_contracts: the established 'machine-readable control file under memory_root, gitignored, atomic temp+replace' precedent (docstring at :10-11 explicitly cites the gate_mode control-file convention). Heartbeats and any watchdog state file should follow this exact pattern rather than inventing a new location." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/hookkit.py:81", + "what": "_trace: per-invocation JSONL observability line to logs/hook-trace-{date}.jsonl, fail-open under contextlib.suppress(OSError), disabled by ZO_HOOK_TRACE=0. Reuse this exact logger shape for watchdog tick decisions (classification, nudge/no-nudge, reason) so check #11 can assert on a log line." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/wrapper.py:74", + "what": "_STARTUP_GRACE_POLLS / _DEAD_CONFIRM_POLLS / _DEAD_RECHECK_INTERVAL plus the consecutive-negative debounce at wrapper.py:718-747 \u2014 a working, already-justified 'do not act on a single negative reading' state machine. Reuse it for stall confirmation instead of writing a parallel one." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/ledger.py:227", + "what": "record_attempt / record_phase_failure / LedgerEntry.attempts + last_failure \u2014 the per-subtask failure counter ralph lacks. Gives the fresh-context loop per-item escalation for free; also the natural progress-delta signal for the watchdog." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/hookkit.py:153", + "what": "_last_assistant_text(transcript_path) \u2014 an existing ZO transcript reader. Extend this into the thinking-only-streak classifier (port of persistent-mode/index.ts:1513) rather than adding a second transcript parser." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/experiment_loop.py:183", + "what": "LoopDecision/LoopVerdict + resolve_policy's precedence (low-token clamps < plan spec < CLI override, :165-180) \u2014 the existing, tested policy-resolution idiom. The watchdog's config (specs/watchdog.md:60-70) should resolve the same way instead of inventing new precedence rules." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/zero-operators/src/zo/orchestrator.py:1280", + "what": "The append-DecisionEntry-on-every-verdict pattern at the loop call site \u2014 reuse for watchdog escalations so 'every stall, reroute, and dropped item is logged' (specs/watchdog.md:58) is satisfied by existing machinery." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/oh-my-claudecode/src/team/team-owner-epoch.ts:29", + "what": "canonicalize + sha256 digest + payload_hash self-verification on every record read (parseRecord :45-57 recomputes the digest and rejects on mismatch). If ZO's heartbeat is ever written by an agent that must not forge liveness for a different agent, this is the ready-made tamper-evident record pattern (same idea as ZO's existing gate_nonce)." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ruflo/v3/@claude-flow/cli/src/services/repo-supervisor.ts:98", + "what": "withLock: O_CREAT|O_EXCL lock file with a 10s mtime-based stale-lock break and a 2s acquisition deadline (:100-124), plus assertNotSymlink (:68). A compact, dependency-free multi-writer guard if the heartbeat directory ends up written by more than one process." + }, + { + "ref": "/Users/sam101fe4x/Documents/code/ruflo/v3/@claude-flow/cli/src/services/headless-worker-executor.ts:1397", + "what": "The hardened headless spawn: spawn('claude', ['--print','--output-format','json']) + child.stdin.end(prompt) + detached:true + killTree via process.kill(-pid, sig) + SIGTERM-then-SIGKILL-after-5s timeout, with in-code rationale for each choice (:1378-1396). This is the production-grade version of ralph.sh:95 and the right template for ZO's per-iteration spawn." + } + ], + "open_questions": [ + "Where should ZO's heartbeat live, and who writes it? memory_root (memory/{project}/) is the ledger/contracts/gate_mode convention (src/zo/ledger.py:3, src/zo/contracts.py:10) and is what the plan means by 'project memory root' \u2014 but the wrapper reads memory_root while subagents run in the DELIVERY repo. Confirm ZO_MEMORY_ROOT (src/zo/hookkit.py:65) is exported into every spawned agent's env, or the writer and reader will disagree on the path.", + "What exactly triggers a heartbeat write on the agent side? OMC's writer is a bridge poll loop it owns (mcp-team-bridge.ts:636); ZO does not own the agent's inner loop. The only per-agent write points ZO controls are hook events (src/zo/hookkit.py:392). Is PostToolUse/SubagentStop cadence sufficient, or does ZO need a separate side-process per agent?", + "Does ZO's headless path have any structured stop reason at all? src/zo/wrapper.py:779 only sees a return code and a log tail, so the entire OMC taxonomy would have to run over text evidence. Confirm what claude --print exposes (an --output-format json envelope carries usage/result \u2014 ruflo headless-worker-executor.ts:1397, :596) before committing to a field-based classifier.", + "Which reset-time source does check #12 use? OMC's is an OAuth usage API ZO does not have (rate-limit-monitor.ts:21). Decide now between (a) poll-until-clear with an injectable clock, (b) parse a resets-at timestamp from the message, (c) fixed backoff \u2014 the acceptance test's determinism depends on this choice.", + "Fresh-context spawn granularity for Phase 4: the plan says 'a fresh builder session per iteration' (plans/zo-v2-rearchitecture.md:111), while evaluate_loop_state operates per completed EXPERIMENT (src/zo/experiment_loop.py:230). Confirm iteration == experiment, and whether git commit checkpoints land per experiment or per ledger subtask (ralph commits per story, ralph/CLAUDE.md:14).", + "Does the watchdog run only inside the LifecycleWrapper poll loop, or also as the specs/watchdog.md:40 cron self-invoke? The spec describes a scheduled orchestrator tick (*/17 cron); the plan says an external checker in the wrapper poll loop and explicitly NOT a monitor agent. These are different wake sources with different failure modes \u2014 pick one as primary for Phase 3 and mark the other deferred.", + "Is `--dangerously-skip-permissions` acceptable for ZO's fresh-context iterations? ralph.sh:95 and OMC's worker contract (src/team/model-contract.ts:190) both use it, but ZO's Phase-1 enforcement plane (sealed-paths deny at src/zo/hookkit.py:373, permissions_overlay.py) is the thing that keeps builders from flipping ledger pass flags. Confirm the overlay is honored in --print mode.", + "After the STATE.md->ledger cutover, does STATE.md remain writable by hand? PRIORS.md PR-036 rule 2 calls STATE.md 'a hand-editable contract' and CLAUDE.md instructs humans to edit it on every commit. If it becomes a pure projection (plans/zo-v2-rearchitecture.md:103-104), hand edits become silently ineffective \u2014 needs an explicit operator-facing decision and probably a warning on divergence." + ], + "notes": "LICENSES \u2014 all three reference repos are MIT, so verbatim porting is permitted with attribution: oh-my-claudecode/LICENSE:1-3 (MIT, Copyright (c) 2025 Yeachan Heo), ruflo/LICENSE:1-3 (MIT, Copyright (c) 2024-2026 ruvnet), ralph/LICENSE:1-3 (MIT, Copyright (c) 2026 snarktank). Practically, everything here is TypeScript/bash and ZO is Python, so 'port-verbatim' means the constants, regex tables and control flow, not the code text. Where a table IS copied literally (the four never-block pattern lists at todo-continuation/index.ts:370/:390/:442/:295, and the tmux rate-limit patterns at tmux-detector.ts:34), attach a source comment with repo + path:line + MIT notice \u2014 that also preserves the issue-number provenance (#213/#777/#1308/#2478) which is the reason those lists are trustworthy.\\n\\nCORRECTIONS TO THE PRIOR RESEARCH DIGESTS (read the source, not the summary):\\n1. The digest attributes 'PID + process-start-time identity' to ruflo (ruflo.md:145). It is not there \u2014 ruflo uses signal-0 liveness only (swarm-tools.ts:85, proxy/lifecycle.ts:64, repo-supervisor.ts:58; no btime/lstart/starttime anywhere under v3/@claude-flow/cli/src). The real cross-platform start-time identity is OMC's src/team/team-owner-epoch.ts:69-148. ruflo contributes the EPERM-is-alive rule, the 24h TTL fallback, and reconcile-on-every-load.\\n2. OMC does NOT fresh-spawn per phase (item g). Autopilot/ralph continue in the SAME session by blocking Stop and re-injecting a prompt (persistent-mode/index.ts:2556-2564). The only fresh-spawn precedents are ralph.sh:95 (claude --print, prompt on stdin) and ruflo's headless worker (headless-worker-executor.ts:1397). Phase 3's fresh-context loop is closer to ralph+ruflo than to anything in OMC.\\n3. OMC's heartbeat is freshness-only and does not use the pid it stores (heartbeat.ts:81 vs types.ts:107) \u2014 the digest's 'heartbeat-file liveness + PID-aware ownership' (oh-my-claudecode.md:54) conflates two unconnected subsystems.\\n\\nRECOMMENDED PHASE 3 BUILD ORDER, by evidence density: (1) heartbeat record = OMC schema + OMC start-identity + a progress marker; (2) stall predicate = freshness AND positive-death/no-delta, with ZO's existing debounce; (3) never-block taxonomy ported verbatim and applied BEFORE any nudge (this alone satisfies half of check #11); (4) bounded persisted nudge budget (idle-nudge constants) escalating to iteration restart; (5) rate-limit wait-and-resume replacing wrapper.py's retry loop, with the edge-triggered all-clear and a verified resume (check #12); (6) fresh-context loop = ralph's contract shape + ruflo's hardened spawn + ZO's oracle-owned ledger flags.\\n\\nAll line numbers in this report were verified by direct Read/grep -n against the working trees on 2026-08-17; no file was modified.\"" + } +] \ No newline at end of file diff --git a/memory/zo-platform/sessions/session-041-2026-08-17.md b/memory/zo-platform/sessions/session-041-2026-08-17.md new file mode 100644 index 0000000..d00695e --- /dev/null +++ b/memory/zo-platform/sessions/session-041-2026-08-17.md @@ -0,0 +1,99 @@ +# Session 041 — 2026-08-17 — v2 Phase 3 / WS-C part 1: the watchdog (PR-A) + +**Type:** Build session (recon → contract → concurrent build → adversarial verify → PR) + +## What happened + +1. **Caught up + Phase 2 gate closed.** Read the session-040 handoff, STATE, + plan, last four DECISION_LOG entries, `specs/watchdog.md`, PR-036/037/046. + PR #108 (WS-B control plane) was open, `CLEAN`, CI green, main unmoved → + squash-merged (`1faf53a`). Cut `claude/v2-phase3-substrate` off it. + Baseline here: 929 passed / 7 skipped (Python 3.14 locally; CI 3.11/3.12). + Repo-local git identity set to the SamT noreply identity already on main. + +2. **Recon swarm (7 read-only mappers + synthesis, ~1.1M tokens).** Output: + `memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md` + (every claim `file:line`; hazards resolved between mappers; test skeleton; + docs cascade; 10 open questions) + `raw-mappers.json`. Also published as a + private artifact for Sam. + +3. **Structural finding (verified by hand before the swarm confirmed it):** + `Orchestrator.advance_phase()` / `mark_subtask_complete()` have zero + runtime callers → automated gate, `_auto_iterate_if_needed`, WS-B ledger + flips and gate-nonce minting are unreachable in production; `zo build` = + one lead session per phase then `end_session()` (which clobbers any lead + edit of STATE.md `## Phases`). Phases advanced only via hand-edited + STATE.md (PR-036/037). → PRIORS PR-047. Reframes Phase 3: the + fresh-context driver is the missing runtime caller. + +4. **Sam's four decisions** (DECISION_LOG 2026-08-17T09:00): driver evaluates + gates for ALL phases (Phase 4 alone gets fresh headless spawns until check + 13 says extend); two PRs (A watchdog, B driver + loop + deferrals); + ledger-wins-with-warning + `zo phase set` override on the restore cutover; + tmux nudges default ON behind the pane-ready / no-permission-dialog guard. + +5. **PR-A built** per `pr-a-build-contract.md` (contract-first: pinned + `zo.watchdog` API, heartbeat schema, tick algorithm, per-builder file + ownership). First workflow serialized the core module ahead of the other + builders; Sam pushed back on parallelism → restarted with all four + builders concurrent (core-finisher reconciling the already-drafted module) + → integrator → 3 adversarial verifier lenses (19 findings, 2 high: banner + reset times parsed in UTC; static banner could never resume) → fixer (11 + applied with regression tests, 4 rejected with reasons). → PRIORS PR-048. + +6. **Shipped:** `src/zo/watchdog.py` (+ `_watchdog_models.py`, + `_watchdog_text.py`, `_proc.py`), `src/zo/_hook_heartbeat.py` (stdlib-only + writer on a new `PostToolUse *` hook), `src/zo/_wrapper_watchdog.py` + (`WatchdogRunner`), wrapper tick in both loops + rate-limit pause/resume + replacing the headless retry loop, `AgentStatus.PAUSED_RATE_LIMIT/STALLED`, + `ProjectConfig.watchdog`, `--no-watchdog`, `ZO_WATCHDOG=0`, heartbeats + sealed + gitignored (platform root and delivery `.zo/` templates), + `specs/watchdog.md` rewritten to implemented reality (RFC cron tick + superseded), mdx/COMMANDS/build.mdx cascade, README badge 1053. + **929 → 1131 passed / 7 skipped, ruff clean, validate-docs 0 failures.** + Seeded tests for checks 11 + 12 on both loops. + +7. **Live evidence in-session:** the heartbeat hook fired for the lead and for + the workflow subagents (`agent_type=workflow-subagent`, `tick_count` 44 → + `shutdown`) — PostToolUse carries agent identity for subagents. + Correction: `PostToolUseFailure` DOES fire on nonzero-exit Bash (session + 040's caveat withdrawn; no WS-D item needed). + +## Next session — pick up here + +1. Confirm PR-A merged (CI 3.11/3.12 green). If not and green, merge. +2. **PR-B on `claude/v2-phase3-substrate` (or stacked):** `zo.driver. + run_phase_loop` — FIRST runtime caller of `advance_phase` (wiring test + must start at `zo build`, per PR-047); split `_launch_and_monitor` into + `_launch_once` + once-per-run teardown (integration map §2.1); per phase: + `_refresh_gate_mode` → `set_active_phase` → prompt (+ ledger digest + section) → launch → `advance_phase` → COMPLETED/GATED(nonce, stop)/ + ITERATE(git checkpoint via `surrogate.commit_worktree`, relaunch)/ + STALLED(restart under `hard_max_restarts`, `_record_learning`). Phase 4 + iterations = fresh headless `claude --print` (`_launch_headless`, prompt on + stdin, `start_new_session` + process-group kill); other phases keep the + tmux lead. Oracle-side subtask completion from the ledger's own criteria + (`_check_artifacts` + `_finalize_experiments`) — no self-assertion path. + Consume the watchdog outcomes: `STALLED` → restart; `RATE_LIMITED` + + `resume_at` → wait then relaunch. +3. **Deferrals folded into PR-B:** `evaluate_loop_state(..., *, ledger=None)` + (16 test call sites, zero churn); wire `parse_next_md` / + `parse_hypothesis_md` into `_finalize_experiments` (DEAD_END is dead code + today); restore cutover ledger > STATE.md per phase with a loud warning, + PR-036 validation moved to a strict ledger load, PR-037 precedence tests + untouched, `LedgerEntry.completed` + `mark_subtask_completed`, HOLD + write-through at `orchestrator.py:962`, decompose ordering fix + (`_consume_gate_decision` before `_emit_plan_ledger`), `zo phase set` + override logged to DECISION_LOG; fix absolute `Experiment.artifacts_dir` + (check-13 blocker); strengthen `preflight._check_claude_cli` (PR-046). +4. Check 13 (demo-cifar10 ≥ 91.62 % at ≤ 1.15× cost) needs the Linux box with + the claude CLI; land the loop behind an off-by-default flag until green. + Also verify there: sealed-paths under `--print --dangerously-skip-permissions`; + the CPU-evidence idle threshold on a real tmux session. +5. Follow-ups noted, not done: `_watchdog-ticks.jsonl` rotation; + `wrapper.py` 1404 lines (split `_wrapper_tmux.py`); sealed-prefix symlink + resolution in hookkit; feed the failure feed's `is_interrupt` to + `evaluate()`. +6. Playbook (worked twice now): recon swarm → build contract → ALL builders + concurrent (PR-048) → integrator → 3 adversarial lenses → fixer → memory + protocol → commit → PR with evidence. diff --git a/specs/watchdog.md b/specs/watchdog.md index 765b40b..39d17bd 100644 --- a/specs/watchdog.md +++ b/specs/watchdog.md @@ -1,9 +1,20 @@ # Watchdog / Heartbeat — anti-stall for long-running autonomous runs -**Status:** design spec (RFC). Implementation tracked as the follow-up to this PR. -**Owner:** core / orchestrator. +**Status:** implemented (v2 Phase 3 / WS-C, PR-A). Plan oracle checks 11–12 (`plans/zo-v2-rearchitecture.md`). +**Owner:** wrapper (`src/zo/wrapper.py`) + pure policy module (`src/zo/watchdog.py`). **Motivation:** a real failure mode observed in a long autonomous run — the whole team stalled *silently for ~38 hours* and nobody noticed until the human asked "where are we?". +> **Divergence from the original RFC (read this first).** The first draft of +> this spec proposed a *cron-scheduled self-invoke tick* owned by the +> orchestrator agent, with a nudge → respawn → reroute remediation ladder. +> `plans/zo-v2-rearchitecture.md` (Phase 3, "Watchdog first") supersedes that: +> the tick is an **external checker in the LifecycleWrapper poll loop** — a +> Python process that is *not* an LLM, so it cannot itself stall, hallucinate +> liveness, or be killed by the session it watches. Sections 3–6 below describe +> what shipped. Section 1 (the failure mode) and section 2 (requirements) are +> unchanged; requirement 1 ("active heartbeat, not passive wait") is now met by +> the poll loop rather than by cron. + --- ## 1. The failure mode @@ -27,67 +38,105 @@ This is distinct from context-window saturation (already handled by checkpoint 1. **Active heartbeat, not passive wait.** The run must have a wake source that fires on a wall-clock schedule independent of teammate messages, so an all-idle team cannot go unnoticed. 2. **Liveness by evidence, not by silence.** Detect progress from observable state (process is running, output files/heartbeats advancing), never from "no bad message arrived." -3. **Auto-remediation.** On a detected stall, re-mobilize (nudge/respawn the responsible agent, re-issue its task) rather than only alerting. -4. **No single point of failure.** The heartbeat is owned by the orchestrator/run itself, not by one killable monitor agent. -5. **Survives session boundaries.** Session-scoped timers (cron/wakeup) are re-armed at the start of every session; the *policy* lives in durable config so a fresh session re-establishes the watch automatically. -6. **Cheap + quiet.** Low token/compute cost; only speaks when it detects a stall or a state change, not on every tick. +3. **Auto-remediation.** On a detected stall, re-mobilize (nudge the responsible session) rather than only alerting — but never fight a stop the platform itself imposed. +4. **No single point of failure.** The checker is owned by the run's Python wrapper, not by one killable monitor agent. +5. **Survives session boundaries.** Nudge budgets and pause state are persisted on disk so a wrapper restart cannot refill them; the *policy* lives in `.zo/config.yaml` so a fresh session re-establishes the watch automatically. +6. **Cheap + quiet.** No LLM calls; only speaks (comms events) when it detects a stall, a pause, a resume, or an escalation — never on every tick. --- -## 3. Design +## 3. Design (as implemented) -### 3.1 Heartbeat tick (scheduled self-invoke) -A recurring, off-minute schedule (default `*/17 * * * *` — ~every 17 min, jittered, never on `:00`) enqueues a **watchdog tick** prompt to the orchestrator. Session-scoped by nature, so it is re-armed on session start from `project_config`. This is the wake source that closes the "all-idle → never re-invoked" hole. +### 3.1 The tick — external checker in the wrapper poll loop +`LifecycleWrapper.wait_for_completion(..., watchdog=, memory_root=, zo_session_id=)` builds a `WatchdogRunner` (`src/zo/_wrapper_watchdog.py`) when `watchdog.enabled` and a memory root are given. Both poll loops call `_watchdog_tick(process, text=…, can_nudge=…)` once per iteration: -### 3.2 Liveness probe (evidence-based) -Each tick evaluates, per active agent / critical-path job: -- **Agent liveness:** heartbeat file mtime fresh (< `stall_threshold`, default 20 min) AND/OR a live owned process. -- **Job liveness:** the real process PID is running (not just a wrapper/launcher that already exited); output artifacts advancing (new/growing files since last tick). -- **Progress delta:** did any tracked artifact change since the previous tick? +- **tmux** (`_wait_tmux`, the default): one `capture-pane` per poll (last 200 lines) feeds the never-block classifier and the progress digest; `can_nudge=pane_ready_for_nudge(text)` — an idle prompt is nudgeable, a busy pane (spinner, "esc to interrupt", dialog) is not, so a stall behind a hung tool call escalates after `escalate_grace_sec` instead of returning NUDGE forever. The tick runs *before* the liveness reads, so it also fires on the "suspected dead, re-check" path. +- **headless** (`_wait_headless`, `--no-tmux`): a byte cursor over the stdout/stderr logs (rolling 16 KB window); `can_nudge=False` (the `--print` prompt is one-shot argv, there is nothing to type into). -A **stall** = no progress delta AND no live critical-path process for ≥ N consecutive ticks (default N=1 for a hard stall, N=2 for a soft one). +The tick runs `zo.watchdog.evaluate()` — a pure, clock-injected function — and the wrapper performs the side effects. Every tick appends one JSONL line to `<memory_root>/heartbeats/_watchdog-ticks.jsonl` (`ts, action, stalled, reason, never_block, progress`) and persists `WatchdogState` to `<memory_root>/heartbeats/_watchdog.json` (both fail-open). -### 3.3 Long-job monitor (event stream) -For a specific long-running job, attach a streaming monitor on its log with a filter covering **both** progress and failure signatures (`epoch|elapsed|wrote|saved|Traceback|Error|FAILED|Killed|OOM|nan`). Silence-is-not-success: a filter that matches only the happy path stays quiet through a crashloop. This gives immediate notification on crash/completion between heartbeat ticks. +### 3.2 Liveness by evidence — four observers +`evaluate()` receives `progress: bool`, computed by four observers that each return True only on **new** evidence (`zo.watchdog.observe_*` + the runner's CPU sampler): -### 3.4 Remediation policy -On stall detection the watchdog escalates by policy: -1. **Nudge** the responsible agent (re-send its outstanding task). -2. If still dead next tick → **respawn** it from its on-disk resume/checkpoint and re-issue. -3. Prefer a **reliable executor**: if an agent role has failed to engage repeatedly (e.g., two silent dead spawns), route its critical-path task to a known-reliable agent instead of retrying the same role. Log the reroute. -4. **Never silently truncate**: every stall, reroute, and dropped item is logged and surfaced to the human at the next checkpoint. +- **Heartbeats** — hook-written JSON per agent at `<memory_root>/heartbeats/<agent_key>.json` (`HeartbeatRecord`; writer is `zo.hookkit._handle_heartbeat` → `zo._hook_heartbeat.stamp_heartbeat`, wired to `PostToolUse` `*` and stamped from Stop / SubagentStop / PreCompact / SessionEnd). Progress = any key's `tick_count` advanced past what the run has seen; files that pre-date the run are baselined and do not count. Freshness is three-state (`fresh|stale|unknown`); **unknown is never a stall verdict**. +- **Terminal text** — `progress_digest()` = SHA-1 of the pane/log text with spinner glyphs, elapsed counters, token counts, and the idle prompt stripped, so UI churn is not progress. Digest changes during a rate-limit pause are *not* progress (the banner itself churns). +- **Files** — mtime advance / appearance of `plan-ledger.json`, the comms log dir, `<delivery>/.zo/experiments`, plus `WatchdogConfig.progress_paths`. Directories are watched one level deep (a *new* entry counts; an in-place append to a nested file does not bump the parent), so point `progress_paths` at the file that actually grows — a training log or metrics file — when a long tool call is expected. +- **Process-tree CPU time** (`zo._proc.process_tree_cpu_seconds`, `ps -A -o pid=,ppid=,time=`) — the lead's tree burning ≥ 25 % of the wall time between two samples (`CPU_BUSY_FRACTION`) is positive activity: a 40-minute `python train.py` inside one silent Bash call is *working*, not stalled, even though heartbeats (PostToolUse-only) and text are quiet. An idle TUI redraws at ~1 % and does not count; a blocked MCP/network call burns nothing and still stalls. Requires a resolved lead pid (`Popen.pid` headless; the filtered `pgrep` child in tmux). -### 3.5 Configuration (`project_config`) -``` +**Process identity** (`zo._proc`, re-exported by `zo.watchdog`): `pid_alive` (ESRCH → dead, EPERM → alive), `process_start_identity` (`/proc/<pid>/stat` field 22 on Linux, `ps -o lstart=` on macOS), `is_process_dead(pid, recorded_identity)` — **positive proof only**: a recycled pid (identity mismatch) is dead, anything unknown is *not*. tmux resolves the claude child via `pgrep -n -P <pane_pid> -f claude` (newest child whose command line mentions claude — a shell's prompt helper or gitstatusd is never mistaken for the lead; no match → unknown); headless uses `Popen.pid`. Observed progress contradicts a dead verdict (progress wins), and positive-proof death is escalated **once per run**, not once per stall. + +A **stall** = `process_dead is True` OR `now − last_progress_at ≥ stall_threshold_sec`, evaluated only after `startup_grace_sec` and only when no never-block reason applies. + +### 3.3 Never-block taxonomy — stops that are never fought +Before any stall decision the tick classifies the last 60 non-empty lines of terminal text (`classify_never_block`, precedence order): + +| Reason | Meaning | Watchdog behaviour | +|---|---|---| +| `user_abort` | Ctrl-C / `is_interrupt` / "⎿ Interrupted by user" / "⎿ Interrupted · What should Claude do instead?" | never nudge | +| `context_limit` | context full / prompt too long / compaction banner | never nudge; may escalate after `stall_threshold_sec` | +| `rate_limit` | three tiers: **banner** ("You've hit your usage limit · resets at 3pm", `weekly … limit`, `quota exceeded`, `too many requests`, 429 *with* rate/limit vocabulary, `rate_limit_error`, …), **prose** ("rate limit" in running text), **loose** ("limit reached", "try again later", "hit … limit", "resets … at", "5-hour" — only when the SAME line carries rate/usage/quota/request/API vocabulary; "patience limit reached at epoch 30" is not a banner). All three **pause** (§3.5), never nudge; only banner/parsed-reset evidence classifies an *exit* as `RATE_LIMITED` | +| `auth_error` | `authentication_error`, "please run /login", 401/403 *with* auth vocabulary | never nudge; may escalate after `stall_threshold_sec` | +| `awaiting_input` | permission / question dialogs ("Do you want to proceed? ❯ 1. Yes", "esc to cancel") | never nudge (a nudge would answer the dialog) | +| `compacting` | a heartbeat with `status=compacting` inside the last 5 min | counts as progress; resets the stall clock | + +Pattern tables live in `zo._watchdog_text` (ported from oh-my-claudecode under MIT, with contract-specific tightening: no bare `429`, no bare `overloaded`, bare `interrupt` excluded, git-log/diff lines and saved-transcript `cat …` commands stripped first). Ambiguous → do not nudge. + +### 3.4 Remediation policy — nudge (bounded) → escalate +1. **Nudge** (tmux only). After a stall persists for `nudge_delay_sec`, paste `nudge_message` into the lead pane via a **named** tmux buffer (`_paste_and_submit`) and press Enter — but only if `pane_ready_for_nudge(text)`: idle prompt visible, no active task, no permission dialog. Budget: `nudge_budget` per run (default 3), `nudge_delay_sec` between nudges; both persisted in `_watchdog.json`. Each nudge logs a comms `checkpoint(agent="watchdog", subtask="nudge")`; a skipped nudge logs `subtask="nudge-skipped"`. +2. **Escalate** — once per stall — when the budget is exhausted, or nudging is impossible (headless, or a tmux pane that stays busy / in a dialog) and `escalate_grace_sec` has passed, or the process is positively dead (once per run): comms `error(agent="watchdog", error_type="stall", severity="blocking", escalated_to="human")`, `LeadProcess.stalled=True`. Headless additionally `kill_session`s the lead (`kill_headless_on_escalate`) because there is no other lever; tmux keeps the human-facing pane alive and lets the loop end naturally. The wrapper returns `AgentStatus.STALLED`; the CLI prints "Session stalled — watchdog escalated". +3. **Respawn / reroute** — *deferred* to the fresh-context driver (PR-B, plan check 13), which consumes `STALLED` and `hard_max_restarts`. The wrapper never relaunches inside its own poll loop. +4. **Never silently truncate**: first stall detection logs a `warning` error event; every escalation is a `blocking` error event; every tick is traced to `_watchdog-ticks.jsonl`. + +### 3.5 Rate-limit wait-and-resume (check 12) +On `rate_limit`: enter a **pause** (`AgentStatus.PAUSED_RATE_LIMIT`, `paused_until` = parsed reset time + 15 s slack via `parse_rate_limit_reset` — "resets at 3pm", "resets at 14:30", "try again in 5 minutes", "retry-after: 90", ISO timestamps; the newest banner in the transcript wins — else exponential backoff `rate_limit_backoff_base_sec · 2^attempt` capped at `rate_limit_backoff_max_sec`). Banner clock times are the operator's **local** time (Claude Code renders the reset in local time): the runner passes `tz=local_tz()` (`LifecycleWrapper(tz=…)` / `WatchdogRunner(tz=…)` are injectable) — parsing "resets at 3pm" as 15:00 UTC would push a US/Pacific reset to tomorrow. A parsed reset further away than `rate_limit_max_pause_sec` (a stale clock time that rolled over to tomorrow) is not honoured; backoff applies. While paused the loop keeps polling (never one long `sleep`); paused time is excluded from the run timeout **until an escalation raised during the pause** (resume unverified / max pause exceeded) — after the hand-off the wall clock counts again. + +The real TUI never clears the usage-limit line, so "banner gone" is defined by evidence, not by pixels: once `paused_until` has passed, a banner whose rate-limit lines are **unchanged since the pause began** (`WatchdogState.pause_banner_key`) is *stale* and treated as gone — tmux sends up to `resume_nudge_budget` resume nudges, headless waits for evidence; a banner with *new* rate-limit lines is fresh and extends the pause (a real later reset is honoured, a re-printed past time falls back to backoff). A resume is **verified** only by real progress (heartbeat / file / CPU delta) — with the banner still on screen once the reset has passed, or any time once it is gone → `AgentStatus.RUNNING`, `pause_total_sec` accumulated, comms `rate-limit-resume`; the resolved banner (`spent_banner_key`) is ignored until it scrolls out of the 60-line tail, so it cannot re-pause the run. Progress while the banner is fresh and the reset still ahead does not resume (no PAUSE/RESUME flapping). Exceeding `rate_limit_max_pause_sec` escalates. If the process exits while a pause is open the wrapper returns `RATE_LIMITED` with `resume_at` **only with corroboration** — a parsed reset time (`resume_at` set), an unambiguous banner, or a non-zero exit code; a session that finished normally after *mentioning* rate limits stays `COMPLETED`. The CLI prints "rerun `zo continue` after". + +### 3.6 Configuration (`.zo/config.yaml` → `ProjectConfig.watchdog`, model `zo.watchdog.WatchdogConfig`) +```yaml watchdog: enabled: true - tick_cron: "*/17 * * * *" # off-minute; re-armed each session - stall_threshold_min: 20 # heartbeat/artifact staleness => suspect - stall_ticks_hard: 1 # no-process + no-progress => stall now - stall_ticks_soft: 2 # progress-but-slow => stall after N ticks - remediation: [nudge, respawn, reroute] - monitor_failure_signatures: ["Traceback","Error","FAILED","Killed","OOM","nan"] + stall_threshold_sec: 1200 # 20 min; check 11 tests use 600 + startup_grace_sec: 120 + nudge_enabled: true # tmux only; pane-ready guard always applies + nudge_delay_sec: 30 # dwell before first nudge and between nudges + nudge_budget: 3 # per run, persisted + nudge_message: "Continue working on your assigned task and report concrete progress (not ACK-only)." + resume_nudge_budget: 2 + escalate_grace_sec: 120 + kill_headless_on_escalate: true # only after evidence says idle (see §3.2 CPU observer) + rate_limit_backoff_base_sec: 60 + rate_limit_backoff_max_sec: 1800 + rate_limit_max_pause_sec: 21600 # 6 h + hard_max_restarts: 3 # consumed by the PR-B driver + progress_paths: [] # extra files/dirs whose mtime advance is progress (name the file that grows, e.g. a training log) ``` +The block is strict (unknown keys inside `watchdog:` are a validation error); `ProjectConfig` itself ignores unknown *top-level* keys so legacy configs load. Legacy `targets/*.target.md` projects have no config → defaults (ON). Precedence: `--no-watchdog` > env (`ZO_WATCHDOG=0` kill switch, `ZO_WATCHDOG_STALL_SEC=N`) > `.zo/config.yaml` > defaults (`resolve_watchdog_config`). --- -## 4. Integration points -- **`orchestrator.py`** — arm the heartbeat cron on run/session start (re-arm if a prior schedule is gone); own the watchdog-tick handler. -- **`comms.py`** — the watchdog reads agent heartbeats + emits nudge/respawn messages through the existing comms bus. -- **`.claude/agents/lead-orchestrator.md`** — add the standing rule: never end a turn in passive "hold and wait" while critical-path work is outstanding; a heartbeat must be armed. Verify liveness by evidence (pgrep the real PID, artifact deltas), not by absence of bad news. -- **`project_config.py`** — the `watchdog:` block above; default-on. -- A dedicated monitor agent (if any) becomes a *helper*, not the sole mechanism — the orchestrator-owned heartbeat is the backstop. +## 4. Integration points (real files) +- **`src/zo/watchdog.py`** — pure policy: `classify_never_block`, `evaluate(…, tz=)`, observers, persistence, `compute_pause_until`; re-exports `zo._watchdog_models` (`HeartbeatRecord`, `WatchdogConfig`, `WatchdogState`, `StallAction`, `StallVerdict`, `resolve_watchdog_config`), `zo._watchdog_text` (patterns, `rate_limit_match`, `rate_limit_banner_key`) and `zo._proc` (identity, `process_tree_cpu_seconds`). +- **`src/zo/_wrapper_watchdog.py`** — `WatchdogRunner` (config + state + memory root + clock + tz + evidence paths + CPU probe; `tick`, `record_nudge`, `paused_seconds` (capped at an in-pause escalation), `rate_limit_exit_evidence`, `parsed_resume_at`, per-tick JSONL trace). +- **`src/zo/wrapper.py`** — `wait_for_completion(watchdog=, memory_root=, zo_session_id=)`, `_watchdog_tick` from `_wait_tmux` and `_wait_headless`, `_paste_and_submit`, pid + start-identity capture; `src/zo/_wrapper_models.py` — `AgentStatus.PAUSED_RATE_LIMIT`, `STALLED`; `LeadProcess.pid_start_identity/nudges_used/stalled/paused_until/resume_at/pause_total_sec`. +- **`src/zo/hookkit.py`** + **`src/zo/_hook_heartbeat.py`** — `_handle_heartbeat` resolves the memory root and delegates to `stamp_heartbeat` (stdlib-only writer, debounced, atomic), stamped from PostToolUse `*` (`.claude/settings.json`), Stop, SubagentStop, PreCompact, SessionEnd; `heartbeats/` is in `_SEALED_DEFAULTS` so agents cannot forge liveness via Write/Edit; `.gitignore` covers `memory/zo-platform/heartbeats/`, the zo-dir scaffold / `zo migrate` `.zo/.gitignore` covers `memory/heartbeats/` + `memory/plan-ledger.json` + `memory/contracts.json`, and `LifecycleWrapper._start_watchdog` idempotently appends `memory/heartbeats/` to an existing `.zo/.gitignore` so runtime files never enter delivery history. +- **`src/zo/project_config.py`** — `ProjectConfig.watchdog: WatchdogConfig` (round-trips through `save_project_config`). +- **`src/zo/cli.py`** — `ProjectContext.make_project_config()`, `_resolve_watchdog`, `--no-watchdog` on `build`/`continue`, `ZO_SESSION_ID` exported to hooks, `_launch_and_monitor(watchdog=, memory_root=, zo_session_id=)`, `_print_session_outcome` for `STALLED` / `RATE_LIMITED`. +- **`src/zo/comms.py`** — no new event types: `checkpoint(agent="watchdog", subtask=nudge|nudge-skipped|rate-limit-pause|rate-limit-resume)` and `error(error_type="stall")`, both rendered live by `_print_status`. +- **Not touched by design:** `orchestrator.py` (no orchestrator-owned tick), `.claude/agents/lead-orchestrator.md` liveness rule stays advisory prose, `surrogate.py` (identity fix deferred). --- -## 5. Acceptance / tests -- Unit: stall-detection predicate (fresh vs stale heartbeat; live vs dead PID; progress delta present/absent) → correct hard/soft/no-stall classification. -- Unit: remediation escalation (nudge → respawn → reroute) state machine. -- Unit: config parse + defaults + re-arm idempotency (arming twice doesn't double-schedule). -- Integration: simulate an all-idle team with no progress across ticks → watchdog fires remediation; simulate a live-but-slow job → no false stall. -- Regression: the failure-signature filter matches a crashloop/OOM sample (silence-is-not-success guard). +## 5. Acceptance / tests (checks 11–12) +- `tests/unit/test_watchdog.py` — taxonomy positives/negatives/precedence, `parse_rate_limit_reset`, three-state freshness, `evaluate()` scenario table with an injected clock (seeded 10-minute stall → `NUDGE`×3 → `ESCALATE` once; rate-limited never nudged; pause → resume-nudge → verified `RESUME`; headless resume requires progress; `process_dead` → escalate; startup grace; awaiting-input never nudged; compacting resets the clock; each observer), process identity (Linux `/proc` parse, macOS `ps`, EPERM alive, recycled pid dead, unknown not dead), persistence round-trips, `resolve_watchdog_config` kill switch. +- `tests/unit/test_wrapper.py` — `test_seeded_10min_stall_detected_and_escalated_within_one_poll` (check 11), `test_seeded_rate_limited_session_is_never_nudged` (check 11), `test_seeded_rate_limit_pause_auto_resumes_on_reset` + `test_seeded_static_banner_resumes_at_reset_without_rollover` (check 12), the same trio on the headless harness plus `test_seeded_silent_training_with_busy_cpu_is_not_killed`, `test_busy_pane_never_sends_keys_and_escalates_after_grace`, `test_paused_seconds_stops_accruing_at_escalation`, `test_tz_threads_into_banner_parse`, `test_watchdog_tick_runs_on_suspected_dead_path`, `test_single_pane_capture_per_poll`, `test_permission_dialog_is_never_nudged`, `test_paste_and_submit_uses_named_buffer`, `test_watchdog_disabled_when_config_off_or_no_memory_root`. +- `tests/unit/test_hookkit.py` + `tests/integration/test_hooks_shim.py` — heartbeat written keyed by `agent_id` / `lead-<session_id>`, validates as `HeartbeatRecord`, tick_count increments, debounce, status per event, fail-open without a memory root, heartbeats sealed (seeded Write → deny), settings.json PostToolUse entry asserted, shim end-to-end. +- `tests/unit/test_project_config.py::TestWatchdogConfigThreading` — round-trip, legacy defaults, unknown top-level key ignored, seeded typo rejected. `tests/unit/test_cli.py::TestWatchdogCliThreading` — `--no-watchdog` / config / env reach the launch as `enabled=False`, default path passes config + memory root + session id, `_launch_and_monitor` forwards keyword args to `wait_for_completion`, `STALLED` / `RATE_LIMITED` messages. --- -## 6. Out of scope (this PR is spec-only) -Code, tests, and orchestrator wiring land in the follow-up implementation PR against this spec. Context-window saturation handling (checkpoint→respawn) already exists and is unchanged — this feature covers **idle-stall**, the orthogonal hole. +## 6. Scope boundaries +- **In:** stall detection, bounded nudges, rate-limit pause/resume, escalation, per-tick trace, config threading — all inside one `zo build` session. +- **Deferred to PR-B (fresh-context driver, check 13):** acting on `STALLED` / `RATE_LIMITED` above the wrapper (relaunch with a re-derived prompt, `hard_max_restarts`), and the `surrogate.py` pid + start-time identity fix. +- **Not planned:** an LLM monitor agent, a cron tick, respawn/reroute inside the poll loop, psutil, new comms event types. +- Context-window saturation handling (checkpoint→respawn) is unchanged — this feature covers **idle-stall**, the orthogonal hole. diff --git a/src/zo/_hook_heartbeat.py b/src/zo/_hook_heartbeat.py new file mode 100644 index 0000000..04ab0da --- /dev/null +++ b/src/zo/_hook_heartbeat.py @@ -0,0 +1,145 @@ +"""Heartbeat writer for the hook plane (WS-C, plan oracle checks 11-12). + +Stamps ``<memory_root>/heartbeats/<agent_key>.json`` on every routed hook +event so the wrapper-side watchdog (``zo.watchdog`` / ``zo._wrapper_watchdog``) +has per-agent liveness evidence. + +Design constraints (see specs/watchdog.md): + +* **stdlib-only** — this path runs on every ``PostToolUse``; it must never + pay the pydantic import. The JSON body mirrors ``zo.watchdog.HeartbeatRecord`` + field-for-field (a test validates the shape; it is never imported here). +* **advisory / fail-open** — no error escapes to the session; a missing or + unwritable memory root simply means no heartbeat. +* **never creates a memory root** — heartbeats are evidence, not scaffolding. +""" + +from __future__ import annotations + +import contextlib +import json +import os +import re +from datetime import UTC, datetime +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from pathlib import Path + +__all__ = [ + "HEARTBEATS_DIRNAME", + "HEARTBEAT_STATUS_BY_EVENT", + "agent_identity", + "stamp_heartbeat", +] + +HEARTBEATS_DIRNAME = "heartbeats" +HEARTBEAT_SCHEMA_VERSION = 1 +HEARTBEAT_DEBOUNCE_SEC = 2.0 +# hook_event_name → zo.watchdog.HeartbeatStatus value (kept as literals on +# purpose: the heartbeat path must not import zo.watchdog/pydantic). +HEARTBEAT_STATUS_BY_EVENT = { + "PostToolUse": "executing", + "Stop": "ready", + "SubagentStop": "shutdown", + "PreCompact": "compacting", + "SessionEnd": "shutdown", +} +_AGENT_KEY_UNSAFE = re.compile(r"[^A-Za-z0-9._-]") + + +def agent_identity(data: dict) -> tuple[str | None, str | None]: + """Return ``(agent_type, agent_id)`` from a hook payload (None when absent). + + Live SubagentStop / PostToolUseFailure / PostToolUse payloads carry + ``agent_id`` and ``agent_type``; PreToolUse payloads carry neither. + Distinct from ``hookkit._agent_name`` (contract lookup key) on purpose. + """ + def _pick(*keys: str) -> str | None: + for key in keys: + value = data.get(key) + if isinstance(value, str) and value.strip(): + return value.strip() + return None + + return _pick("agent_type", "subagent_type"), _pick("agent_id") + + +def _env_int(name: str) -> int | None: + raw = os.environ.get(name, "").strip() + return int(raw) if raw.isdigit() else None + + +def _read_json_dict(path: Path) -> dict: + """Best-effort JSON object read; ``{}`` on any error (fail-open).""" + try: + loaded = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + return loaded if isinstance(loaded, dict) else {} + + +def _atomic_write_json(path: Path, record: dict) -> None: + """Write ``record`` via tmp + ``os.replace`` (no partial file is ever visible).""" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(f".{path.name}.{os.getpid()}.tmp") + try: + tmp.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8") + os.replace(tmp, path) + finally: + with contextlib.suppress(OSError): + tmp.unlink() + + +def _heartbeat_record(data: dict, *, agent_key: str, tick_count: int) -> dict: + """Build the JSON body — field set mirrors ``zo.watchdog.HeartbeatRecord``.""" + agent_type, agent_id = agent_identity(data) + event = str(data.get("hook_event_name") or "PostToolUse") + last_event = data.get("tool_name") if event == "PostToolUse" else event + return { + "schema_version": HEARTBEAT_SCHEMA_VERSION, + "agent_key": agent_key, + "agent_id": agent_id, + "agent_type": agent_type, + "session_id": str(data.get("session_id") or "unknown"), + "zo_session_id": os.environ.get("ZO_SESSION_ID") or None, + "pid": _env_int("ZO_LEAD_PID"), + "process_start_identity": os.environ.get("ZO_LEAD_PID_IDENTITY") or None, + "last_tick_at": datetime.now(UTC).isoformat(), + "status": HEARTBEAT_STATUS_BY_EVENT.get(event, "executing"), + "last_event": str(last_event or event), + "tick_count": tick_count, + } + + +def stamp_heartbeat(data: dict, *, memory_root: Path) -> None: + """Stamp ``<memory_root>/heartbeats/<agent_key>.json`` (advisory, fail-open). + + ``agent_key`` is ``agent_id`` when present else ``lead-<session_id>``, + sanitised for the filesystem (the record's ``agent_key`` field equals the + filename stem). PostToolUse writes are debounced to one per + ``HEARTBEAT_DEBOUNCE_SEC``; other events always stamp and ``tick_count`` + is monotonic across stamps. + + Args: + data: The hook payload (already parsed JSON dict). + memory_root: Existing per-project memory root; the caller resolves + it and skips the stamp when it is unknown or not a directory. + """ + session_id = str(data.get("session_id") or "unknown") + _agent_type, agent_id = agent_identity(data) + agent_key = _AGENT_KEY_UNSAFE.sub("_", agent_id or f"lead-{session_id}") + path = memory_root / HEARTBEATS_DIRNAME / f"{agent_key}.json" + event = str(data.get("hook_event_name") or "PostToolUse") + try: + with contextlib.suppress(OSError): + age = datetime.now(UTC).timestamp() - path.stat().st_mtime + if event == "PostToolUse" and 0 <= age < HEARTBEAT_DEBOUNCE_SEC: + return + previous = _read_json_dict(path).get("tick_count", 0) + valid = isinstance(previous, int) and not isinstance(previous, bool) and previous >= 0 + tick_count = (previous if valid else 0) + 1 + record = _heartbeat_record(data, agent_key=agent_key, tick_count=tick_count) + _atomic_write_json(path, record) + except OSError: + return diff --git a/src/zo/_proc.py b/src/zo/_proc.py new file mode 100644 index 0000000..234ccc9 --- /dev/null +++ b/src/zo/_proc.py @@ -0,0 +1,237 @@ +"""Process identity helpers for the watchdog (WS-C, oracle checks 11-12). + +A pid alone is not proof of anything: pids are recycled. We pair a pid with a +platform-tagged *start identity* (``linux:<starttime ticks>``, +``darwin:<epoch>:<usec>``) so a recycled pid is detected as death, while an +unknown/malformed identity is never treated as positive proof of death. + +Ported from oh-my-claudecode ``src/team/team-owner-epoch.ts`` (MIT License, +Copyright (c) 2025 Yeachan Heo) with the ``ps -o lstart=`` darwin fallback +only (no ``sysctl kern.proc.pid`` binary parsing); stdlib only, no psutil. +""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +from datetime import datetime +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Callable + +__all__ = [ + "identities_may_match", + "is_process_dead", + "is_valid_process_start_identity", + "parse_linux_stat_starttime", + "parse_ps_time", + "pid_alive", + "process_start_identity", + "process_tree_cpu_seconds", +] + +_MAX_IDENTITY_LEN = 1024 +_LINUX_ID = re.compile(r"^linux:[1-9]\d*$") +_DARWIN_ID = re.compile(r"^darwin:([1-9]\d*):(\d+)$") +_CONTROL_CHARS = re.compile(r"[\x00-\x1f\x7f]") +_PS_LSTART_FORMAT = "%a %b %d %H:%M:%S %Y" + + +def _platform(platform: str | None) -> str: + return platform or sys.platform + + +def pid_alive(pid: int) -> bool: + """Signal-0 liveness: ESRCH → False; EPERM → True (alive, not ours).""" + if not isinstance(pid, int) or pid < 1: + return False + try: + os.kill(pid, 0) + except ProcessLookupError: + return False + except PermissionError: + return True + except OSError: + return False + return True + + +def parse_linux_stat_starttime(stat_text: str) -> str | None: + """Return field 22 (``starttime``) of a ``/proc/<pid>/stat`` line. + + The command name (field 2) is wrapped in parentheses and may itself + contain spaces or ``)``, so parsing splits after the LAST ``)``. + """ + close = stat_text.rfind(")") + if close < 0: + return None + fields = stat_text[close + 1:].split() + # fields[0] is state (field 3); starttime is field 22 → index 19. + if len(fields) < 20 or not fields[19].isdigit(): + return None + return fields[19] + + +def _read_proc_stat(pid: int) -> str: + with open(f"/proc/{pid}/stat", encoding="utf-8") as fh: + return fh.read() + + +def _ps_lstart(pid: int, run: Callable) -> str: + env = {**os.environ, "LC_ALL": "C", "LANG": "C"} + result = run( + ["ps", "-o", "lstart=", "-p", str(pid)], + capture_output=True, text=True, env=env, timeout=5, + ) + stdout = getattr(result, "stdout", "") or "" + return stdout.strip() + + +def process_start_identity( + pid: int, *, platform: str | None = None, run: Callable = subprocess.run, +) -> str | None: + """Platform-tagged process start identity, or ``None`` on any failure. + + linux: ``linux:<starttime>`` from ``/proc/<pid>/stat``; + darwin: ``darwin:<epoch_seconds>:0`` from ``ps -o lstart=`` (LC_ALL=C); + other: ``<platform>:<raw ps lstart>``. + """ + if not isinstance(pid, int) or pid < 1: + return None + plat = _platform(platform) + try: + if plat.startswith("linux"): + ticks = parse_linux_stat_starttime(_read_proc_stat(pid)) + return f"linux:{ticks}" if ticks else None + started = _ps_lstart(pid, run) + if not started: + return None + if plat == "darwin": + epoch = int(datetime.strptime(started, _PS_LSTART_FORMAT).timestamp()) + return f"darwin:{epoch}:0" + return f"{plat}:{started}" + except Exception: # identity is best-effort and never raises + return None + + +def is_valid_process_start_identity(value: object, *, platform: str | None = None) -> bool: + """Regex allowlist for identities (≤ 1024 chars, platform-tagged).""" + if not isinstance(value, str) or not value or len(value) > _MAX_IDENTITY_LEN: + return False + plat = _platform(platform) + if plat.startswith("linux"): + return _LINUX_ID.match(value) is not None + if plat == "darwin": + match = _DARWIN_ID.match(value) + return match is not None and int(match.group(2)) < 1_000_000 + sep = value.find(":") + if sep <= 0 or value[:sep] != plat: + return False + rest = value[sep + 1:] + return bool(rest) and _CONTROL_CHARS.search(rest) is None + + +def identities_may_match(recorded: str, observed: str) -> bool: + """Equal, or darwin same-second where either usec component is the ``0`` wildcard.""" + if recorded == observed: + return True + rec = _DARWIN_ID.match(recorded or "") + obs = _DARWIN_ID.match(observed or "") + return ( + rec is not None and obs is not None + and rec.group(1) == obs.group(1) + and (rec.group(2) == "0" or obs.group(2) == "0") + ) + + +def parse_ps_time(value: str) -> float | None: + """Seconds from a ``ps -o time=`` field: ``[[dd-]hh:]mm:ss[.cc]``; ``None`` if malformed.""" + text = (value or "").strip() + if not text: + return None + days = 0 + if "-" in text: + day_part, text = text.split("-", 1) + if not day_part.isdigit(): + return None + days = int(day_part) + parts = text.split(":") + if not 1 <= len(parts) <= 3: + return None + try: + secs = float(parts[-1]) + for unit, part in zip((60, 3600), reversed(parts[:-1]), strict=False): + secs += int(part) * unit + except ValueError: + return None + return days * 86400 + secs + + +def _ps_table(run: Callable) -> str: + env = {**os.environ, "LC_ALL": "C", "LANG": "C"} + result = run( + ["ps", "-A", "-o", "pid=,ppid=,time="], + capture_output=True, text=True, env=env, timeout=5, + ) + return getattr(result, "stdout", "") or "" + + +def process_tree_cpu_seconds(pid: int, *, run: Callable = subprocess.run) -> float | None: + """Cumulative CPU seconds of ``pid`` and all its descendants (``ps -A``); ``None`` on failure. + + A positive-activity signal for the watchdog: a lead whose process tree + keeps burning CPU (a long silent training run inside one tool call) is + not stalled even when heartbeats and text are silent. Never raises. + """ + if not isinstance(pid, int) or pid < 1: + return None + try: + children: dict[int, list[int]] = {} + cpu: dict[int, float] = {} + for line in _ps_table(run).splitlines(): + fields = line.split() + if len(fields) != 3 or not fields[0].isdigit() or not fields[1].isdigit(): + continue + secs = parse_ps_time(fields[2]) + if secs is None: + continue + cpu[int(fields[0])] = secs + children.setdefault(int(fields[1]), []).append(int(fields[0])) + if pid not in cpu: + return None + total, stack, seen = 0.0, [pid], set() + while stack: + current = stack.pop() + if current in seen: + continue + seen.add(current) + total += cpu.get(current, 0.0) + stack.extend(children.get(current, ())) + return total + except Exception: # advisory evidence: never raises + return None + + +def is_process_dead( + pid: int | None, recorded_identity: str | None, *, + platform: str | None = None, run: Callable = subprocess.run, +) -> bool: + """POSITIVE PROOF ONLY: ``True`` iff the pid is gone or provably recycled. + + ``pid`` None → False; ESRCH → True; alive + valid recorded identity + + valid observed identity that cannot match → True; anything unknown, + malformed, or EPERM → False. + """ + if pid is None or not isinstance(pid, int) or pid < 1: + return False + if not pid_alive(pid): + return True + if not is_valid_process_start_identity(recorded_identity, platform=platform): + return False + observed = process_start_identity(pid, platform=platform, run=run) + if not is_valid_process_start_identity(observed, platform=platform): + return False + return not identities_may_match(str(recorded_identity), str(observed)) diff --git a/src/zo/_watchdog_models.py b/src/zo/_watchdog_models.py new file mode 100644 index 0000000..c9c0529 --- /dev/null +++ b/src/zo/_watchdog_models.py @@ -0,0 +1,178 @@ +"""Pydantic models + config for the watchdog (WS-C, oracle checks 11-12). + +Split out of ``zo.watchdog`` (which re-exports every name here) so the policy +module stays under the 500-line rule. Nothing in this module does I/O. +""" + +from __future__ import annotations + +import os +from datetime import datetime # noqa: TC003 — pydantic resolves field annotations at runtime +from enum import StrEnum +from typing import TYPE_CHECKING + +from pydantic import BaseModel, ConfigDict, Field + +from zo._watchdog_text import NeverBlockReason, _aware + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + +__all__ = [ + "SCHEMA_VERSION", "Freshness", "HeartbeatRecord", "HeartbeatStatus", "StallAction", + "StallVerdict", "WatchdogConfig", "WatchdogState", "new_state", "resolve_watchdog_config", +] + +SCHEMA_VERSION = 1 + + +# ---------------------------------------------------------------- heartbeats + +class HeartbeatStatus(StrEnum): + """Hook-derived liveness status of one agent key.""" + + READY = "ready" + EXECUTING = "executing" + COMPACTING = "compacting" + SHUTDOWN = "shutdown" + + +class HeartbeatRecord(BaseModel): + """One ``<memory_root>/heartbeats/<agent_key>.json`` file.""" + + schema_version: int = SCHEMA_VERSION + agent_key: str + agent_id: str | None = None + agent_type: str | None = None + session_id: str + zo_session_id: str | None = None + pid: int | None = None + process_start_identity: str | None = None + last_tick_at: datetime + status: HeartbeatStatus = HeartbeatStatus.EXECUTING + last_event: str = "" + tick_count: int = 0 + + +class Freshness(StrEnum): + """Three-state heartbeat freshness; UNKNOWN is never a stall verdict.""" + + FRESH = "fresh" + STALE = "stale" + UNKNOWN = "unknown" + + +# ------------------------------------------------------------ config + state + +class WatchdogConfig(BaseModel): + """Per-project watchdog policy (``ProjectConfig.watchdog``).""" + + model_config = ConfigDict(extra="forbid") + + enabled: bool = True + stall_threshold_sec: int = 1200 + startup_grace_sec: int = 120 + nudge_enabled: bool = True + nudge_delay_sec: int = 30 + nudge_budget: int = 3 + nudge_message: str = ( + "Continue working on your assigned task and report concrete progress (not ACK-only)." + ) + resume_nudge_budget: int = 2 + escalate_grace_sec: int = 120 + kill_headless_on_escalate: bool = True + rate_limit_backoff_base_sec: int = 60 + rate_limit_backoff_max_sec: int = 1800 + rate_limit_max_pause_sec: int = 6 * 3600 + hard_max_restarts: int = 3 + progress_paths: list[str] = Field(default_factory=list) + + +def resolve_watchdog_config( + project: WatchdogConfig | None = None, *, env: Mapping[str, str] | None = None, +) -> WatchdogConfig: + """Project config + env overrides (``ZO_WATCHDOG=0`` kill switch, ``ZO_WATCHDOG_STALL_SEC``).""" + env = os.environ if env is None else env + base = project if project is not None else WatchdogConfig() + update: dict[str, object] = {} + if env.get("ZO_WATCHDOG", "").strip().lower() in {"0", "false", "no", "off"}: + update["enabled"] = False + stall = env.get("ZO_WATCHDOG_STALL_SEC", "").strip() + if stall.isdigit() and int(stall) > 0: + update["stall_threshold_sec"] = int(stall) + return base.model_copy(update=update) + + +class WatchdogState(BaseModel): + """Per-run watchdog state, persisted at ``<memory_root>/heartbeats/_watchdog.json``. + + ``pause_banner_key`` identifies the banner text the current pause was + entered on (unchanged at expiry → stale banner, not a fresh limit); + ``spent_banner_key`` is the banner of a pause resolved by verified + progress (still visible in the tail but no longer evidence); + ``pause_evidence`` records the tier of evidence (``reset`` — a parsed + reset time; ``banner`` / ``prose`` / ``loose``) for exit classification; + ``dead_escalated_at`` makes the positive-proof-dead escalation fire once. + """ + + zo_session_id: str = "" + started_at: datetime + last_tick_at: datetime | None = None + ticks: int = 0 + last_progress_at: datetime + baseline_ticks: dict[str, int] = Field(default_factory=dict) + seen_ticks: dict[str, int] = Field(default_factory=dict) + last_digest: str | None = None + last_file_mtimes: dict[str, float] = Field(default_factory=dict) + stall_since: datetime | None = None + nudges_used: int = 0 + last_nudge_at: datetime | None = None + resume_nudges_used: int = 0 + escalated_at: datetime | None = None + dead_escalated_at: datetime | None = None + stall_events: int = 0 + paused_at: datetime | None = None + paused_until: datetime | None = None + paused_reason: str | None = None + pause_attempts: int = 0 + pause_banner_key: str | None = None + pause_evidence: str | None = None + spent_banner_key: str | None = None + total_paused_sec: float = 0.0 + last_never_block: str | None = None + + +def new_state(*, now: datetime, zo_session_id: str = "", + heartbeats: Sequence[HeartbeatRecord] = ()) -> WatchdogState: + """Fresh state; pre-existing heartbeat files are baselined and do not count as progress.""" + baseline = {hb.agent_key: hb.tick_count for hb in heartbeats} + return WatchdogState( + zo_session_id=zo_session_id, started_at=_aware(now), last_progress_at=_aware(now), + baseline_ticks=dict(baseline), seen_ticks=dict(baseline), + ) + + +# ------------------------------------------------------------ stall policy + +class StallAction(StrEnum): + """What the caller should do this tick.""" + + NONE = "none" + NUDGE = "nudge" + RESUME_NUDGE = "resume_nudge" + PAUSE = "pause" + RESUME = "resume" + ESCALATE = "escalate" + + +class StallVerdict(BaseModel): + """Outcome of one ``evaluate()`` tick.""" + + action: StallAction + stalled: bool + reason: str + never_block: NeverBlockReason | None = None + freshness: Freshness = Freshness.UNKNOWN + process_dead: bool | None = None + progress: bool = False + evaluated_at: datetime diff --git a/src/zo/_watchdog_text.py b/src/zo/_watchdog_text.py new file mode 100644 index 0000000..63280f5 --- /dev/null +++ b/src/zo/_watchdog_text.py @@ -0,0 +1,300 @@ +"""Terminal-text classifiers for the watchdog (WS-C, oracle checks 11-12). + +Pure functions over captured pane / stdout text: ANSI normalization, +progress digest, never-block taxonomy, rate-limit reset parsing and the +pane-ready guard. No I/O. Public names are re-exported by ``zo.watchdog``. + +Pattern tables ported from oh-my-claudecode (MIT License, Copyright (c) 2025 +Yeachan Heo): ``src/hooks/todo-continuation/index.ts`` (context #213, rate +#777, auth #1308, user-abort — bare ``interrupt`` excluded per #2478), +``src/features/rate-limit-wait/tmux-detector.ts`` (rate-limit screen text, +git-output stripping, saved-transcript reject), ``src/team/tmux-session.ts`` +(``paneLooksReady`` / ``paneHasActiveTask``). Contract adjustments: no bare +``429`` / ``overloaded``; ``awaiting_input`` category added. +""" + +from __future__ import annotations + +import hashlib +import re +from datetime import UTC, datetime, time, timedelta, tzinfo +from enum import StrEnum +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Callable, Sequence + + from zo.watchdog import HeartbeatRecord + +_TAIL_LINES = 60 +_PANE_TAIL_LINES = 40 + + +def _aware(dt: datetime) -> datetime: + return dt if dt.tzinfo is not None else dt.replace(tzinfo=UTC) + + +def _secs(later: datetime, earlier: datetime) -> float: + return (_aware(later) - _aware(earlier)).total_seconds() + + +class NeverBlockReason(StrEnum): + """Conditions under which the watchdog must never nudge.""" + + USER_ABORT = "user_abort" + CONTEXT_LIMIT = "context_limit" + RATE_LIMIT = "rate_limit" + AUTH_ERROR = "auth_error" + AWAITING_INPUT = "awaiting_input" + COMPACTING = "compacting" + + +def _rx(*patterns: str) -> tuple[re.Pattern[str], ...]: + return tuple(re.compile(p, re.IGNORECASE | re.MULTILINE) for p in patterns) + + +_ANSI = re.compile(r"\x1b(?:\[[0-?]*[ -/]*[@-~]|\][^\x07\x1b]*(?:\x07|\x1b\\)|[@-Z\\-_])") +GIT_OUTPUT_LINE_PATTERNS = _rx( + r"^(commit [0-9a-f]{7,}|Author:|Date:|Merge: [0-9a-f]{6,}|diff --git|index [0-9a-f]+\.\." + r"|@@ |[-+]{3} [ab]/)", +) +_SAVED_TRANSCRIPT_CMD = re.compile( + r"^\s*(?:[$#%]|❯)?\s*(?:cat|bat|less|more|tail|head|sed|awk)\b.*" + r"(?:hud|transcript|terminal|output|copied|\.txt)\b", re.IGNORECASE, +) +_IDLE_PROMPT = re.compile(r"^\s*(?:[│┃║▌▐▏▕╎┆┊]\s*)?[›>❯]\s*") +_IDLE_PROMPT_LINE = re.compile(r"^\s*(?:[│┃║▌▐▏▕╎┆┊]\s*)?[›>❯]\s*$") +_VOLATILE = _rx( + r"[⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏·✻✽✶✳✢]", r"esc to interrupt", r"\(\d+[smh]\b.*?\)", + r"\b\d+\s*tokens?\b", r"\b\d+[smh]\s+elapsed\b", r"\? for shortcuts", +) +_ACTIVE_TASK = _rx( + r"esc to interrupt", r"background terminal running", + r"^[·✻]\s+[A-Za-z][A-Za-z0-9''-]*(?:\s+[A-Za-z][A-Za-z0-9''-]*){0,3}(?:…|\.{3})$", +) + +# Rate-limit vocabulary in three tiers (same table as the contract, split so +# that exit classification can demand corroboration and prose cannot pause): +# banner — unambiguous platform/API banners; sufficient alone, and enough +# to classify a session exit as RATE_LIMITED; +# prose — "rate limit" in running text; enough to pause (never nudge) +# but not, on its own, to classify an exit; +# loose — generic phrases ("limit reached", "try again later", "hit … limit", +# "resets … at", "5-hour") that only count when the SAME line also +# carries rate/usage/quota vocabulary — "patience limit reached at +# epoch 30" and "the 5-hour window in the plan" are not banners. +RATE_LIMIT_BANNER_PATTERNS = _rx( + r"usage limit", r"quota exceeded", r"too many requests", r"hit your limit", + r"\bweekly\s+(?:usage\s+)?(?:limit|quota|cap|allowance|allocation)\b", + r"you(?:'|’)ve\s+(?:hit|reached)\s+(?:your\s+)?(?:session\s+|usage\s+)?limit", + r"\blimit\s+resets?\b", r"stop\s+and\s+wait\s+for\s+limit\s+to\s+reset", + r"\b429\b(?!\d)(?=.*(?:rate|limit|request))", r"rate_limit(?:ed|_error)?", + r"too_many_requests", r"quota_(?:exceeded|limit|exhausted)", +) +RATE_LIMIT_PROSE_PATTERNS = _rx(r"rate limit") +RATE_LIMIT_LOOSE_PATTERNS = _rx( + r"try again later", r"limit reached", r"hit .+ limit", r"resets? .+ at", r"5[- ]?hour", +) +_RATE_LIMIT_CONTEXT = re.compile(r"usage|rate|quota|requests?|session limit|\bapi\b", re.I) +RATE_LIMIT_TEXT_PATTERNS = ( + RATE_LIMIT_BANNER_PATTERNS + RATE_LIMIT_PROSE_PATTERNS + RATE_LIMIT_LOOSE_PATTERNS +) +CONTEXT_LIMIT_PATTERNS = _rx( + r"\b(?:context_limit|context_window|context_exceeded|context_full|max_context|token_limit" + r"|max_tokens|conversation_too_long|input_too_long)\b", + r"context (?:window )?(?:is )?(?:full|low|exceeded)", r"prompt is too long", + r"compact(?:ing)? (?:the )?conversation", +) +AUTH_ERROR_PATTERNS = _rx( + r"\b(?:authentication_error|authentication_failed|auth_error|unauthorized|unauthorised" + r"|forbidden|invalid_token|token_invalid|token_expired|expired_token|oauth_expired" + r"|oauth_token_expired|invalid_grant|insufficient_scope)\b", + r"\b40[13]\b(?=.*(?:unauthori[sz]ed|forbidden|auth))", + r"\bplease (?:run )?/login\b", r"\bnot logged in\b", r"\binvalid api key\b", +) +# "Interrupted by user" / "Interrupted · What should Claude do instead?" are +# rendered by the Claude Code TUI behind a ``⎿`` connector, so the anchor +# tolerates any leading non-word prefix; bare "interrupt" stays excluded. +USER_ABORT_PATTERNS = _rx( + r"(?<!esc to )\b(?:aborted|abort|cancel)\b", r"user_cancel", r"user_interrupt", r"ctrl_c", + r"manual_stop", + r"^\W*interrupted(?:\s+by\s+user\b|\W+what\s+should\s+claude\s+do\s+instead)", +) +# Menu items are anchored to a selection cursor / line start so that ordinary +# numbered lists ("1. Read the file") and indexing ("arr[0]") in Claude's +# output do not read as a dialog (that would silently disable nudging). +AWAITING_INPUT_PATTERNS = _rx( + r"do you want to (?:proceed|allow|continue)", r"^\s*❯\s*\d+\.\s", r"^\s*\[\d+\]\s", + r"esc to cancel", r"yes,? (?:and )?(?:don't|do not) ask again", r"allow (?:once|always)", + r"press enter", r"enter to confirm", r"select an option", r"choice:", + r"waiting for (?:your )?(?:input|response|approval)", + r"trust (?:this|the) (?:folder|workspace)", +) + + +def _any(patterns: tuple[re.Pattern[str], ...], text: str) -> bool: + return any(p.search(text) for p in patterns) + + +def rate_limit_match(text: str) -> str | None: + """Tier of the strongest rate-limit evidence in ``text``: ``"banner"`` | + ``"prose"`` | ``"loose"`` | ``None`` (loose needs same-line context).""" + if _any(RATE_LIMIT_BANNER_PATTERNS, text): + return "banner" + if _any(RATE_LIMIT_PROSE_PATTERNS, text): + return "prose" + for line in text.split("\n"): + if _any(RATE_LIMIT_LOOSE_PATTERNS, line) and _RATE_LIMIT_CONTEXT.search(line): + return "loose" + return None + + +def rate_limit_banner_key(text: str) -> str: + """The rate-limit evidence lines of the tail, joined — identifies *which* + banner is on screen so an unchanged (stale) banner is told apart from a + freshly printed one and from one that has already been resolved.""" + lines = _tail_lines(text, _TAIL_LINES) + hits = [ln for ln in lines if rate_limit_match(ln) is not None] + return "\n".join(hits) + + +_TEXT_TAXONOMY: tuple[tuple[NeverBlockReason, Callable[[str], bool]], ...] = ( + (NeverBlockReason.USER_ABORT, lambda t: _any(USER_ABORT_PATTERNS, t)), + (NeverBlockReason.CONTEXT_LIMIT, lambda t: _any(CONTEXT_LIMIT_PATTERNS, t)), + (NeverBlockReason.RATE_LIMIT, lambda t: rate_limit_match(t) is not None), + (NeverBlockReason.AUTH_ERROR, lambda t: _any(AUTH_ERROR_PATTERNS, t)), + (NeverBlockReason.AWAITING_INPUT, lambda t: _any(AWAITING_INPUT_PATTERNS, t)), +) + + +def normalize_terminal_text(text: str) -> str: + """Strip ANSI/CR; drop git-output lines and saved-transcript command lines.""" + clean = _ANSI.sub("", text or "").replace("\r", "") + kept = [ + line for line in clean.split("\n") + if not any(p.match(line.lstrip()) for p in GIT_OUTPUT_LINE_PATTERNS) + and not _SAVED_TRANSCRIPT_CMD.match(line) + ] + return "\n".join(kept) + + +def _tail_lines(text: str, count: int) -> list[str]: + lines = [ln.rstrip() for ln in normalize_terminal_text(text).split("\n") if ln.strip()] + return lines[-count:] + + +def progress_digest(text: str) -> str: + """sha1 of the normalized text with volatile UI churn (spinners, counters) removed.""" + stable: list[str] = [] + for line in normalize_terminal_text(text).split("\n"): + for pattern in _VOLATILE: + line = pattern.sub("", line) + if line.strip() and not _IDLE_PROMPT_LINE.match(line): + stable.append(line.strip()) + return hashlib.sha1("\n".join(stable).encode("utf-8")).hexdigest() + + +def _any_compacting(heartbeats: Sequence[HeartbeatRecord], now: datetime, window: float) -> bool: + return any( + str(hb.status) == "compacting" and 0 <= _secs(now, hb.last_tick_at) <= window + for hb in heartbeats + ) + + +def classify_never_block( + text: str, *, is_interrupt: bool | None = None, + heartbeats: Sequence[HeartbeatRecord] = (), now: datetime | None = None, + compacting_window_sec: float = 300.0, +) -> NeverBlockReason | None: + """Precedence: interrupt → USER_ABORT; then text (last 60 lines) USER_ABORT > + CONTEXT_LIMIT > RATE_LIMIT > AUTH_ERROR > AWAITING_INPUT; then COMPACTING.""" + if is_interrupt: + return NeverBlockReason.USER_ABORT + tail = "\n".join(_tail_lines(text, _TAIL_LINES)) + for reason, matches in _TEXT_TAXONOMY: + if matches(tail): + return reason + if heartbeats and _any_compacting(heartbeats, now or datetime.now(UTC), compacting_window_sec): + return NeverBlockReason.COMPACTING + return None + + +# --------------------------------------------------- rate-limit reset parsing + +_RESET_IN = re.compile( + r"(?:resets?|try again|retry)\s+in\s+(\d+)\s*(second|sec|s|minute|min|m|hour|hr|h)s?\b", re.I) +_RETRY_AFTER = re.compile(r"retry[- ]after[: ]+(\d+)", re.I) +_RESET_AT = re.compile(r"resets?\s+(?:at\s+)?(\d{1,2})(?::(\d{2}))?\s*(am|pm)?\b", re.I) +_ISO_TS = re.compile( + r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:Z|[+-]\d{2}:?\d{2})?") +_RESET_VOCAB = re.compile(r"reset|retry|limit|again", re.I) +_UNIT_SEC = {"s": 1, "sec": 1, "second": 1, "m": 60, "min": 60, "minute": 60, + "h": 3600, "hr": 3600, "hour": 3600} + + +def _clock_time(match: re.Match[str], now: datetime, local: tzinfo) -> datetime | None: + hour, minute = int(match.group(1)), int(match.group(2) or 0) + ampm = (match.group(3) or "").lower() + if not ampm and match.group(2) is None: + return None # "resets 3" is not a clock time + if ampm == "pm" and hour < 12: + hour += 12 + elif ampm == "am" and hour == 12: + hour = 0 + if hour > 23 or minute > 59: + return None + local_now = _aware(now).astimezone(local) + candidate = datetime.combine(local_now.date(), time(hour, minute), tzinfo=local) + return candidate if candidate > local_now else candidate + timedelta(days=1) + + +def _iso_reset(text: str, now: datetime, local: tzinfo) -> datetime | None: + for line in text.split("\n"): + if not _RESET_VOCAB.search(line): + continue + for match in _ISO_TS.finditer(line): + try: + stamp = datetime.fromisoformat(match.group(0).replace("Z", "+00:00")) + except ValueError: + continue + stamp = stamp if stamp.tzinfo else stamp.replace(tzinfo=local) + if stamp > _aware(now): + return stamp + return None + + +def parse_rate_limit_reset( + text: str, *, now: datetime, tz: tzinfo | None = None, +) -> datetime | None: + """Reset time from banner text ("resets at 3pm", "try again in 5 minutes", + "retry-after: 90", ISO-8601 on a reset/limit line); ``None`` if absent. + + Clock times are interpreted in ``tz`` (else ``now.tzinfo``) — Claude Code + prints the reset in the operator's LOCAL time, so runtime callers must + pass the local zone. Within a tier the LAST occurrence wins: transcripts + grow downward, so the newest banner is the authoritative one. + """ + local = tz or _aware(now).tzinfo or UTC + clean = normalize_terminal_text(text) + if hits := list(_RESET_IN.finditer(clean)): + m = hits[-1] + return _aware(now) + timedelta(seconds=int(m.group(1)) * _UNIT_SEC[m.group(2).lower()]) + if hits := list(_RETRY_AFTER.finditer(clean)): + return _aware(now) + timedelta(seconds=int(hits[-1].group(1))) + stamps = [_clock_time(m, now, local) for m in _RESET_AT.finditer(clean)] + if any(stamps): + return [s for s in stamps if s is not None][-1] + return _iso_reset(clean, now, local) + + +def pane_ready_for_nudge(text: str) -> bool: + """Idle prompt visible AND no active task AND no awaiting-input dialog (last 40 lines).""" + lines = _tail_lines(text, _PANE_TAIL_LINES) + if not lines: + return False + ready = any(_IDLE_PROMPT.match(ln) for ln in lines[-5:]) + tail = "\n".join(lines) + active = any(p.search(tail) for p in _ACTIVE_TASK) + awaiting = any(p.search(tail) for p in AWAITING_INPUT_PATTERNS) + return ready and not active and not awaiting + diff --git a/src/zo/_wrapper_models.py b/src/zo/_wrapper_models.py index 7b42610..46eb9d5 100644 --- a/src/zo/_wrapper_models.py +++ b/src/zo/_wrapper_models.py @@ -21,10 +21,20 @@ class AgentStatus(StrEnum): ERRORED = "errored" RATE_LIMITED = "rate_limited" TIMED_OUT = "timed_out" + # WS-C watchdog states (plan oracle checks 11-12). + PAUSED_RATE_LIMIT = "paused_rate_limit" + STALLED = "stalled" class LeadProcess(BaseModel): - """Tracks the lead orchestrator Claude Code session.""" + """Tracks the lead orchestrator Claude Code session. + + Watchdog fields (WS-C): ``pid_start_identity`` pairs the pid with a + platform-tagged start time so a recycled pid is never mistaken for the + live lead; ``nudges_used`` / ``stalled`` / ``paused_until`` / + ``resume_at`` / ``pause_total_sec`` mirror the runner's decisions so + the CLI can render them without reading the watchdog state file. + """ pid: int | None = None status: AgentStatus = AgentStatus.SPAWNING @@ -35,6 +45,12 @@ class LeadProcess(BaseModel): stdout_log: Path | None = None stderr_log: Path | None = None tmux_pane_id: str | None = None + pid_start_identity: str | None = None + nudges_used: int = 0 + stalled: bool = False + paused_until: datetime | None = None + resume_at: datetime | None = None + pause_total_sec: float = 0.0 model_config = ConfigDict(arbitrary_types_allowed=True) diff --git a/src/zo/_wrapper_watchdog.py b/src/zo/_wrapper_watchdog.py new file mode 100644 index 0000000..04bc6d7 --- /dev/null +++ b/src/zo/_wrapper_watchdog.py @@ -0,0 +1,325 @@ +"""Watchdog runner for the lifecycle wrapper (WS-C, oracle checks 11-12). + +``WatchdogRunner`` is the *external checker* that ``LifecycleWrapper`` ticks +once per poll iteration from both ``_wait_tmux`` and ``_wait_headless``. It +owns the persisted ``WatchdogState``, gathers evidence (heartbeat files, the +captured pane / stdout text, progress-path mtimes, process-tree CPU time, +positive-proof process death), calls the pure ``zo.watchdog.evaluate`` policy +and persists the state plus a one-line JSONL trace per tick. It performs NO +side effects on the session — nudging, pausing, escalating and killing are +the wrapper's job, so that comms, tmux and ``LeadProcess`` stay in one place. + +Fail-open discipline: every advisory path (state persistence, trace writes, +file observation, CPU sampling) swallows errors; the decision path is pure +and never fires on unknown evidence (``process_dead=None`` is "unknown", not +"dead"). +""" + +from __future__ import annotations + +import contextlib +import json +from datetime import UTC, datetime +from typing import TYPE_CHECKING + +from zo.watchdog import ( + HEARTBEATS_DIRNAME, + NeverBlockReason, + StallVerdict, + WatchdogConfig, + WatchdogState, + evaluate, + is_process_dead, + load_all_heartbeats, + load_state, + new_state, + observe_files, + observe_heartbeats, + observe_text, + process_tree_cpu_seconds, + save_state, + sweep_stale_heartbeats, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Sequence + from datetime import tzinfo + from pathlib import Path + + from zo._wrapper_models import LeadProcess + from zo.watchdog import HeartbeatRecord + +__all__ = ["CPU_BUSY_FRACTION", "TICK_TRACE_FILENAME", "WatchdogRunner", "local_tz"] + +TICK_TRACE_FILENAME = "_watchdog-ticks.jsonl" +# The lead's process tree must burn at least this fraction of the wall time +# between two samples for CPU time to count as progress: an idle TUI redraws +# at ~1 %, a training run inside a silent tool call sits at ≥ 100 %. +CPU_BUSY_FRACTION = 0.25 +_CPU_MIN_INTERVAL_SEC = 1.0 + + +def _utc_now() -> datetime: + return datetime.now(UTC) + + +def local_tz() -> tzinfo: + """The operator's local zone (Claude Code prints reset times in local time).""" + return datetime.now().astimezone().tzinfo or UTC + + +class WatchdogRunner: + """Per-run watchdog: evidence gathering + policy call + persistence. + + Args: + config: Resolved ``WatchdogConfig`` for this run. + memory_root: Per-project memory root; heartbeats and the state file + live under ``<memory_root>/heartbeats/``. + zo_session_id: Comms session id (correlation only). + clock: Injectable wall clock returning a tz-aware ``datetime``. + tz: Zone used to interpret banner clock times ("resets at 3pm"); + defaults to the operator's local zone. + progress_paths: Extra files/dirs whose mtime advance counts as + progress (ledger, comms dir, experiments dir, config extras). + dead_probe: Injectable positive-proof death check + ``(pid, recorded_identity) -> bool``; defaults to + :func:`zo.watchdog.is_process_dead`. + cpu_probe: Injectable ``pid -> cumulative CPU seconds of the process + tree`` (``None`` = unknown); defaults to + :func:`zo.watchdog.process_tree_cpu_seconds`. + """ + + def __init__( + self, + *, + config: WatchdogConfig, + memory_root: Path, + zo_session_id: str = "", + clock: Callable[[], datetime] | None = None, + tz: tzinfo | None = None, + progress_paths: Sequence[Path] = (), + dead_probe: Callable[[int | None, str | None], bool] | None = None, + cpu_probe: Callable[[int], float | None] | None = None, + ) -> None: + self.config = config + self.memory_root = memory_root + self.zo_session_id = zo_session_id + self._clock = clock or _utc_now + self.tz: tzinfo = tz or local_tz() + self.progress_paths: list[Path] = list(progress_paths) + self._dead_probe = dead_probe or is_process_dead + self._cpu_probe = cpu_probe or process_tree_cpu_seconds + self._cpu_sample: tuple[float, datetime] | None = None + self.state: WatchdogState = new_state(now=self._clock(), zo_session_id=zo_session_id) + self.last_verdict: StallVerdict | None = None + self.last_new_stall: bool = False + + # ------------------------------------------------------------ lifecycle + + def clock(self) -> datetime: + """Current time from the injected clock (tz-aware).""" + now = self._clock() + return now if now.tzinfo is not None else now.replace(tzinfo=UTC) + + def start(self, now: datetime | None = None) -> None: + """Baseline pre-existing heartbeats, sweep stale files, carry a prior nudge budget. + + Pre-existing heartbeat files are baselined so they never count as + progress; a persisted state for the SAME ``zo_session_id`` carries its + ``nudges_used`` forward so a wrapper restart cannot refill the budget. + """ + now = now or self.clock() + with contextlib.suppress(Exception): + sweep_stale_heartbeats(self.memory_root, now=now) + heartbeats = self._load_heartbeats() + self.state = new_state(now=now, zo_session_id=self.zo_session_id, heartbeats=heartbeats) + prior = load_state(self.memory_root) + if prior is not None and self.zo_session_id and prior.zo_session_id == self.zo_session_id: + self.state.nudges_used = prior.nudges_used + self.state.stall_events = prior.stall_events + with contextlib.suppress(Exception): + observe_files(self.state, self.progress_paths) # baseline mtimes + self.persist() + + def stop(self) -> None: + """Persist final state (fail-open).""" + self.persist() + + def persist(self) -> None: + """Atomically save state; never raises.""" + with contextlib.suppress(Exception): + save_state(self.memory_root, self.state) + + # ------------------------------------------------------------- evidence + + def _load_heartbeats(self) -> list[HeartbeatRecord]: + try: + return load_all_heartbeats(self.memory_root) + except Exception: # unreadable heartbeats are unknown, not stale + return [] + + def _observe_cpu(self, pid: int | None, now: datetime) -> bool: + """True iff the lead's process tree burned ≥ ``CPU_BUSY_FRACTION`` of the + wall time since the previous sample (a busy tree is not stalled).""" + if pid is None: + return False + try: + cpu = self._cpu_probe(pid) + except Exception: + cpu = None + if cpu is None: + self._cpu_sample = None + return False + prev, self._cpu_sample = self._cpu_sample, (cpu, now) + if prev is None: + return False + wall = (now - prev[1]).total_seconds() + return wall >= _CPU_MIN_INTERVAL_SEC and (cpu - prev[0]) >= CPU_BUSY_FRACTION * wall + + def _observe(self, text: str, heartbeats: Sequence[HeartbeatRecord], *, + pid: int | None, now: datetime) -> bool: + """Run every observer (each updates state) and OR their verdicts.""" + progressed = False + with contextlib.suppress(Exception): + progressed = observe_heartbeats(self.state, heartbeats) or progressed + if text.strip(): # an empty capture is unknown, not evidence + with contextlib.suppress(Exception): + progressed = observe_text(self.state, text) or progressed + with contextlib.suppress(Exception): + progressed = observe_files(self.state, self.progress_paths) or progressed + cpu_busy = self._observe_cpu(pid, now) # always sampled, even when already progressed + return progressed or cpu_busy + + def _process_dead(self, process: LeadProcess, override: bool | None) -> bool | None: + if override is not None: + return override + if process.pid is None: + return None # unknown identity is never proof of death + try: + return bool(self._dead_probe(process.pid, process.pid_start_identity)) + except Exception: + return None + + # ----------------------------------------------------------------- tick + + def tick( + self, *, process: LeadProcess, text: str, can_nudge: bool, + now: datetime | None = None, process_dead: bool | None = None, + ) -> StallVerdict: + """Gather evidence, evaluate the policy, persist state + trace, return the verdict. + + Args: + process: The lead process (pid/identity are read, never mutated). + text: Captured pane text (tmux) or the rolling stdout/stderr window. + can_nudge: True only for a tmux pane that is ready for input + (headless and busy/dialog panes pass False). + now: Injected clock value; defaults to ``self.clock()``. + process_dead: Explicit liveness override (headless passes False + while ``Popen.poll()`` is None); ``None`` → probe by pid. + """ + now = now or self.clock() + heartbeats = self._load_heartbeats() + progress = self._observe(text, heartbeats, pid=process.pid, now=now) + stall_events_before = self.state.stall_events + verdict = evaluate( + self.state, self.config, now=now, text=text, heartbeats=heartbeats, + progress=progress, process_dead=self._process_dead(process, process_dead), + can_nudge=can_nudge, tz=self.tz, + ) + self.last_verdict = verdict + self.last_new_stall = self.state.stall_events > stall_events_before + self.persist() + self._trace(verdict) + return verdict + + def settle(self) -> None: + """Re-baseline progress-path mtimes after the wrapper's own writes. + + The comms log dir is a progress path; the wrapper's nudge/escalate + checkpoints land there and must not read back as agent progress on + the next tick. + """ + with contextlib.suppress(Exception): + observe_files(self.state, self.progress_paths) + + def record_nudge(self, now: datetime | None = None, *, resume: bool = False) -> None: + """Account for a nudge that was actually delivered (caller-side counters).""" + now = now or self.clock() + if resume: + self.state.resume_nudges_used += 1 + else: + self.state.nudges_used += 1 + self.state.last_nudge_at = now + self.persist() + + # -------------------------------------------------------------- queries + + def paused_seconds(self, now: datetime | None = None) -> float: + """Total paused seconds so far, including a pause still in effect. + + An open pause stops accruing at an escalation raised during it (resume + unverified / max pause exceeded): the wall-clock timeout must not stay + suspended forever once the watchdog has already handed off to a human. + """ + total = self.state.total_paused_sec + paused_at, esc = self.state.paused_at, self.state.escalated_at + if paused_at is not None: + end = now or self.clock() + if esc is not None and esc >= paused_at: + end = min(end, esc) + total += max(0.0, (end - paused_at).total_seconds()) + return total + + @property + def last_never_block(self) -> str | None: + """Never-block reason recorded by the most recent tick (or ``None``).""" + return self.state.last_never_block + + @property + def is_rate_limited(self) -> bool: + """True iff the most recent tick classified the text as a rate limit.""" + return self.state.last_never_block == NeverBlockReason.RATE_LIMIT.value + + def rate_limit_exit_evidence(self, *, rc: int | None = None) -> bool: + """Should a session that just ended be classified ``RATE_LIMITED``? + + True iff a rate-limit pause is still open (never resumed) AND it is + corroborated beyond "some tick matched": a parsed reset time, an + unambiguous platform banner, or a non-zero exit code. Prose ("the API + hit a rate limit earlier and retried") in a session that finished + normally stays ``COMPLETED``. + """ + if self.state.paused_at is None: + return False + if self.state.pause_evidence in ("reset", "banner"): + return True + return rc is not None and rc != 0 + + def parsed_resume_at(self) -> datetime | None: + """``paused_until`` when it came from a parsed reset time, else ``None``.""" + if self.state.pause_evidence == "reset": + return self.state.paused_until + return None + + def progress_since_escalation(self) -> bool: + """True iff progress was observed after the last escalation.""" + esc = self.state.escalated_at + return esc is None or self.state.last_progress_at > esc + + # ---------------------------------------------------------------- trace + + def _trace(self, verdict: StallVerdict) -> None: + """Append one JSONL line per tick (advisory; never raises).""" + line = { + "ts": verdict.evaluated_at.isoformat(), + "action": verdict.action.value, + "stalled": verdict.stalled, + "reason": verdict.reason, + "never_block": verdict.never_block.value if verdict.never_block else None, + "progress": verdict.progress, + } + path = self.memory_root / HEARTBEATS_DIRNAME / TICK_TRACE_FILENAME + with contextlib.suppress(Exception): + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "a", encoding="utf-8") as fh: + fh.write(json.dumps(line) + "\n") diff --git a/src/zo/cli.py b/src/zo/cli.py index a405711..e55a2f3 100644 --- a/src/zo/cli.py +++ b/src/zo/cli.py @@ -27,7 +27,9 @@ if TYPE_CHECKING: from zo.memory import MemoryManager + from zo.project_config import ProjectConfig from zo.target import TargetConfig + from zo.watchdog import WatchdogConfig console = Console() @@ -85,6 +87,18 @@ def make_target(self) -> TargetConfig: target_path = self.zo_root / "targets" / f"{self.project_name}.target.md" return parse_target(target_path) + def make_project_config(self) -> ProjectConfig | None: + """Load ``.zo/config.yaml`` as a ProjectConfig; None for legacy layouts. + + Legacy (``targets/*.target.md``) projects have no config file, so + callers fall back to hardcoded defaults (e.g. watchdog ON). + """ + if self.layout != "zo-dir": + return None + from zo.project_config import load_project_config + + return load_project_config(self.delivery_repo) + def _detect_delivery_repo(project_name: str | None = None) -> Path | None: """Check cwd for a .zo/config.yaml marker. Return cwd if found.""" @@ -706,6 +720,42 @@ def _generate_session_summary(events: list[str], team_name: str) -> None: console.print() +def _print_session_outcome(process, zo_root: Path) -> None: # noqa: ANN001 + """Print the end-of-session line, with watchdog-specific outcomes first. + + ``STALLED`` (watchdog escalated) and ``RATE_LIMITED`` (session ended + while rate-limited; ``resume_at`` may carry the parsed reset time) get + actionable messages; everything else falls through to the generic + completed / ended-with-status lines. Status is compared by string value + so both ``AgentStatus`` members and plain strings (test doubles) work. + """ + status = str(getattr(process, "status", "") or "") + if status == "stalled": + console.print( + "[red bold]Session stalled[/] — watchdog escalated after its nudge " + f"budget; see [{_DIM}]{zo_root / 'logs' / 'comms'}[/] and " + f"[{_DIM}]<memory_root>/heartbeats/_watchdog-ticks.jsonl[/]." + ) + return + if status == "rate_limited": + resume_at = getattr(process, "resume_at", None) + if resume_at is None: + when = "reset time unknown" + elif hasattr(resume_at, "strftime"): + when = f"resets at {resume_at.strftime('%Y-%m-%d %H:%M %Z').strip()}" + else: + when = f"resets at {resume_at}" + console.print( + f"[red bold]Session ended rate-limited[/] — {when}; " + "rerun [bold]zo continue[/] after the limit resets." + ) + return + if status == "completed": + console.print("[green bold]Session completed.[/]") + else: + console.print(f"[red bold]Session ended with status:[/] {status}") + + def _launch_and_monitor( *, wrapper, # noqa: ANN001 @@ -728,6 +778,9 @@ def _launch_and_monitor( surrogate_id: str | None = None, surrogate_worktree: Path | None = None, consolidate_on_exit: bool = True, + watchdog: WatchdogConfig | None = None, + memory_root: Path | None = None, + zo_session_id: str = "", ) -> None: """Shared launch → monitor → end-session flow for build and draft. @@ -743,6 +796,13 @@ def _launch_and_monitor( bypass_permissions: When True, Claude Code tool-call permission prompts are auto-approved. Set by ``--bypass-permissions`` or implied by ``--gate-mode full-auto``. + watchdog: Resolved anti-stall policy (WS-C). ``None`` or + ``enabled=False`` → the wrapper runs its poll loop exactly as + before, with no watchdog tick. + memory_root: Per-project memory root; the watchdog persists its + state and reads hook heartbeats under ``<memory_root>/heartbeats/``. + zo_session_id: Comms session id, forwarded so watchdog state and + heartbeats correlate with the comms log. """ # Surrogate liveness registry: detect concurrent sessions on this project so # we neither disturb a live peer's permission overlay nor consolidate @@ -920,13 +980,11 @@ def _print_status(team_status, pane_snapshot=""): # noqa: ANN001 process = wrapper.wait_for_completion( process, on_status=_print_status, gate_mode_file=gate_mode_file, project_name=project_name, delivery_repo=delivery_repo, + watchdog=watchdog, memory_root=memory_root, zo_session_id=zo_session_id, ) console.print() - if process.status == "completed": - console.print("[green bold]Session completed.[/]") - else: - console.print(f"[red bold]Session ended with status:[/] {process.status}") + _print_session_outcome(process, zo_root) # Generate a Haiku-summarised 2-3 bullet wrap-up from buffered # events. This is the only Haiku call ZO makes during a run @@ -994,6 +1052,11 @@ def _print_status(team_status, pane_snapshot=""): # noqa: ANN001 help="Auto-approve Claude Code tool-call permission prompts. Useful when " "you want to walk away from the terminal. Implied by --gate-mode full-auto.", ) +@click.option( + "--no-watchdog", is_flag=True, + help="Disable the anti-stall watchdog for this run (no stall detection, " + "nudges, or rate-limit pause). Equivalent to ZO_WATCHDOG=0.", +) def build( plan_path: Path, gate_mode: str | None, @@ -1003,6 +1066,7 @@ def build( max_iterations: int | None, no_headlines: bool, bypass_permissions: bool, + no_watchdog: bool = False, ) -> None: """Launch a project from a plan.md file. @@ -1077,6 +1141,11 @@ def build( extra_env["ZO_DELIVERY_ROOT"] = str(target.target_repo) extra_env["ZO_CONTRACTS_PATH"] = str(memory.memory_root / "contracts.json") + # 2a. Watchdog policy (WS-C): project config block (None for legacy + # layouts → defaults) + env overrides; --no-watchdog is a single- + # concern kill switch for this run. + wd_cfg = _resolve_watchdog(ctx, no_watchdog=no_watchdog) + # 3. Detect mode from state state_check = memory.read_state() detected_mode = "build" if state_check.phase == "init" else "continue" @@ -1096,6 +1165,9 @@ def build( log_dir=zo_root / "logs" / "comms", project=project_name, session_id=session_id, ) + # Heartbeat ↔ comms correlation: the hookkit heartbeat writer stamps + # this id into <memory_root>/heartbeats/*.json (WS-C). + extra_env["ZO_SESSION_ID"] = session_id db_path = memory.memory_root / "index.db" semantic = SemanticIndex(db_path=db_path) @@ -1163,9 +1235,28 @@ def build( extra_env=extra_env, headlines_disabled=effective_headlines_disabled, bypass_permissions=effective_bypass_permissions, + watchdog=wd_cfg, + memory_root=memory.memory_root, + zo_session_id=session_id, ) +def _resolve_watchdog(ctx: ProjectContext, *, no_watchdog: bool) -> WatchdogConfig: + """Project ``watchdog:`` block (defaults for legacy) + env + ``--no-watchdog``. + + Precedence: CLI kill switch > env (``ZO_WATCHDOG``, ``ZO_WATCHDOG_STALL_SEC``) + > ``.zo/config.yaml`` > ``WatchdogConfig`` defaults. (``make_target`` has + already validated the same file, so a malformed block fails loudly there.) + """ + from zo.watchdog import resolve_watchdog_config + + pcfg = ctx.make_project_config() + wd_cfg = resolve_watchdog_config(pcfg.watchdog if pcfg is not None else None) + if no_watchdog: + wd_cfg = wd_cfg.model_copy(update={"enabled": False}) + return wd_cfg + + @cli.command("continue") @click.argument("project_name", required=False, default=None) @click.option( @@ -1203,6 +1294,10 @@ def build( help="Auto-approve Claude Code tool-call permission prompts. " "Implied by --gate-mode full-auto.", ) +@click.option( + "--no-watchdog", is_flag=True, + help="Disable the anti-stall watchdog for this run (see `zo build --help`).", +) def continue_( project_name: str | None, repo: str | None, @@ -1213,6 +1308,7 @@ def continue_( max_iterations: int | None, no_headlines: bool, bypass_permissions: bool, + no_watchdog: bool = False, ) -> None: """Resume a paused project or reconnect on a new machine. @@ -1279,6 +1375,7 @@ def continue_( max_iterations=max_iterations, no_headlines=no_headlines, bypass_permissions=bypass_permissions, + no_watchdog=no_watchdog, ) @@ -2133,7 +2230,11 @@ def migrate(project_name: str, repo: str | None, clean: bool) -> None: "local.yaml\n\n" "# SQLite databases (regenerated from DECISION_LOG)\n" "memory/index.db\n" - "memory/draft_index.db\n", + "memory/draft_index.db\n\n" + "# Control-plane runtime files (regenerated per run; never delivery history)\n" + "memory/heartbeats/\n" + "memory/plan-ledger.json\n" + "memory/contracts.json\n", encoding="utf-8", ) diff --git a/src/zo/hookkit.py b/src/zo/hookkit.py index a574e3c..6826b50 100644 --- a/src/zo/hookkit.py +++ b/src/zo/hookkit.py @@ -11,12 +11,17 @@ session-end ensure a session summary exists for today post-tool-failure append a structured failure record (JSONL feed) sealed-paths deny Write/Edit into sealed or off-limits paths + heartbeat stamp <memory_root>/heartbeats/<agent_key>.json (WS-C) Every handler is fail-open: infrastructure problems (missing files, unparseable stdin, unknown agent) exit 0 with no output. Only genuine violations produce blocking JSON on stdout. This mirrors the existing ``.claude/hooks/*.sh`` convention — the enforcement plane must never brick a session. + +The ``heartbeat`` path is stdlib-only (no pydantic import) because it runs +on every PostToolUse — the writer lives in ``zo._hook_heartbeat``; +``zo.contracts`` is imported lazily by the handlers that need it. """ from __future__ import annotations @@ -31,7 +36,8 @@ from datetime import UTC, datetime from pathlib import Path -from zo.contracts import CONTRACTS_FILENAME, load_contracts, validate_agent_stop +from zo._hook_heartbeat import HEARTBEATS_DIRNAME, stamp_heartbeat +from zo._hook_heartbeat import agent_identity as _agent_identity __all__ = ["main"] @@ -44,9 +50,13 @@ _STUB_MARKER = re.compile( r"^\+.*(\bTODO\b|\bFIXME\b|\bXXX\b|NotImplementedError|raise NotImplemented\b)" ) +# Literal "contracts.json" (== zo.contracts.CONTRACTS_FILENAME) keeps this +# module free of the pydantic import; "heartbeats" seals the whole subtree +# via the prefix match in _handle_sealed_paths (agents cannot forge liveness). +_CONTRACTS_FILENAME = "contracts.json" _SEALED_DEFAULTS = ( - "gate_mode", "gate_nonce", "gate_decision", CONTRACTS_FILENAME, - "plan-ledger.json", "sealed_paths", + "gate_mode", "gate_nonce", "gate_decision", _CONTRACTS_FILENAME, + "plan-ledger.json", "sealed_paths", HEARTBEATS_DIRNAME, ) @@ -117,12 +127,35 @@ def _agent_name(data: dict) -> str | None: return None +def _stamp_heartbeat(data: dict, fallback_event: str) -> None: + """Fail-open heartbeat stamp for handlers wired to non-PostToolUse events. + + ``fallback_event`` supplies ``hook_event_name`` when the payload lacks + it (older payload shapes / tests) without mutating ``data`` — the trace + line records the payload's real ``stdin_keys``. + """ + with contextlib.suppress(Exception): + payload = data if data.get("hook_event_name") else { + **data, "hook_event_name": fallback_event, + } + _handle_heartbeat(payload) + + def _handle_subagent_stop(data: dict) -> None: + try: + _subagent_stop(data) + finally: + _stamp_heartbeat(data, "SubagentStop") + + +def _subagent_stop(data: dict) -> None: if data.get("stop_hook_active"): return agent = _agent_name(data) if agent is None: return + from zo.contracts import validate_agent_stop + repo_root = _repo_root() contracts_env = os.environ.get("ZO_CONTRACTS_PATH") if contracts_env: @@ -131,7 +164,7 @@ def _handle_subagent_stop(data: dict) -> None: memory_root = _memory_root(repo_root) if memory_root is None: return - contracts_path = memory_root / CONTRACTS_FILENAME + contracts_path = memory_root / _CONTRACTS_FILENAME delivery_root = Path(os.environ.get("ZO_DELIVERY_ROOT", str(repo_root))) violations = validate_agent_stop(contracts_path, agent, delivery_root) if not violations: @@ -194,6 +227,13 @@ def _added_stub_lines(repo_root: Path) -> list[str]: def _handle_drift_guard(data: dict) -> None: + try: + _drift_guard(data) + finally: + _stamp_heartbeat(data, "Stop") + + +def _drift_guard(data: dict) -> None: if os.environ.get("ZO_DRIFT_GUARD", "1") == "0" or data.get("stop_hook_active"): return # Live Stop payloads carry the last message directly (verified in the @@ -231,6 +271,13 @@ def _memory_manager(memory_root: Path): def _handle_precompact(data: dict) -> None: + try: + _precompact(data) + finally: + _stamp_heartbeat(data, "PreCompact") + + +def _precompact(data: dict) -> None: repo_root = _repo_root() memory_root = _memory_root(repo_root) if memory_root is None or not (memory_root / "STATE.md").exists(): @@ -255,6 +302,13 @@ def _handle_precompact(data: dict) -> None: def _handle_session_end(data: dict) -> None: + try: + _session_end(data) + finally: + _stamp_heartbeat(data, "SessionEnd") + + +def _session_end(data: dict) -> None: repo_root = _repo_root() memory_root = _memory_root(repo_root) if memory_root is None or not (memory_root / "STATE.md").exists(): @@ -289,6 +343,7 @@ def _handle_post_tool_failure(data: dict) -> None: feed_dir = Path( os.environ.get("ZO_FAILURE_FEED_DIR", str(repo_root / "logs" / "comms")) ) + agent_type, agent_id = _agent_identity(data) try: feed_dir.mkdir(parents=True, exist_ok=True) record = { @@ -299,6 +354,10 @@ def _handle_post_tool_failure(data: dict) -> None: "tool_name": data.get("tool_name", "unknown"), "error": str(data.get("error", data.get("tool_response", "")))[:2000], "input_preview": json.dumps(data.get("tool_input", {}))[:500], + # WS-C: user-abort evidence for the watchdog taxonomy + identity. + "is_interrupt": data.get("is_interrupt"), + "agent_id": agent_id, + "agent_type": agent_type, } date = datetime.now(UTC).strftime("%Y-%m-%d") path = feed_dir / f"failures-{date}.jsonl" @@ -350,7 +409,9 @@ def _handle_sealed_paths(data: dict) -> None: if deny_reason is None: agent = _agent_name(data) if agent is not None and memory_root is not None: - doc = load_contracts(memory_root / CONTRACTS_FILENAME) + from zo.contracts import load_contracts + + doc = load_contracts(memory_root / _CONTRACTS_FILENAME) if doc is not None: normalized = agent.strip().lower().replace(" ", "-") for entry in doc.agents: @@ -379,6 +440,22 @@ def _handle_sealed_paths(data: dict) -> None: ) +# -- heartbeat (WS-C) --------------------------------------------------------- + + +def _handle_heartbeat(data: dict) -> None: + """Stamp ``<memory_root>/heartbeats/<agent_key>.json`` (advisory, fail-open). + + The writer lives in ``zo._hook_heartbeat`` (stdlib-only by design — it + runs on every PostToolUse and must not import pydantic). This handler + only resolves the memory root and never creates one. + """ + memory_root = _memory_root(_repo_root()) + if memory_root is None or not memory_root.is_dir(): + return + stamp_heartbeat(data, memory_root=memory_root) + + _HANDLERS = { "subagent-stop": _handle_subagent_stop, "drift-guard": _handle_drift_guard, @@ -386,6 +463,7 @@ def _handle_sealed_paths(data: dict) -> None: "session-end": _handle_session_end, "post-tool-failure": _handle_post_tool_failure, "sealed-paths": _handle_sealed_paths, + "heartbeat": _handle_heartbeat, } diff --git a/src/zo/project_config.py b/src/zo/project_config.py index 7a05c81..f3b73d3 100644 --- a/src/zo/project_config.py +++ b/src/zo/project_config.py @@ -18,7 +18,9 @@ from pathlib import Path # noqa: TC003 — used at runtime import yaml -from pydantic import BaseModel +from pydantic import BaseModel, ConfigDict, Field + +from zo.watchdog import WatchdogConfig # --------------------------------------------------------------------------- # Models @@ -38,8 +40,16 @@ class ProjectConfig(BaseModel): git_author_name: Name used in commits from ZO agents. git_author_email: Email used in commits from ZO agents. enforce_isolation: When True, writes to blocked paths halt execution. + watchdog: Anti-stall policy for the lead session (WS-C, specs/watchdog.md). + Nested block; absent in legacy configs → defaults (watchdog ON). """ + # Documented choice (WS-C, PR-A): unknown top-level keys are IGNORED, not + # forbidden — legacy `.zo/config.yaml` files may carry keys from older or + # newer ZO versions and must still load. The nested ``watchdog`` block is + # strict (``WatchdogConfig`` forbids extras) so policy typos are caught. + model_config = ConfigDict(extra="ignore") + project_name: str alias: str = "" workflow_mode: str = "classical_ml" @@ -49,6 +59,7 @@ class ProjectConfig(BaseModel): git_author_name: str = "ZO Agent" git_author_email: str = "zo-agent@zero-operators.dev" enforce_isolation: bool = True + watchdog: WatchdogConfig = Field(default_factory=WatchdogConfig) class LocalConfig(BaseModel): diff --git a/src/zo/scaffold.py b/src/zo/scaffold.py index 64947d0..54d7ca6 100644 --- a/src/zo/scaffold.py +++ b/src/zo/scaffold.py @@ -336,6 +336,11 @@ # SQLite databases (regenerated from DECISION_LOG) memory/index.db memory/draft_index.db + +# Control-plane runtime files (regenerated per run; never delivery history) +memory/heartbeats/ +memory/plan-ledger.json +memory/contracts.json """ # Files that are platform-independent (always written verbatim). diff --git a/src/zo/watchdog.py b/src/zo/watchdog.py new file mode 100644 index 0000000..7940562 --- /dev/null +++ b/src/zo/watchdog.py @@ -0,0 +1,451 @@ +"""Watchdog — WS-C execution substrate (plan oracle checks 11-12). + +Pure logic: no I/O in the classifier and predicate paths; process identity +and file helpers are small, injectable, and fail-open. + +Pattern tables live in ``zo._watchdog_text``, models/config in +``zo._watchdog_models`` and process identity in ``zo._proc``; all are +re-exported here. Ported from oh-my-claudecode (MIT License, Copyright (c) +2025 Yeachan Heo): ``todo-continuation/index.ts``, +``rate-limit-wait/tmux-detector.ts``, ``team/idle-nudge.ts`` (nudge defaults), +``team/tmux-session.ts``, ``team/team-owner-epoch.ts``. Contract adjustments: +no bare ``429``/``overloaded``, ``awaiting_input`` added, bare ``interrupt`` +excluded (OMC #2478). +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from typing import TYPE_CHECKING + +from zo._proc import ( + identities_may_match, + is_process_dead, + is_valid_process_start_identity, + pid_alive, + process_start_identity, + process_tree_cpu_seconds, +) +from zo._watchdog_models import ( + SCHEMA_VERSION, + Freshness, + HeartbeatRecord, + HeartbeatStatus, + StallAction, + StallVerdict, + WatchdogConfig, + WatchdogState, + new_state, + resolve_watchdog_config, +) +from zo._watchdog_text import ( + AUTH_ERROR_PATTERNS, + AWAITING_INPUT_PATTERNS, + CONTEXT_LIMIT_PATTERNS, + GIT_OUTPUT_LINE_PATTERNS, + RATE_LIMIT_TEXT_PATTERNS, + USER_ABORT_PATTERNS, + NeverBlockReason, + _aware, + _secs, + classify_never_block, + normalize_terminal_text, + pane_ready_for_nudge, + parse_rate_limit_reset, + progress_digest, + rate_limit_banner_key, + rate_limit_match, +) +from zo.ledger import _atomic_write + +if TYPE_CHECKING: + from collections.abc import Sequence + from datetime import tzinfo + from pathlib import Path + +__all__ = [ + "AUTH_ERROR_PATTERNS", "AWAITING_INPUT_PATTERNS", "CONTEXT_LIMIT_PATTERNS", "Freshness", + "GIT_OUTPUT_LINE_PATTERNS", "HEARTBEATS_DIRNAME", "HEARTBEAT_STALE_SWEEP_SEC", + "HeartbeatRecord", "HeartbeatStatus", "NeverBlockReason", "RATE_LIMIT_TEXT_PATTERNS", + "SCHEMA_VERSION", "StallAction", "StallVerdict", "USER_ABORT_PATTERNS", + "WATCHDOG_STATE_FILENAME", "WatchdogConfig", "WatchdogState", "classify_freshness", + "classify_never_block", "compute_pause_until", "evaluate", "heartbeat_path", + "identities_may_match", "is_process_dead", "is_valid_process_start_identity", + "load_all_heartbeats", "load_heartbeat", "load_state", "new_state", "normalize_terminal_text", + "observe_files", "observe_heartbeats", "observe_text", "pane_ready_for_nudge", + "parse_rate_limit_reset", "pid_alive", "process_start_identity", "process_tree_cpu_seconds", + "progress_digest", "rate_limit_banner_key", "rate_limit_match", "resolve_watchdog_config", + "save_state", "sweep_stale_heartbeats", "write_heartbeat", +] + +HEARTBEATS_DIRNAME = "heartbeats" +WATCHDOG_STATE_FILENAME = "_watchdog.json" +HEARTBEAT_STALE_SWEEP_SEC = 24 * 3600 +_RESET_SLACK_SEC = 15 + + +# ---------------------------------------------------------------- heartbeats + +def heartbeat_path(memory_root: Path, agent_key: str) -> Path: + """``<memory_root>/heartbeats/<agent_key>.json``.""" + return memory_root / HEARTBEATS_DIRNAME / f"{agent_key}.json" + + +def load_heartbeat(path: Path) -> HeartbeatRecord | None: + """Parse one heartbeat file; ``None`` on any error (fail-open).""" + try: + return HeartbeatRecord.model_validate_json(path.read_text(encoding="utf-8")) + except Exception: # unreadable/malformed heartbeats are unknown, not stale + return None + + +def _heartbeat_files(memory_root: Path) -> list[Path]: + hb_dir = memory_root / HEARTBEATS_DIRNAME + try: + return sorted(p for p in hb_dir.glob("*.json") if not p.name.startswith("_")) + except OSError: + return [] + + +def load_all_heartbeats(memory_root: Path) -> list[HeartbeatRecord]: + """All parsable heartbeats; ignores ``_watchdog.json`` and unparsable files.""" + records = (load_heartbeat(p) for p in _heartbeat_files(memory_root)) + return [r for r in records if r is not None] + + +def write_heartbeat(memory_root: Path, record: HeartbeatRecord) -> Path: + """Atomically write ``record`` (tmp + ``os.replace``; ``mkdir -p``).""" + path = heartbeat_path(memory_root, record.agent_key) + _atomic_write(path, record.model_dump_json(indent=2)) + return path + + +def sweep_stale_heartbeats( + memory_root: Path, *, now: datetime, older_than_sec: int = HEARTBEAT_STALE_SWEEP_SEC, +) -> int: + """Delete heartbeat files older than ``older_than_sec``; returns the count removed.""" + removed = 0 + for path in _heartbeat_files(memory_root): + record = load_heartbeat(path) + try: + stamp = record.last_tick_at if record else datetime.fromtimestamp( + path.stat().st_mtime, tz=UTC) + if _secs(now, stamp) > older_than_sec: + path.unlink() + removed += 1 + except OSError: + continue + return removed + + +def classify_freshness( + record: HeartbeatRecord | None, *, now: datetime, stale_after_sec: float, +) -> Freshness: + """``None`` → UNKNOWN; naive datetimes are treated as UTC.""" + if record is None: + return Freshness.UNKNOWN + return Freshness.FRESH if _secs(now, record.last_tick_at) < stale_after_sec else Freshness.STALE + + +def compute_pause_until( + now: datetime, reset_at: datetime | None, *, attempt: int, config: WatchdogConfig, +) -> datetime: + """``reset_at`` + 15 s slack, else exponential backoff capped by config.""" + if reset_at is not None: + return _aware(reset_at) + timedelta(seconds=_RESET_SLACK_SEC) + backoff = config.rate_limit_backoff_base_sec * (2 ** max(0, min(attempt, 30))) + return _aware(now) + timedelta(seconds=min(backoff, config.rate_limit_backoff_max_sec)) + + +def load_state(memory_root: Path) -> WatchdogState | None: + """Load persisted state; ``None`` on any error (fail-open).""" + path = memory_root / HEARTBEATS_DIRNAME / WATCHDOG_STATE_FILENAME + try: + return WatchdogState.model_validate_json(path.read_text("utf-8")) + except Exception: + return None + + +def save_state(memory_root: Path, state: WatchdogState) -> Path: + """Atomically persist ``state``; returns the path written.""" + path = memory_root / HEARTBEATS_DIRNAME / WATCHDOG_STATE_FILENAME + _atomic_write(path, state.model_dump_json(indent=2)) + return path + + +# ------------------------------------------------------- evidence observers + +def observe_heartbeats(state: WatchdogState, heartbeats: Sequence[HeartbeatRecord]) -> bool: + """True iff any key's ``tick_count`` advanced past what was seen (and its baseline).""" + progressed = False + for hb in heartbeats: + seen = max(state.seen_ticks.get(hb.agent_key, 0), state.baseline_ticks.get(hb.agent_key, 0)) + if hb.tick_count > seen: + progressed = True + state.seen_ticks[hb.agent_key] = max(seen, hb.tick_count) + return progressed + + +def observe_text(state: WatchdogState, text: str) -> bool: + """True iff the progress digest changed (first observation → False). + + While a rate-limit pause is active a digest change is recorded but NOT + reported as progress: the banner appearing/disappearing churns the text, + and a resume must be verified by heartbeat or file evidence instead. + """ + digest = progress_digest(text) + changed = state.last_digest is not None and digest != state.last_digest + state.last_digest = digest + return changed and state.paused_at is None + + +def _newest_mtime(path: Path) -> float | None: + try: + stat = path.stat() + if not path.is_dir(): + return stat.st_mtime + return max([stat.st_mtime, *(p.stat().st_mtime for p in path.iterdir())]) + except OSError: + return None + + +def observe_files(state: WatchdogState, paths: Sequence[Path]) -> bool: + """True iff any path's mtime advanced or a previously-missing path appeared. + + Directories use the newest mtime among the dir and its direct entries. + Missing paths are recorded as ``-1.0`` so their later appearance counts. + """ + progressed = False + for path in paths: + key = str(path) + mtime = _newest_mtime(path) + prev = state.last_file_mtimes.get(key) + current = -1.0 if mtime is None else mtime + if prev is not None and current > prev: + progressed = True + state.last_file_mtimes[key] = max(prev if prev is not None else current, current) + return progressed + + +# ------------------------------------------------------------ stall policy + +def _freshness(heartbeats: Sequence[HeartbeatRecord], now: datetime, stale: float) -> Freshness: + verdicts = {classify_freshness(hb, now=now, stale_after_sec=stale) for hb in heartbeats} + if not verdicts: + return Freshness.UNKNOWN + return Freshness.FRESH if Freshness.FRESH in verdicts else Freshness.STALE + + +def _already_escalated(state: WatchdogState, since: datetime | None) -> bool: + return (state.escalated_at is not None and since is not None + and _secs(state.escalated_at, since) >= 0) + + +def _escalate_once(state: WatchdogState, now: datetime, since: datetime | None, + reason: str) -> tuple[StallAction, str]: + if _already_escalated(state, since): + return StallAction.NONE, f"already escalated: {reason}" + state.escalated_at = now + return StallAction.ESCALATE, reason + + +def _mark_stall(state: WatchdogState, now: datetime) -> None: + if state.stall_since is None: + state.stall_since = now + state.stall_events += 1 + + +def _pause_end_for_accounting(state: WatchdogState, now: datetime) -> datetime: + """Paused time stops accruing at an escalation raised during the pause.""" + esc = state.escalated_at + if state.paused_at is not None and esc is not None and _secs(esc, state.paused_at) >= 0: + return min(now, esc) + return now + + +def _end_pause(state: WatchdogState, now: datetime, *, spent_banner: str | None) -> None: + if state.paused_at is not None: + end = _pause_end_for_accounting(state, now) + state.total_paused_sec += max(0.0, _secs(end, state.paused_at)) + state.paused_at = state.paused_until = None + state.paused_reason = state.pause_banner_key = state.pause_evidence = None + state.spent_banner_key = spent_banner + state.resume_nudges_used = 0 + + +def _pause_target(state: WatchdogState, config: WatchdogConfig, now: datetime, text: str, + tz: tzinfo | None, *, attempt: int, previous: datetime | None) -> datetime: + """``paused_until`` for a (re)entered pause; records the evidence tier. + + A parsed reset is honoured only if it lies in ``(now, now + max_pause]``: + a stale clock time that rolled over to tomorrow ("resets at 3pm" re-read + after 3pm) or a reset behind ``previous`` is not fresh information, so the + exponential backoff applies instead. + """ + reset = parse_rate_limit_reset(text, now=now, tz=tz) + usable = (reset is not None and 0 < _secs(reset, now) <= config.rate_limit_max_pause_sec + and (previous is None or _secs(reset, previous) > 0)) + state.pause_evidence = "reset" if usable else rate_limit_match(text) + state.pause_banner_key = rate_limit_banner_key(text) + return compute_pause_until(now, reset if usable else None, attempt=attempt, config=config) + + +def _rate_limited(state: WatchdogState, config: WatchdogConfig, now: datetime, text: str, + tz: tzinfo | None, can_nudge: bool) -> tuple[StallAction, str]: + """Step 3: enter/extend a pause; escalate once past the max pause. + + A banner that is byte-for-byte the one the pause was entered on is STALE + once ``paused_until`` has passed (a static transcript never clears the + line): it is treated as gone → resume nudge / headless wait, never a + day-rollover extension. + """ + if state.paused_at is None: + state.paused_at, state.paused_reason, state.pause_attempts = now, "rate_limit", 1 + state.paused_until = _pause_target(state, config, now, text, tz, attempt=0, previous=None) + return StallAction.PAUSE, f"rate limit detected; paused until {state.paused_until:%H:%M:%S}" + if _secs(now, state.paused_at) > config.rate_limit_max_pause_sec: + return _escalate_once(state, now, state.paused_at, "rate-limit pause exceeded max") + if state.paused_until is not None and _secs(now, state.paused_until) >= 0: + if rate_limit_banner_key(text) == state.pause_banner_key: + return _paused_banner_gone(state, config, now, can_nudge, stale=True) + state.pause_attempts += 1 + state.paused_until = _pause_target( + state, config, now, text, tz, attempt=state.pause_attempts - 1, + previous=state.paused_until) + return StallAction.PAUSE, f"rate limit persists; pause extended to {state.paused_until}" + return StallAction.PAUSE, f"rate-limit pause in effect until {state.paused_until}" + + +def _paused_banner_gone(state: WatchdogState, config: WatchdogConfig, now: datetime, + can_nudge: bool, *, stale: bool = False) -> tuple[StallAction, str]: + """Step 4: banner gone (or stale), no progress yet — resume-nudge (tmux) or + wait/escalate (headless / pane busy).""" + what = "banner stale" if stale else "banner gone" + until = state.paused_until or state.paused_at or now + if _secs(now, until) < 0: + return StallAction.NONE, f"paused ({what}) until {until}" + if can_nudge and state.resume_nudges_used < config.resume_nudge_budget: + return StallAction.RESUME_NUDGE, f"pause elapsed and {what}; resume nudge" + dwell_ok = (state.last_nudge_at is None + or _secs(now, state.last_nudge_at) >= config.nudge_delay_sec) + if dwell_ok and _secs(now, until) >= config.rate_limit_backoff_base_sec: + return _escalate_once(state, now, state.paused_at, + "no progress after rate-limit pause; resume unverified") + return StallAction.NONE, "pause elapsed; waiting for verified resume progress" + + +def _other_never_block(state: WatchdogState, config: WatchdogConfig, now: datetime, + reason: NeverBlockReason) -> tuple[StallAction, str]: + """Step 5: never nudge; compaction is progress; auth/context may escalate after threshold.""" + if reason == NeverBlockReason.COMPACTING: + state.last_progress_at, state.stall_since = now, None + return StallAction.NONE, "compacting: stall clock reset" + stalled = _secs(now, state.last_progress_at) >= config.stall_threshold_sec + if stalled and reason in (NeverBlockReason.AUTH_ERROR, NeverBlockReason.CONTEXT_LIMIT): + _mark_stall(state, now) + return _escalate_once(state, now, state.stall_since, + f"stalled under {reason.value}; nudging cannot help") + return StallAction.NONE, f"never-block ({reason.value}): not nudging" + + +def _dead_action(state: WatchdogState, now: datetime) -> tuple[StallAction, str]: + """Positive proof of death escalates ONCE per run (the pid cannot come back).""" + _mark_stall(state, now) + if state.dead_escalated_at is not None: + return StallAction.NONE, "already escalated: lead process is dead" + state.dead_escalated_at = state.escalated_at = now + return StallAction.ESCALATE, "lead process is dead (positive proof)" + + +def _stalled_action(state: WatchdogState, config: WatchdogConfig, now: datetime, *, + can_nudge: bool) -> tuple[StallAction, str]: + """Step 8: nudge with dwell/budget, else escalate once.""" + _mark_stall(state, now) + since = state.stall_since or now + dwell_ok = (state.last_nudge_at is None + or _secs(now, state.last_nudge_at) >= config.nudge_delay_sec) + first_ok = _secs(now, since) >= config.nudge_delay_sec or state.nudges_used > 0 + if can_nudge and state.nudges_used < config.nudge_budget and dwell_ok and first_ok: + return StallAction.NUDGE, ( + f"no progress for {int(_secs(now, state.last_progress_at))}s; " + f"nudge {state.nudges_used + 1}/{config.nudge_budget}") + if can_nudge and state.nudges_used >= config.nudge_budget and dwell_ok: + return _escalate_once(state, now, since, "nudge budget exhausted without progress") + if not can_nudge and _secs(now, since) >= config.escalate_grace_sec: + return _escalate_once(state, now, since, + "stalled and nudging impossible (headless or busy pane)") + return StallAction.NONE, "stalled; waiting on nudge dwell" + + +def _classify(state: WatchdogState, text: str, *, is_interrupt: bool | None, + heartbeats: Sequence[HeartbeatRecord], now: datetime) -> NeverBlockReason | None: + """Step 2, minus a banner already resolved by a verified resume (still on screen).""" + reason = classify_never_block(text, is_interrupt=is_interrupt, heartbeats=heartbeats, now=now) + if (reason == NeverBlockReason.RATE_LIMIT and state.paused_at is None + and state.spent_banner_key and rate_limit_banner_key(text) == state.spent_banner_key): + return None + if reason != NeverBlockReason.RATE_LIMIT: + state.spent_banner_key = None # the resolved banner has scrolled away + return reason + + +def _progress(state: WatchdogState, now: datetime, text: str, + reason: NeverBlockReason | None) -> StallVerdict | None: + """Step 1: progress resets the stall clock; verified resume once the pause elapsed.""" + state.last_progress_at, state.stall_since = now, None + if state.paused_at is None: + return None + until = state.paused_until or state.paused_at + if reason == NeverBlockReason.RATE_LIMIT and _secs(now, until) < 0: + return None # banner fresh and reset not reached: stay paused + spent = rate_limit_banner_key(text) if reason == NeverBlockReason.RATE_LIMIT else None + _end_pause(state, now, spent_banner=spent or None) + return StallVerdict(action=StallAction.RESUME, stalled=False, evaluated_at=now, + reason="progress observed after rate-limit pause; resumed") + + +def evaluate( + state: WatchdogState, config: WatchdogConfig, *, now: datetime, text: str, + heartbeats: Sequence[HeartbeatRecord], progress: bool, process_dead: bool | None, + is_interrupt: bool | None = None, can_nudge: bool, tz: tzinfo | None = None, +) -> StallVerdict: + """The whole stall/nudge/pause decision policy for one tick (pure, clock-injected). + + ``tz`` is the operator's local zone for banner clock times ("resets at + 3pm"); ``None`` falls back to ``now.tzinfo``. ``can_nudge`` is False for + headless AND for a tmux pane that is busy / showing a dialog, so a stall + that cannot be nudged still escalates after ``escalate_grace_sec``. + """ + now, can_nudge = _aware(now), can_nudge and config.nudge_enabled + state.last_tick_at, state.ticks = now, state.ticks + 1 + reason = _classify(state, text, is_interrupt=is_interrupt, heartbeats=heartbeats, now=now) + state.last_never_block = reason.value if reason else None + base = { + "never_block": reason, "process_dead": process_dead, "progress": progress, + "freshness": _freshness(heartbeats, now, config.stall_threshold_sec), + } + if progress and (resumed := _progress(state, now, text, reason)) is not None: + return resumed.model_copy(update=base) + if reason == NeverBlockReason.RATE_LIMIT: + action, why = _rate_limited(state, config, now, text, tz, can_nudge) + return StallVerdict(action=action, stalled=False, reason=why, evaluated_at=now, **base) + if state.paused_at is not None and reason is None: + action, why = _paused_banner_gone(state, config, now, can_nudge) + return StallVerdict(action=action, stalled=False, reason=why, evaluated_at=now, **base) + if reason is not None: + action, why = _other_never_block(state, config, now, reason) + stalled = _secs(now, state.last_progress_at) >= config.stall_threshold_sec + return StallVerdict(action=action, stalled=stalled, reason=why, evaluated_at=now, **base) + if _secs(now, state.started_at) < config.startup_grace_sec: + return StallVerdict(action=StallAction.NONE, stalled=False, reason="startup grace", + evaluated_at=now, **base) + idle = _secs(now, state.last_progress_at) + if process_dead is True and not progress: # observed progress contradicts a dead verdict + action, why = _dead_action(state, now) + return StallVerdict(action=action, stalled=True, reason=why, evaluated_at=now, **base) + if idle < config.stall_threshold_sec: + state.stall_since = None + return StallVerdict(action=StallAction.NONE, stalled=False, evaluated_at=now, + reason=f"healthy: last progress {int(idle)}s ago", **base) + action, why = _stalled_action(state, config, now, can_nudge=can_nudge) + return StallVerdict(action=action, stalled=True, reason=why, evaluated_at=now, **base) diff --git a/src/zo/wrapper.py b/src/zo/wrapper.py index fe00402..b04307b 100644 --- a/src/zo/wrapper.py +++ b/src/zo/wrapper.py @@ -12,6 +12,12 @@ * **headless** (``--no-tmux`` or not inside tmux): runs Claude Code with ``--print --output-format json`` in a background subprocess with stdout/stderr piped to log files. + +Both poll loops tick the WS-C watchdog (``zo._wrapper_watchdog.WatchdogRunner``, +plan oracle checks 11-12) once per iteration when ``wait_for_completion`` is +given a ``WatchdogConfig`` and a memory root: stalls are nudged (tmux, bounded, +pane-ready guarded) then escalated; rate limits pause the session and resume +on verified progress; headless stalls are killed and returned as ``STALLED``. """ from __future__ import annotations @@ -20,8 +26,6 @@ import contextlib import json import os -import random -import re import shlex import signal import subprocess @@ -36,8 +40,21 @@ TeamMember, TeamStatus, ) +from zo._wrapper_watchdog import WatchdogRunner, local_tz +from zo.watchdog import ( + StallAction, + StallVerdict, + WatchdogConfig, + pane_ready_for_nudge, + parse_rate_limit_reset, + process_start_identity, + rate_limit_match, +) if TYPE_CHECKING: + from collections.abc import Callable + from datetime import tzinfo + from zo.comms import CommsLogger __all__ = [ @@ -48,12 +65,34 @@ "TeamStatus", ] -_RATE_LIMIT_PATTERNS: list[re.Pattern[str]] = [ - re.compile(r"429", re.IGNORECASE), - re.compile(r"rate.?limit", re.IGNORECASE), - re.compile(r"overloaded", re.IGNORECASE), - re.compile(r"too many requests", re.IGNORECASE), -] +# Rolling window of recent headless output fed to the watchdog each poll. +_HEADLESS_TEXT_WINDOW_CHARS = 16 * 1024 +# tmux pane capture depth per poll (shared by the watchdog and on_status). +_PANE_CAPTURE_LINES = 200 +# Named tmux buffer for wrapper pastes so the operator's buffer is untouched. +_TMUX_BUFFER_NAME = "zo-nudge" +# Watchdog runtime files must never land in a delivery repo's history. +_HEARTBEATS_GITIGNORE_ENTRY = "memory/heartbeats/" + + +def _ensure_heartbeats_gitignored(memory_root: Path) -> None: + """Idempotently ignore ``memory/heartbeats/`` in a zo-dir ``.zo/.gitignore``. + + Only touches an EXISTING ``<memory_root>/../.gitignore`` (the ``.zo/`` + scaffold writes one); legacy layouts keep memory outside the delivery + repo and need nothing. Same pattern as ``surrogate._ensure_surrogates_gitignored``. + """ + gitignore = Path(memory_root).parent / ".gitignore" + if not gitignore.is_file(): + return + existing = gitignore.read_text(encoding="utf-8") + if _HEARTBEATS_GITIGNORE_ENTRY in existing.split(): + return + with open(gitignore, "a", encoding="utf-8") as fh: + if existing and not existing.endswith("\n"): + fh.write("\n") + fh.write(f"\n# Watchdog runtime files (heartbeats, state, tick trace)\n" + f"{_HEARTBEATS_GITIGNORE_ENTRY}\n") class LifecycleWrapper: @@ -63,8 +102,14 @@ class LifecycleWrapper: comms: CommsLogger instance for audit trail events. claude_bin: Path or name of the ``claude`` CLI binary. log_dir: Directory for stdout/stderr logs (default ``logs/wrapper``). - max_retries: Max retries on rate-limit errors. - base_backoff: Base backoff in seconds for rate-limit waits. + max_retries: Retained for API compatibility; the in-loop rate-limit + retry was replaced by the watchdog pause/resume (WS-C). + base_backoff: Retained for API compatibility (see ``max_retries``). + clock: Injectable wall clock (tz-aware ``datetime``) used by the + watchdog runner; defaults to ``datetime.now(UTC)``. + tz: Zone in which rate-limit banner clock times ("resets at 3pm") + are interpreted; defaults to the operator's local zone (Claude + Code renders the reset in local time). """ # tmux liveness-detection guards (see ``_wait_tmux``). @@ -88,6 +133,8 @@ def __init__( log_dir: Path | None = None, max_retries: int = 3, base_backoff: float = 30.0, + clock: Callable[[], datetime] | None = None, + tz: tzinfo | None = None, ) -> None: self._comms = comms self._claude_bin = claude_bin @@ -95,8 +142,19 @@ def __init__( self._log_dir.mkdir(parents=True, exist_ok=True) self._max_retries = max_retries self._base_backoff = base_backoff + self._clock: Callable[[], datetime] = clock or (lambda: datetime.now(UTC)) + self._tz: tzinfo = tz or local_tz() # Restore callable for the settings.local.json overlay (set in _launch_tmux). self._bypass_restore_fn: object | None = None + # Headless subprocess handle + log handles (set in _launch_headless). + self._proc: subprocess.Popen | None = None + self._stdout_fh: Any | None = None + self._stderr_fh: Any | None = None + # WS-C watchdog runner (built in wait_for_completion when configured). + self._wd: WatchdogRunner | None = None + self._wd_text_window: str = "" + self._wd_last_skip: tuple[str, datetime | None] | None = None + self._out_cursors: dict[str, int] = {} # --- Launch --- @@ -217,6 +275,9 @@ def _launch_tmux( capture_output=True, text=True, timeout=10, ) pane_id = result.stdout.strip() + # Best-effort: the pane's shell pid, so the claude child can be + # resolved after startup (WS-C process identity; unknown ≠ dead). + shell_pid = self._tmux_pane_pid(pane_id) # 2. Start claude interactively (NO -p, NO --dangerously-skip-permissions) # --dangerously-skip-permissions exits immediately in interactive mode. @@ -249,25 +310,11 @@ def _launch_tmux( # that the pane has substantial content and has stabilised # (same content for 2 consecutive polls). self._wait_for_tui_ready(pane_id, timeout_seconds=30) + lead_pid, lead_identity = self._resolve_tmux_lead_identity(shell_pid) - # 4. Load the prompt into tmux's paste buffer and paste it - # into the Claude TUI input field. - subprocess.run( - ["tmux", "load-buffer", str(prompt_file)], - capture_output=True, text=True, timeout=10, - ) - subprocess.run( - ["tmux", "paste-buffer", "-t", pane_id], - capture_output=True, text=True, timeout=10, - ) - - # 5. Send Enter to submit the prompt. Wait briefly for the - # paste to be ingested by the TUI before pressing Enter. - time.sleep(1) - subprocess.run( - ["tmux", "send-keys", "-t", pane_id, "Enter"], - capture_output=True, text=True, timeout=10, - ) + # 4./5. Paste the prompt into the Claude TUI input field via a + # named tmux buffer and submit it with Enter. + self._paste_and_submit(pane_id, prompt) # 6. Verify the prompt was submitted by checking that pane # content changed after the paste (not still showing the @@ -275,7 +322,8 @@ def _launch_tmux( self._verify_prompt_submitted(pane_id, prompt_file) lead = LeadProcess( - pid=None, status=AgentStatus.SPAWNING, + pid=lead_pid, pid_start_identity=lead_identity, + status=AgentStatus.SPAWNING, started_at=datetime.now(UTC), team_name=team_name, stdout_log=stdout_log, stderr_log=stderr_log, tmux_pane_id=pane_id, @@ -301,6 +349,66 @@ def _capture_pane(pane_id: str) -> str: except Exception: # noqa: BLE001 return "" + @staticmethod + def _paste_and_submit(pane_id: str, text: str) -> None: + """Paste ``text`` into a tmux pane via a NAMED buffer, then press Enter. + + Uses ``load-buffer -b zo-nudge -`` (stdin) + ``paste-buffer -b + zo-nudge -d`` so the operator's default paste buffer is never + clobbered and no temp file is needed. Shared by the launch path + (lead prompt) and the watchdog nudge path. + """ + subprocess.run( + ["tmux", "load-buffer", "-b", _TMUX_BUFFER_NAME, "-"], + input=text, capture_output=True, text=True, timeout=10, + ) + subprocess.run( + ["tmux", "paste-buffer", "-b", _TMUX_BUFFER_NAME, "-d", "-t", pane_id], + capture_output=True, text=True, timeout=10, + ) + # Wait briefly for the paste to be ingested by the TUI before Enter. + time.sleep(1) + subprocess.run( + ["tmux", "send-keys", "-t", pane_id, "Enter"], + capture_output=True, text=True, timeout=10, + ) + + @staticmethod + def _tmux_pane_pid(pane_id: str) -> int | None: + """``#{pane_pid}`` (the pane's shell pid) or ``None`` (best-effort).""" + if not pane_id: + return None + try: + result = subprocess.run( + ["tmux", "display-message", "-t", pane_id, "-p", "#{pane_pid}"], + capture_output=True, text=True, timeout=5, + ) + return int(result.stdout.strip()) + except Exception: # noqa: BLE001 — identity is advisory + return None + + @staticmethod + def _resolve_tmux_lead_identity(shell_pid: int | None) -> tuple[int | None, str | None]: + """Resolve the claude child of the pane shell (``pgrep``) + its start identity. + + Only children whose command line mentions ``claude`` qualify (an + interactive shell also has prompt helpers, gitstatusd, …), newest + first (``-n``). Returns ``(None, None)`` when unresolvable — unknown + is never dead, and a wrong pid would be worse than none. + """ + if shell_pid is None: + return None, None + try: + result = subprocess.run( + ["pgrep", "-n", "-P", str(shell_pid), "-f", "claude"], + capture_output=True, text=True, timeout=5, + ) + first = result.stdout.strip().splitlines()[0].strip() + pid = int(first) + except Exception: # noqa: BLE001 — identity is advisory + return None, None + return pid, process_start_identity(pid) + def _wait_for_tui_ready( self, pane_id: str, *, timeout_seconds: int = 30, ) -> None: @@ -383,19 +491,11 @@ def _verify_prompt_submitted( subtask="paste-retry", progress="Paste may have missed — retrying once.", ) - subprocess.run( - ["tmux", "load-buffer", str(prompt_file)], - capture_output=True, text=True, timeout=10, - ) - subprocess.run( - ["tmux", "paste-buffer", "-t", pane_id], - capture_output=True, text=True, timeout=10, - ) - time.sleep(1) - subprocess.run( - ["tmux", "send-keys", "-t", pane_id, "Enter"], - capture_output=True, text=True, timeout=10, - ) + try: + retry_text = prompt_file.read_text(encoding="utf-8") + except OSError: + retry_text = "" + self._paste_and_submit(pane_id, retry_text) # Final check time.sleep(2) @@ -454,7 +554,6 @@ def _launch_headless( stdout_fh = open(stdout_log, "w", encoding="utf-8") # noqa: SIM115 stderr_fh = open(stderr_log, "w", encoding="utf-8") # noqa: SIM115 - import os env = os.environ.copy() if extra_env: env.update(extra_env) @@ -464,6 +563,7 @@ def _launch_headless( ) lead = LeadProcess( pid=proc.pid, status=AgentStatus.SPAWNING, + pid_start_identity=self._safe_start_identity(proc.pid), started_at=datetime.now(UTC), team_name=team_name, stdout_log=stdout_log, stderr_log=stderr_log, ) @@ -554,6 +654,9 @@ def wait_for_completion( gate_mode_file: Path | None = None, project_name: str = "", delivery_repo: Path | None = None, + watchdog: WatchdogConfig | None = None, + memory_root: Path | None = None, + zo_session_id: str = "", ) -> LeadProcess: """Poll until the lead session completes. @@ -570,12 +673,20 @@ def wait_for_completion( delivery_repo: Delivery repo path. When provided (with *project_name*), the wrapper auto-splits a training dashboard pane when training metrics appear. + watchdog: WS-C watchdog policy. The external checker runs + once per poll when this is enabled AND *memory_root* is + given; otherwise the loops behave exactly as before. + memory_root: Per-project memory root holding + ``heartbeats/`` (heartbeat files, watchdog state, tick trace). + zo_session_id: Comms session id for heartbeat/state correlation. """ self._gate_mode_file = gate_mode_file self._last_gate_mode: str | None = None self._training_pane_id: str | None = None self._project_name = project_name self._delivery_repo = delivery_repo + self._start_watchdog(watchdog, memory_root=memory_root, + zo_session_id=zo_session_id, delivery_repo=delivery_repo) try: if process.tmux_pane_id: return self._wait_tmux(process, poll_interval=poll_interval, @@ -584,6 +695,200 @@ def wait_for_completion( timeout=timeout, on_status=on_status) finally: self._close_training_pane() + if self._wd is not None: + self._wd.stop() + + # --- Watchdog (WS-C, oracle checks 11-12) --- + + def _start_watchdog( + self, watchdog: WatchdogConfig | None, *, memory_root: Path | None, + zo_session_id: str, delivery_repo: Path | None, + ) -> None: + """Build the runner when configured; fail-open (no runner) on any error.""" + self._wd = None + if watchdog is None or not watchdog.enabled or memory_root is None: + return + root = Path(memory_root) + paths: list[Path] = [root / "plan-ledger.json"] + comms_dir = getattr(self._comms, "_log_dir", None) + if comms_dir: + paths.append(Path(comms_dir)) + if delivery_repo is not None: + experiments = Path(delivery_repo) / ".zo" / "experiments" + if experiments.exists(): + paths.append(experiments) + paths.extend(Path(p) for p in watchdog.progress_paths) + try: + runner = WatchdogRunner( + config=watchdog, memory_root=root, zo_session_id=zo_session_id, + clock=self._clock, tz=self._tz, progress_paths=paths, + ) + runner.start(runner.clock()) + except Exception as exc: # noqa: BLE001 — advisory: never break the session + self._wd_log_error("watchdog_init", "warning", + f"Watchdog disabled for this run: {exc!r}") + return + self._wd = runner + with contextlib.suppress(Exception): # advisory: never break the session + _ensure_heartbeats_gitignored(root) + + def _wd_checkpoint(self, subtask: str, progress: str, *, + blockers: list[str] | None = None) -> None: + """Advisory comms checkpoint from the watchdog (never raises).""" + with contextlib.suppress(Exception): + self._comms.log_checkpoint(agent="watchdog", phase="lifecycle", + subtask=subtask, progress=progress, + blockers=blockers) + + def _wd_log_error(self, error_type: str, severity: str, description: str, *, + escalated_to: str = "") -> None: + """Advisory comms error from the watchdog (never raises).""" + with contextlib.suppress(Exception): + self._comms.log_error(agent="watchdog", error_type=error_type, + severity=severity, description=description, + escalated_to=escalated_to) + + def _watchdog_tick( + self, process: LeadProcess, *, text: str, can_nudge: bool, + process_dead: bool | None = None, + ) -> StallVerdict | None: + """One external-checker tick; applies side effects to ``process`` in place. + + Returns the verdict, or ``None`` when no runner is configured or the + tick itself failed (fail-open: no decision fires on unknown evidence). + """ + wd = self._wd + if wd is None: + return None + try: + verdict = wd.tick(process=process, text=text, can_nudge=can_nudge, + now=wd.clock(), process_dead=process_dead) + except Exception as exc: # noqa: BLE001 — evidence gathering is advisory + self._wd_log_error("watchdog_tick", "warning", f"Watchdog tick failed: {exc!r}") + return None + if wd.last_new_stall: + self._wd_log_error("stall", "warning", f"Stall detected: {verdict.reason}") + if verdict.action in (StallAction.NUDGE, StallAction.RESUME_NUDGE): + self._wd_nudge(process, verdict, text=text) + elif verdict.action == StallAction.PAUSE: + self._wd_pause(process, verdict) + elif verdict.action == StallAction.RESUME: + self._wd_resume(process, verdict) + elif verdict.action == StallAction.ESCALATE: + self._wd_escalate(process, verdict) + wd.settle() + return verdict + + def _wd_nudge(self, process: LeadProcess, verdict: StallVerdict, *, text: str) -> None: + """Deliver a nudge into the tmux pane, guarded by ``pane_ready_for_nudge``.""" + wd = self._wd + if wd is None: + return + resume = verdict.action == StallAction.RESUME_NUDGE + budget = wd.config.resume_nudge_budget if resume else wd.config.nudge_budget + used = wd.state.resume_nudges_used if resume else wd.state.nudges_used + label = "resume nudge" if resume else "nudge" + if not process.tmux_pane_id or not pane_ready_for_nudge(text): + # Log once per stall/pause episode, not every poll. + episode = (label, wd.state.stall_since or wd.state.paused_at) + if episode != self._wd_last_skip: + self._wd_last_skip = episode + self._wd_checkpoint( + "nudge-skipped", + f"{label} {used + 1}/{budget} skipped: pane busy or awaiting input", + blockers=[verdict.reason]) + return + try: + self._paste_and_submit(process.tmux_pane_id, wd.config.nudge_message) + except Exception as exc: # noqa: BLE001 — tmux hiccup; do not consume budget + self._wd_log_error("nudge_failed", "warning", f"Nudge paste failed: {exc!r}") + return + wd.record_nudge(wd.clock(), resume=resume) + process.nudges_used = wd.state.nudges_used + self._wd_checkpoint("nudge", f"{label} {used + 1}/{budget}: {verdict.reason}") + + def _wd_pause(self, process: LeadProcess, verdict: StallVerdict) -> None: + """Enter the paused state on FIRST detection only (extensions are silent).""" + wd = self._wd + if wd is None: + return + state = wd.state + process.paused_until = state.paused_until + if state.paused_at != verdict.evaluated_at: + return + process.status = AgentStatus.PAUSED_RATE_LIMIT + self._wd_checkpoint( + "rate-limit-pause", + f"Rate limit detected; paused until {state.paused_until}: {verdict.reason}", + blockers=["rate_limit"]) + + def _wd_resume(self, process: LeadProcess, verdict: StallVerdict) -> None: + """Verified resume: progress observed after a rate-limit pause.""" + wd = self._wd + if wd is None: + return + process.status = AgentStatus.RUNNING + process.paused_until = None + process.pause_total_sec = wd.state.total_paused_sec + self._wd_checkpoint( + "rate-limit-resume", + f"Resumed after rate-limit pause (verified={verdict.progress}; " + f"paused {process.pause_total_sec:.0f}s total): {verdict.reason}") + + def _wd_escalate(self, process: LeadProcess, verdict: StallVerdict) -> None: + """Escalate to the human; headless additionally kills the session.""" + wd = self._wd + if wd is None: + return + self._wd_log_error("stall", "blocking", verdict.reason, escalated_to="human") + process.stalled = True + if process.tmux_pane_id or not wd.config.kill_headless_on_escalate: + return # tmux: human-facing pane is never killed; keep waiting + killed = self.kill_session(process) + process.exit_code = killed.exit_code + process.completed_at = killed.completed_at + process.status = AgentStatus.STALLED + + def _read_new_output(self, process: LeadProcess) -> str: + """Byte-cursor read of NEW stdout/stderr bytes; maintains the rolling window.""" + new_chunks: list[str] = [] + for path in (process.stdout_log, process.stderr_log): + if path is None: + continue + key = str(path) + try: + with open(path, "rb") as fh: + fh.seek(self._out_cursors.get(key, 0)) + data = fh.read() + self._out_cursors[key] = fh.tell() + except OSError: + continue + if data: + new_chunks.append(data.decode("utf-8", errors="replace")) + new_text = "".join(new_chunks) + if new_text: + window = self._wd_text_window + new_text + self._wd_text_window = window[-_HEADLESS_TEXT_WINDOW_CHARS:] + return new_text + + def _elapsed(self, start_time: float) -> float: + """Wall-clock seconds since ``start_time`` minus any rate-limit pause.""" + elapsed = time.monotonic() - start_time + if self._wd is not None: + elapsed -= self._wd.paused_seconds() + return elapsed + + def _timed_out(self, process: LeadProcess, timeout: float | None, + start_time: float) -> LeadProcess | None: + """Return the TIMED_OUT process if the (pause-adjusted) budget is spent.""" + if not timeout or self._elapsed(start_time) <= timeout: + return None + process = process.model_copy(update={"status": AgentStatus.TIMED_OUT}) + self._comms.log_error( + agent="wrapper", error_type="timeout", severity="blocking", + description=f"Lead session timed out after {timeout}s", + ) + return process def _maybe_open_training_pane(self) -> None: """Open a training dashboard split-pane if metrics file appears. @@ -709,6 +1014,14 @@ def _wait_tmux( while True: self._check_gate_mode_change() self._maybe_open_training_pane() + # ONE pane capture per poll, shared by the watchdog and on_status. + pane_text = self._capture_tmux_pane(pane_id, lines=_PANE_CAPTURE_LINES) + # Watchdog tick BEFORE the liveness reads so it also runs on the + # suspected-dead ``continue`` path below. A busy pane (spinner, + # "esc to interrupt", dialog) cannot be nudged, so the policy + # escalates a persistent stall there instead of nudging forever. + self._watchdog_tick(process, text=pane_text, + can_nudge=pane_ready_for_nudge(pane_text)) pane_exists = self._tmux_pane_alive(pane_id) claude_running = pane_exists and self._tmux_claude_running(pane_id) @@ -726,16 +1039,7 @@ def _wait_tmux( # Confirmed: Claude exited — clean up the shell window. if pane_exists: self._kill_tmux_window(pane_id) - process = process.model_copy(update={ - "exit_code": 0, "completed_at": datetime.now(UTC), - "status": AgentStatus.COMPLETED, - }) - self._comms.log_checkpoint( - agent="wrapper", phase="lifecycle", - subtask="completion", - progress="Lead session completed, agent window closed", - ) - return process + return self._tmux_final_status(process) # Suspected exit but not yet confirmed — re-check soon # rather than waiting a full poll interval. if on_status: @@ -748,18 +1052,39 @@ def _wait_tmux( if on_status: team_status = self.monitor_team(process.team_name) - pane_snapshot = self._capture_tmux_pane(pane_id, lines=5) + pane_snapshot = "\n".join(pane_text.splitlines()[-5:]) on_status(team_status, pane_snapshot) - if timeout and (time.monotonic() - start_time) > timeout: - process = process.model_copy(update={"status": AgentStatus.TIMED_OUT}) - self._comms.log_error( - agent="wrapper", error_type="timeout", severity="blocking", - description=f"Lead session timed out after {timeout}s", - ) - return process + timed_out = self._timed_out(process, timeout, start_time) + if timed_out is not None: + return timed_out time.sleep(poll_interval) + def _tmux_final_status(self, process: LeadProcess) -> LeadProcess: + """Terminal status once the tmux pane/claude is confirmed gone. + + ``STALLED`` if the watchdog escalated and nothing progressed since; + ``RATE_LIMITED`` (with ``resume_at``) if the session died while a + *corroborated* rate-limit pause was in effect (parsed reset time or + an unambiguous banner — prose in a final summary does not count); + else the normal ``COMPLETED``. + """ + wd = self._wd + status = AgentStatus.COMPLETED + update: dict[str, Any] = {"exit_code": 0, "completed_at": datetime.now(UTC)} + if process.stalled and (wd is None or not wd.progress_since_escalation()): + status = AgentStatus.STALLED + elif wd is not None and wd.rate_limit_exit_evidence(): + status = AgentStatus.RATE_LIMITED + update["resume_at"] = wd.parsed_resume_at() + update["status"] = status + process = process.model_copy(update=update) + self._comms.log_checkpoint( + agent="wrapper", phase="lifecycle", subtask="completion", + progress=f"Lead session completed, agent window closed (status={status.value})", + ) + return process + def _wait_headless( self, process: LeadProcess, @@ -768,59 +1093,83 @@ def _wait_headless( timeout: float | None, on_status: Any | None, ) -> LeadProcess: - """Wait for the headless subprocess to exit.""" + """Wait for the headless subprocess to exit. + + Rate limits are handled by the watchdog pause/resume state (evaluated + every poll — never a blocking backoff sleep). When the process exits + while rate-limited the status is ``RATE_LIMITED`` with a parsed + ``resume_at``; the driver above the wrapper relaunches. + """ start_time = time.monotonic() - retries = 0 process = process.model_copy(update={"status": AgentStatus.RUNNING}) + self._wd_text_window = "" + self._out_cursors = {} while True: self._check_gate_mode_change() + self._read_new_output(process) rc = self._proc.poll() if self._proc else -1 + if rc is None: + # Alive: process_dead=False is authoritative here (Popen.poll). + verdict = self._watchdog_tick(process, text=self._wd_text_window, + can_nudge=False, process_dead=False) + if process.status == AgentStatus.STALLED: + return process + if verdict is not None and verdict.action == StallAction.PAUSE: + # The banner is consumed: unlike a live pane it never + # disappears from a log window, so only NEW output is + # classified from here on (new-lines-only cursor). + self._wd_text_window = "" if rc is not None: self._close_log_handles() - process = process.model_copy(update={ - "exit_code": rc, "completed_at": datetime.now(UTC), - "status": AgentStatus.COMPLETED if rc == 0 else AgentStatus.ERRORED, - }) - self._comms.log_checkpoint( - agent="wrapper", phase="lifecycle", subtask="completion", - progress=f"Lead session exited code={rc}", - ) - return process - - output = self._read_tail(process.stdout_log) - if self._detect_rate_limit(output): - if retries >= self._max_retries: - process = process.model_copy(update={"status": AgentStatus.RATE_LIMITED}) - self._comms.log_error( - agent="wrapper", error_type="rate_limit", severity="blocking", - description=f"Rate limited after {retries} retries", - ) - return process - wait_secs = self._backoff_wait(retries) - self._comms.log_checkpoint( - agent="wrapper", phase="lifecycle", subtask="rate-limit-backoff", - progress=f"Rate limited, retry {retries + 1}/{self._max_retries}, " - f"waiting {wait_secs:.0f}s", - ) - time.sleep(wait_secs) - retries += 1 - continue + return self._headless_exit_status(process, rc) if on_status: team_status = self.monitor_team(process.team_name) on_status(team_status, "") - if timeout and (time.monotonic() - start_time) > timeout: - process = process.model_copy(update={"status": AgentStatus.TIMED_OUT}) - self._comms.log_error( - agent="wrapper", error_type="timeout", severity="blocking", - description=f"Lead session timed out after {timeout}s", - ) - return process + timed_out = self._timed_out(process, timeout, start_time) + if timed_out is not None: + return timed_out time.sleep(poll_interval) + def _headless_exit_status(self, process: LeadProcess, rc: int) -> LeadProcess: + """Classify a headless exit: RATE_LIMITED (+resume_at) / COMPLETED / ERRORED. + + RATE_LIMITED needs corroboration: the watchdog's pause was backed by a + parsed reset time or an unambiguous banner, or the process exited + non-zero with rate-limit text in its final output. A successful run + whose JSON result merely *mentions* rate limits stays COMPLETED. + """ + window = self._wd_text_window + wd = self._wd + rate_limited = ( + (wd is not None and wd.rate_limit_exit_evidence(rc=rc)) + or (rc != 0 and self._detect_rate_limit(window)) + ) + update: dict[str, Any] = {"exit_code": rc, "completed_at": datetime.now(UTC)} + if process.stalled and (wd is None or not wd.progress_since_escalation()): + update["status"] = AgentStatus.STALLED # escalated, kill disabled, died stalled + elif rate_limited: + resume_at = wd.parsed_resume_at() if wd is not None else None + if resume_at is None: + resume_at = parse_rate_limit_reset(window, now=self._clock(), tz=self._tz) + update.update({"status": AgentStatus.RATE_LIMITED, "resume_at": resume_at}) + self._comms.log_error( + agent="wrapper", error_type="rate_limit", severity="blocking", + description=(f"Lead session exited code={rc} while rate limited; " + f"resume_at={resume_at}"), + ) + else: + update["status"] = AgentStatus.COMPLETED if rc == 0 else AgentStatus.ERRORED + process = process.model_copy(update=update) + self._comms.log_checkpoint( + agent="wrapper", phase="lifecycle", subtask="completion", + progress=f"Lead session exited code={rc}", + ) + return process + def kill_session(self, process: LeadProcess) -> LeadProcess: """Terminate the lead session. SIGTERM, wait 5s, SIGKILL if needed.""" if process.tmux_pane_id: @@ -892,12 +1241,24 @@ def parse_session_result(self, process: LeadProcess) -> dict[str, str]: @staticmethod def _detect_rate_limit(output: str) -> bool: - """Return True if output contains rate-limit / overload patterns.""" - return any(pat.search(output) for pat in _RATE_LIMIT_PATTERNS) + """Exit-classification only: does the final output carry a rate-limit banner? - def _backoff_wait(self, attempt: int) -> float: - """Exponential backoff: base * 2^attempt + random(0, 5).""" - return self._base_backoff * (2 ** attempt) + random.uniform(0, 5) + Uses the watchdog's tiered table (no bare ``429`` / ``overloaded``; + loose phrases need same-line rate/usage/quota vocabulary — ``val_loss + 0.4291``, ``GPU overloaded`` and ``patience limit reached`` are not + rate limits). + """ + return rate_limit_match(output) is not None + + @staticmethod + def _safe_start_identity(pid: object) -> str | None: + """Best-effort process start identity for a freshly spawned pid.""" + if not isinstance(pid, int) or isinstance(pid, bool): + return None + try: + return process_start_identity(pid) + except Exception: # noqa: BLE001 — identity is advisory + return None # --- Private: resolve claude binary --- diff --git a/tests/integration/test_hooks_shim.py b/tests/integration/test_hooks_shim.py index 56cab0d..cc6ad9d 100644 --- a/tests/integration/test_hooks_shim.py +++ b/tests/integration/test_hooks_shim.py @@ -63,6 +63,9 @@ def contracts_env(tmp_path: Path) -> dict[str, str]: "ZO_CONTRACTS_PATH": str(contracts), "ZO_DELIVERY_ROOT": str(delivery), "ZO_REPO_ROOT": str(REPO_ROOT), + # WS-C: every routed handler now stamps a heartbeat under the memory + # root — keep that in tmp, never in the live platform repo. + "ZO_MEMORY_ROOT": str(tmp_path / "mem"), } @@ -146,3 +149,149 @@ def test_drift_guard_on_stop_and_sealed_paths_on_pretooluse(self) -> None: for h in entry["hooks"] ] assert any("sealed-paths" in c for c in ptu_cmds) + + def test_heartbeat_wired_on_post_tool_use(self) -> None: + """WS-C (oracle check 11): the heartbeat writer runs on every tool + call (matcher ``*``) as a SECOND PostToolUse entry; the existing + Write|Edit cascade-reminder entry is untouched and no new hook + events were introduced.""" + settings = json.loads( + (REPO_ROOT / ".claude" / "settings.json").read_text() + ) + post = settings["hooks"]["PostToolUse"] + heartbeat_entries = [ + (entry["matcher"], h) + for entry in post + for h in entry["hooks"] + if "zo-hookkit.sh heartbeat" in h["command"] + ] + assert len(heartbeat_entries) == 1 + matcher, hook = heartbeat_entries[0] + assert matcher == "*" + assert hook["timeout"] <= 5 + assert hook["command"].endswith("|| exit 0") + cascade = [ + entry for entry in post + if entry["matcher"] == "Write|Edit" + and any("cascade-reminder.sh" in h["command"] for h in entry["hooks"]) + ] + assert len(cascade) == 1 + assert set(settings["hooks"]) == { + "SessionStart", "PreToolUse", "PostToolUse", "Stop", "SubagentStop", + "PreCompact", "SessionEnd", "PostToolUseFailure", + } + + def test_shim_exports_hook_event_ts(self) -> None: + assert 'export ZO_HOOK_EVENT_TS="$(date -u +%s)"' in SHIM.read_text() + + +# ---- heartbeat end-to-end through the shim (oracle check 11) ----------------- + + +@pytest.fixture() +def heartbeat_env(tmp_path: Path) -> dict[str, str]: + """Sandbox both roots so the shim writes only into tmp.""" + repo = tmp_path / "repo" + repo.mkdir() + mem = tmp_path / "mem" + mem.mkdir() + return {"ZO_REPO_ROOT": str(repo), "ZO_MEMORY_ROOT": str(mem)} + + +class TestShimHeartbeat: + def test_end_to_end_heartbeat_write(self, heartbeat_env, tmp_path: Path) -> None: + from zo.watchdog import HeartbeatRecord, HeartbeatStatus + + payload = { + "hook_event_name": "PostToolUse", "session_id": "shim-1", + "tool_name": "Bash", "agent_id": "agent-shim", "agent_type": "data-engineer", + } + code, out = _run_shim("heartbeat", payload, heartbeat_env) + assert code == 0 + assert out == "" + path = tmp_path / "mem" / "heartbeats" / "agent-shim.json" + rec = HeartbeatRecord.model_validate_json(path.read_text(encoding="utf-8")) + assert rec.agent_key == "agent-shim" + assert rec.status is HeartbeatStatus.EXECUTING + assert rec.last_event == "Bash" + assert rec.tick_count == 1 + # nothing leaked into the live platform memory root + live = REPO_ROOT / "memory" / "zo-platform" / "heartbeats" / "agent-shim.json" + assert not live.exists() + + def test_lead_key_when_no_identity(self, heartbeat_env, tmp_path: Path) -> None: + code, out = _run_shim( + "heartbeat", {"hook_event_name": "PostToolUse", "session_id": "shim-2"}, + heartbeat_env, + ) + assert code == 0 and out == "" + assert (tmp_path / "mem" / "heartbeats" / "lead-shim-2.json").exists() + + def test_malformed_stdin_exits_zero_silent(self, heartbeat_env) -> None: + import os + + env = {**os.environ, **heartbeat_env} + result = subprocess.run( + ["bash", str(SHIM), "heartbeat"], + input="{not json at all", capture_output=True, text=True, + timeout=30, env=env, cwd=str(REPO_ROOT), check=False, + ) + assert result.returncode == 0 + assert result.stdout == "" + + def test_no_memory_root_is_silent_noop(self, tmp_path: Path) -> None: + import os + + repo = tmp_path / "repo" + repo.mkdir() + env = {k: v for k, v in os.environ.items() if k != "ZO_MEMORY_ROOT"} + env["ZO_REPO_ROOT"] = str(repo) + result = subprocess.run( + ["bash", str(SHIM), "heartbeat"], + input=json.dumps({"session_id": "x"}), capture_output=True, text=True, + timeout=30, env=env, cwd=str(REPO_ROOT), check=False, + ) + assert result.returncode == 0 and result.stdout == "" + assert not list(tmp_path.rglob("heartbeats")) + + +# ---- .gitignore guard for control-plane runtime files --------------------- + + +class TestGitignoreControlPlane: + """Runtime files under the platform memory root must never be tracked; + the ``!memory/zo-platform/`` re-include makes ordering load-bearing.""" + + @pytest.fixture(autouse=True) + def _require_git_repo(self) -> None: + probe = subprocess.run( + ["git", "-C", str(REPO_ROOT), "rev-parse", "--is-inside-work-tree"], + capture_output=True, text=True, check=False, + ) + if probe.returncode != 0 or probe.stdout.strip() != "true": + pytest.skip("not inside a git work tree") + + @pytest.mark.parametrize( + "rel", + [ + "memory/zo-platform/heartbeats/agent-1.json", + "memory/zo-platform/heartbeats/_watchdog.json", + "memory/zo-platform/plan-ledger.json", + "memory/zo-platform/contracts.json", + ], + ) + def test_control_plane_paths_are_ignored(self, rel: str) -> None: + result = subprocess.run( + ["git", "-C", str(REPO_ROOT), "check-ignore", "-v", rel], + capture_output=True, text=True, check=False, + ) + assert result.returncode == 0, f"{rel} is NOT ignored" + assert ".gitignore" in result.stdout + + def test_platform_memory_itself_stays_tracked(self) -> None: + result = subprocess.run( + ["git", "-C", str(REPO_ROOT), "check-ignore", "-q", + "memory/zo-platform/STATE.md"], + capture_output=True, text=True, check=False, + ) + assert result.returncode == 1 # 1 == not ignored diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py index edc68e7..1fe4225 100644 --- a/tests/unit/test_cli.py +++ b/tests/unit/test_cli.py @@ -1548,3 +1548,320 @@ def test_does_not_use_legacy_logs_training_path( log_dir = mock_live.call_args[0][0] assert "logs/training" not in str(log_dir) assert ".zo/experiments" in str(log_dir).replace("\\", "/") + + +# --------------------------------------------------------------------------- +# Watchdog CLI threading (WS-C) +# --------------------------------------------------------------------------- + + +# ---- watchdog config → CLI → wrapper wiring (oracle checks 11-12) ---- + + +_FIXTURE_PLAN = Path(__file__).resolve().parents[1] / "fixtures" / "test-project" / "plan.md" + + +def _make_zo_dir_project(tmp_path: Path, **config_kwargs: object) -> tuple[Path, Path]: + """A .zo/ delivery repo carrying the fixture plan; returns (repo, plan_path).""" + import shutil + + from zo.project_config import ( + LocalConfig, + ProjectConfig, + save_local_config, + save_project_config, + ) + + repo = tmp_path / "delivery" + repo.mkdir() + save_project_config( + repo, ProjectConfig(project_name="churn-prediction", **config_kwargs), + ) + save_local_config(repo, LocalConfig()) # skip the new-machine prompt + plans = repo / ".zo" / "plans" + plans.mkdir(parents=True) + plan_path = plans / "churn-prediction.md" + shutil.copy(_FIXTURE_PLAN, plan_path) + return repo, plan_path + + +def _invoke_build(runner: click.testing.CliRunner, tmp_path: Path, args: list[str]): # noqa: ANN202 + """Run ``zo build`` end-to-end up to the launch seam; returns (result, launch mock).""" + zo_root = tmp_path / "zo" + zo_root.mkdir(exist_ok=True) + with patch("zo.cli._zo_root", return_value=zo_root), \ + patch("zo.cli._launch_and_monitor") as lam: + result = runner.invoke(cli, ["build", *args, "--gate-mode", "full-auto"]) + return result, lam + + +class _StubProcess: + """Minimal LeadProcess stand-in for ``_launch_and_monitor`` tests.""" + + tmux_pane_id = None + pid = 4242 + team_name = "zo-churn-prediction" + + def __init__(self, status: str = "completed", resume_at=None) -> None: # noqa: ANN001 + from datetime import UTC, datetime + + self.status = status + self.resume_at = resume_at + self.started_at = datetime.now(UTC) + + +class _StubWrapper: + """Records the kwargs the CLI passes to ``wait_for_completion``.""" + + def __init__(self, final: _StubProcess | None = None) -> None: + self.final = final + self.wait_kwargs: dict = {} + + def launch_lead_session(self, prompt: str, **kw: object) -> _StubProcess: + return _StubProcess() + + def wait_for_completion(self, process: _StubProcess, **kw: object) -> _StubProcess: + self.wait_kwargs = dict(kw) + return self.final or process + + def read_task_list(self, team_name: str) -> list: + return [] + + +class TestWatchdogCliThreading: + """``--no-watchdog`` / config / env reach the wrapper; STALLED and + RATE_LIMITED outcomes are reported with actionable text. + + Seeded half: a run started with the kill switch (flag, env, or file) + must hand the wrapper ``enabled=False``. Wiring half: the default path + hands the wrapper a config plus the project memory root, and + ``_launch_and_monitor`` forwards them as keyword arguments to + ``wait_for_completion``. + """ + + @pytest.fixture(autouse=True) + def _no_ambient_watchdog_env(self, monkeypatch: pytest.MonkeyPatch) -> None: + """``_resolve_watchdog`` reads os.environ: a developer/CI shell that + exports ZO_WATCHDOG / ZO_WATCHDOG_STALL_SEC must not skew these asserts.""" + monkeypatch.delenv("ZO_WATCHDOG", raising=False) + monkeypatch.delenv("ZO_WATCHDOG_STALL_SEC", raising=False) + + def test_seeded_no_watchdog_flag_disables_watchdog( + self, runner: click.testing.CliRunner, tmp_path: Path, + ) -> None: + """Seeded: ``zo build --no-watchdog`` → launch receives ``enabled=False``.""" + _, plan_path = _make_zo_dir_project(tmp_path) + result, lam = _invoke_build(runner, tmp_path, [str(plan_path), "--no-watchdog"]) + + assert result.exit_code == 0, result.output + kw = lam.call_args.kwargs + assert kw["watchdog"].enabled is False + # Single concern: the flag flips only ``enabled``; policy stays default. + assert kw["watchdog"].stall_threshold_sec == 1200 + + def test_default_build_passes_enabled_config_and_memory_root( + self, runner: click.testing.CliRunner, tmp_path: Path, + ) -> None: + """Wiring: no flag → enabled config, project memory root, comms session id.""" + from zo.watchdog import WatchdogConfig + + repo, plan_path = _make_zo_dir_project(tmp_path) + result, lam = _invoke_build(runner, tmp_path, [str(plan_path)]) + + assert result.exit_code == 0, result.output + kw = lam.call_args.kwargs + assert isinstance(kw["watchdog"], WatchdogConfig) + assert kw["watchdog"].enabled is True + assert kw["memory_root"] == repo / ".zo" / "memory" + assert kw["zo_session_id"].startswith("s-") + # Heartbeat ↔ comms correlation: the same id is exported to hooks. + assert kw["extra_env"]["ZO_SESSION_ID"] == kw["zo_session_id"] + # Heartbeat WRITER root (hooks, via env) == watchdog READER root: if these + # ever diverge the evidence channel silently goes dark (fail-open). + assert kw["extra_env"]["ZO_MEMORY_ROOT"] == str(kw["memory_root"]) + + def test_project_config_watchdog_block_threads_through( + self, runner: click.testing.CliRunner, tmp_path: Path, + ) -> None: + """``.zo/config.yaml`` watchdog values (not defaults) reach the launch.""" + from zo.watchdog import WatchdogConfig + + _, plan_path = _make_zo_dir_project( + tmp_path, watchdog=WatchdogConfig(stall_threshold_sec=600, nudge_budget=1), + ) + result, lam = _invoke_build(runner, tmp_path, [str(plan_path)]) + + assert result.exit_code == 0, result.output + wd = lam.call_args.kwargs["watchdog"] + assert wd.stall_threshold_sec == 600 + assert wd.nudge_budget == 1 + assert wd.enabled is True + + def test_seeded_project_config_disabled_watchdog_threads_through( + self, runner: click.testing.CliRunner, tmp_path: Path, + ) -> None: + """Seeded: ``watchdog: {enabled: false}`` in config.yaml disables it.""" + from zo.watchdog import WatchdogConfig + + _, plan_path = _make_zo_dir_project(tmp_path, watchdog=WatchdogConfig(enabled=False)) + result, lam = _invoke_build(runner, tmp_path, [str(plan_path)]) + + assert result.exit_code == 0, result.output + assert lam.call_args.kwargs["watchdog"].enabled is False + + def test_seeded_env_kill_switch_disables_watchdog( + self, runner: click.testing.CliRunner, tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Seeded: ``ZO_WATCHDOG=0`` in the environment disables it (ops override).""" + monkeypatch.setenv("ZO_WATCHDOG", "0") + _, plan_path = _make_zo_dir_project(tmp_path) + result, lam = _invoke_build(runner, tmp_path, [str(plan_path)]) + + assert result.exit_code == 0, result.output + assert lam.call_args.kwargs["watchdog"].enabled is False + + def test_resolve_watchdog_legacy_layout_defaults(self, tmp_path: Path) -> None: + """Legacy (targets/*.target.md) projects have no config → defaults, ON.""" + from zo.cli import ProjectContext, _resolve_watchdog + + ctx = ProjectContext( + layout="legacy", delivery_repo=tmp_path, plan_path=tmp_path / "p.md", + project_name="legacy", zo_root=tmp_path, + ) + assert ctx.make_project_config() is None + wd = _resolve_watchdog(ctx, no_watchdog=False) + assert wd.enabled is True + assert _resolve_watchdog(ctx, no_watchdog=True).enabled is False + + def test_make_project_config_loads_zo_dir_layout(self, tmp_path: Path) -> None: + from zo.cli import ProjectContext + from zo.watchdog import WatchdogConfig + + repo, plan_path = _make_zo_dir_project( + tmp_path, watchdog=WatchdogConfig(nudge_budget=2), + ) + ctx = ProjectContext( + layout="zo-dir", delivery_repo=repo, plan_path=plan_path, + project_name="churn-prediction", zo_root=tmp_path, + ) + pcfg = ctx.make_project_config() + assert pcfg is not None + assert pcfg.project_name == "churn-prediction" + assert pcfg.watchdog.nudge_budget == 2 + + def test_continue_forwards_no_watchdog_to_build( + self, runner: click.testing.CliRunner, tmp_path: Path, + ) -> None: + """``zo continue --no-watchdog`` delegates the flag to ``build``.""" + repo, _ = _make_zo_dir_project(tmp_path) + with patch("zo.cli._zo_root", return_value=tmp_path / "zo"), \ + patch("zo.cli.build") as build_cmd: + result = runner.invoke( + cli, ["continue", "--repo", str(repo), "--no-watchdog"], + ) + assert result.exit_code == 0, result.output + assert build_cmd.call_args.kwargs["no_watchdog"] is True + + def test_continue_default_forwards_watchdog_enabled( + self, runner: click.testing.CliRunner, tmp_path: Path, + ) -> None: + repo, _ = _make_zo_dir_project(tmp_path) + with patch("zo.cli._zo_root", return_value=tmp_path / "zo"), \ + patch("zo.cli.build") as build_cmd: + result = runner.invoke(cli, ["continue", "--repo", str(repo)]) + assert result.exit_code == 0, result.output + assert build_cmd.call_args.kwargs["no_watchdog"] is False + + def test_launch_and_monitor_forwards_watchdog_kwargs_to_wait( + self, tmp_path: Path, + ) -> None: + """Wiring: ``_launch_and_monitor`` passes watchdog/memory_root/zo_session_id + to ``wait_for_completion`` as keyword arguments.""" + from zo.cli import _launch_and_monitor + from zo.watchdog import WatchdogConfig + + wrapper = _StubWrapper() + cfg = WatchdogConfig(enabled=False) + _launch_and_monitor( + wrapper=wrapper, prompt="P", team_name="zo-churn-prediction", + zo_root=tmp_path / "zo", no_tmux=True, model="opus", + project_name="churn-prediction", + watchdog=cfg, memory_root=tmp_path / "mem", zo_session_id="s-abc", + ) + assert wrapper.wait_kwargs["watchdog"] is cfg + assert wrapper.wait_kwargs["watchdog"].enabled is False + assert wrapper.wait_kwargs["memory_root"] == tmp_path / "mem" + assert wrapper.wait_kwargs["zo_session_id"] == "s-abc" + + def test_launch_and_monitor_defaults_watchdog_kwargs_to_none( + self, tmp_path: Path, + ) -> None: + """Callers that do not thread a watchdog (report/draft/init) pass None.""" + from zo.cli import _launch_and_monitor + + wrapper = _StubWrapper() + _launch_and_monitor( + wrapper=wrapper, prompt="P", team_name="draft-x", + zo_root=tmp_path / "zo", no_tmux=True, model="opus", + ) + assert wrapper.wait_kwargs["watchdog"] is None + assert wrapper.wait_kwargs["memory_root"] is None + assert wrapper.wait_kwargs["zo_session_id"] == "" + + def _run_with_final_status(self, tmp_path: Path, final: _StubProcess) -> str: + from io import StringIO + + from rich.console import Console + + import zo.cli as cli_module + from zo.cli import _launch_and_monitor + + buf = StringIO() + original = cli_module.console + cli_module.console = Console(file=buf, force_terminal=False, width=200) + try: + _launch_and_monitor( + wrapper=_StubWrapper(final=final), prompt="P", + team_name="zo-churn-prediction", zo_root=tmp_path / "zo", + no_tmux=True, model="opus", project_name="churn-prediction", + ) + finally: + cli_module.console = original + return buf.getvalue() + + def test_seeded_stalled_status_prints_stalled_message(self, tmp_path: Path) -> None: + """Seeded: the wrapper returns STALLED → operator sees the escalation line.""" + out = self._run_with_final_status(tmp_path, _StubProcess(status="stalled")) + assert "Session stalled" in out + assert "watchdog escalated" in out + assert "Session ended with status" not in out + + def test_seeded_rate_limited_status_prints_resume_hint(self, tmp_path: Path) -> None: + """Seeded: RATE_LIMITED with a parsed ``resume_at`` → 'resets at … zo continue'.""" + from datetime import UTC, datetime + + final = _StubProcess( + status="rate_limited", resume_at=datetime(2026, 8, 17, 15, 0, tzinfo=UTC), + ) + out = self._run_with_final_status(tmp_path, final) + assert "rate-limited" in out + assert "resets at 2026-08-17 15:00" in out + assert "zo continue" in out + + def test_rate_limited_without_resume_at_still_actionable(self, tmp_path: Path) -> None: + out = self._run_with_final_status(tmp_path, _StubProcess(status="rate_limited")) + assert "reset time unknown" in out + assert "zo continue" in out + + def test_completed_status_unchanged(self, tmp_path: Path) -> None: + out = self._run_with_final_status(tmp_path, _StubProcess(status="completed")) + assert "Session completed." in out + + def test_agent_status_enum_members_match_string_branches(self) -> None: + """The wrapper's enum values hit the CLI branches by string value.""" + from zo._wrapper_models import AgentStatus + + assert str(AgentStatus.STALLED) == "stalled" + assert str(AgentStatus.RATE_LIMITED) == "rate_limited" + assert str(AgentStatus.PAUSED_RATE_LIMIT) == "paused_rate_limit" diff --git a/tests/unit/test_hookkit.py b/tests/unit/test_hookkit.py index dcbcc3b..f7eaa7f 100644 --- a/tests/unit/test_hookkit.py +++ b/tests/unit/test_hookkit.py @@ -5,14 +5,22 @@ fail-open test proving infrastructure problems never block a session. Handlers are driven through ``main()`` with stdin/stdout patched — the same interface the bash shim uses. + +WS-C (plan oracle checks 11-12) adds the ``heartbeat`` writer: the JSON it +stamps is validated against ``zo.watchdog.HeartbeatRecord`` here (tests may +import pydantic models; the handler itself must not). """ from __future__ import annotations import io import json +import os import subprocess -from typing import TYPE_CHECKING +import sys +import time +from datetime import UTC, datetime +from pathlib import Path import pytest @@ -24,9 +32,19 @@ WorkflowDecomposition, ) from zo.contracts import emit_contracts +from zo.watchdog import HeartbeatRecord, HeartbeatStatus + -if TYPE_CHECKING: - from pathlib import Path +@pytest.fixture(autouse=True) +def _sandbox_roots(tmp_path: Path, monkeypatch) -> None: + """Never let a handler resolve the real repo/memory root from cwd. + + Handlers fall back to ``os.getcwd()`` (``_repo_root``) and then to + ``<repo>/memory/zo-platform`` — under pytest that is the live platform + repo. Individual tests override these when they need specific roots. + """ + monkeypatch.setenv("ZO_REPO_ROOT", str(tmp_path / "_repo")) + monkeypatch.setenv("ZO_MEMORY_ROOT", str(tmp_path / "_mem")) def _run(event: str, payload: dict, monkeypatch, capsys) -> dict | None: @@ -270,6 +288,35 @@ def test_two_failures_two_lines(self, tmp_path: Path, monkeypatch, capsys): lines = list(feed.glob("failures-*.jsonl"))[0].read_text().splitlines() assert len(lines) == 2 + def test_record_carries_is_interrupt_and_identity( + self, tmp_path: Path, monkeypatch, capsys, + ): + """WS-C: user-abort evidence (``is_interrupt``) + agent identity are + kept on the failure record — the live payload carries all three.""" + feed = tmp_path / "feed" + monkeypatch.setenv("ZO_FAILURE_FEED_DIR", str(feed)) + payload = { + "session_id": "s-4", "tool_name": "Bash", "error": "interrupted", + "is_interrupt": True, "agent_id": "agent-7", "agent_type": "model-builder", + } + _run("post-tool-failure", payload, monkeypatch, capsys) + record = json.loads(next(feed.glob("failures-*.jsonl")).read_text().splitlines()[0]) + assert record["is_interrupt"] is True + assert record["agent_id"] == "agent-7" + assert record["agent_type"] == "model-builder" + # existing fields untouched + assert record["event_type"] == "error" and record["tool_name"] == "Bash" + + def test_record_identity_fields_default_none( + self, tmp_path: Path, monkeypatch, capsys, + ): + feed = tmp_path / "feed" + monkeypatch.setenv("ZO_FAILURE_FEED_DIR", str(feed)) + _run("post-tool-failure", {"tool_name": "A", "error": "x"}, monkeypatch, capsys) + record = json.loads(next(feed.glob("failures-*.jsonl")).read_text().splitlines()[0]) + assert record["is_interrupt"] is None + assert record["agent_id"] is None and record["agent_type"] is None + # ---- sealed-paths (oracle check 7) ------------------------------------------- @@ -322,6 +369,23 @@ def test_off_limits_write_denied_for_contracted_agent( assert out["hookSpecificOutput"]["permissionDecision"] == "deny" assert "off-limits" in out["hookSpecificOutput"]["permissionDecisionReason"] + def test_seeded_write_into_heartbeats_denied( + self, tmp_path: Path, memory_root: Path, monkeypatch, capsys, + ): + """WS-C: agents must not forge liveness — the whole heartbeats + subtree is sealed (prefix match), not just a single file.""" + monkeypatch.setenv("ZO_REPO_ROOT", str(tmp_path)) + monkeypatch.setenv("ZO_MEMORY_ROOT", str(memory_root)) + for target in ("heartbeats/x.json", "heartbeats/nested/agent-1.json"): + out = _run( + "sealed-paths", + {"tool_input": {"file_path": str(memory_root / target)}}, + monkeypatch, capsys, + ) + assert out is not None, target + assert out["hookSpecificOutput"]["permissionDecision"] == "deny" + assert "Sealed path" in out["hookSpecificOutput"]["permissionDecisionReason"] + def test_ordinary_write_is_silent( self, tmp_path: Path, memory_root: Path, monkeypatch, capsys, ): @@ -415,3 +479,226 @@ def test_inline_last_message_used_over_transcript( ) assert out is not None assert out["decision"] == "block" + + +# ---- heartbeat writer (oracle check 11) -------------------------------------- + + +@pytest.fixture() +def hb_root(tmp_path: Path, monkeypatch) -> Path: + """Sandboxed memory root for heartbeat writes; strips lead-env correlation.""" + mem = tmp_path / "mem" + mem.mkdir() + monkeypatch.setenv("ZO_REPO_ROOT", str(tmp_path / "repo")) + monkeypatch.setenv("ZO_MEMORY_ROOT", str(mem)) + for name in ("ZO_SESSION_ID", "ZO_LEAD_PID", "ZO_LEAD_PID_IDENTITY"): + monkeypatch.delenv(name, raising=False) + return mem + + +def _hb_load(mem: Path, key: str) -> HeartbeatRecord: + """Read a heartbeat file and validate it against the watchdog model.""" + return HeartbeatRecord.model_validate_json( + (mem / "heartbeats" / f"{key}.json").read_text(encoding="utf-8") + ) + + +def _age_file(path: Path, seconds: float) -> None: + stamp = time.time() - seconds + os.utime(path, (stamp, stamp)) + + +_SUBAGENT_PAYLOAD = { + "hook_event_name": "PostToolUse", "session_id": "s-hb", "tool_name": "Bash", + "agent_id": "agent-42", "agent_type": "data-engineer", +} + + +class TestHeartbeat: + """(a) the writer produces exactly the ``HeartbeatRecord`` shape and + (b) it is fail-open — never output, never a non-zero exit.""" + + def test_subagent_payload_keyed_by_agent_id(self, hb_root: Path, monkeypatch, capsys): + assert _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) is None + rec = _hb_load(hb_root, "agent-42") + assert rec.agent_key == "agent-42" + assert rec.agent_id == "agent-42" and rec.agent_type == "data-engineer" + assert rec.session_id == "s-hb" + assert rec.status is HeartbeatStatus.EXECUTING + assert rec.last_event == "Bash" + assert rec.tick_count == 1 + assert rec.schema_version == 1 + assert rec.last_tick_at.tzinfo is not None + assert abs((datetime.now(UTC) - rec.last_tick_at).total_seconds()) < 30 + + def test_exact_field_set_matches_heartbeat_record(self, hb_root: Path, monkeypatch, capsys): + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + raw = json.loads((hb_root / "heartbeats" / "agent-42.json").read_text()) + assert set(raw) == set(HeartbeatRecord.model_fields) + assert raw["last_tick_at"].endswith("+00:00") + + def test_lead_payload_keyed_by_session_id(self, hb_root: Path, monkeypatch, capsys): + payload = {"hook_event_name": "PostToolUse", "session_id": "s-lead", "tool_name": "Read"} + _run("heartbeat", payload, monkeypatch, capsys) + rec = _hb_load(hb_root, "lead-s-lead") + assert rec.agent_key == "lead-s-lead" + assert rec.agent_id is None and rec.agent_type is None + assert rec.last_event == "Read" + + def test_tick_count_increments(self, hb_root: Path, monkeypatch, capsys): + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + _age_file(hb_root / "heartbeats" / "agent-42.json", 10) + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + assert _hb_load(hb_root, "agent-42").tick_count == 2 + + def test_seeded_post_tool_use_burst_is_debounced(self, hb_root: Path, monkeypatch, capsys): + """Two PostToolUse stamps inside 2 s collapse to one write; a Stop + event immediately after still stamps (only PostToolUse debounces).""" + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + assert _hb_load(hb_root, "agent-42").tick_count == 1 + stop = {**_SUBAGENT_PAYLOAD, "hook_event_name": "Stop"} + _run("heartbeat", stop, monkeypatch, capsys) + rec = _hb_load(hb_root, "agent-42") + assert rec.tick_count == 2 and rec.status is HeartbeatStatus.READY + + def test_corrupt_existing_file_restarts_count(self, hb_root: Path, monkeypatch, capsys): + hb_dir = hb_root / "heartbeats" + hb_dir.mkdir() + (hb_dir / "agent-42.json").write_text("{not json") + _age_file(hb_dir / "agent-42.json", 10) + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + assert _hb_load(hb_root, "agent-42").tick_count == 1 + + @pytest.mark.parametrize( + ("event", "hook_event", "status"), + [ + ("heartbeat", "PostToolUse", HeartbeatStatus.EXECUTING), + ("drift-guard", "Stop", HeartbeatStatus.READY), + ("subagent-stop", "SubagentStop", HeartbeatStatus.SHUTDOWN), + ("precompact", "PreCompact", HeartbeatStatus.COMPACTING), + ("session-end", "SessionEnd", HeartbeatStatus.SHUTDOWN), + ], + ) + def test_status_mapping_per_event_via_wired_handlers( + self, hb_root: Path, monkeypatch, capsys, event: str, hook_event: str, status, + ): + """Wiring: every routed handler stamps a heartbeat with the status + the contract assigns to its hook event; ``last_event`` is the + hook_event_name for non-PostToolUse events.""" + payload = {"hook_event_name": hook_event, "session_id": "s-map", "tool_name": "Bash"} + assert _run(event, payload, monkeypatch, capsys) is None + rec = _hb_load(hb_root, "lead-s-map") + assert rec.status is status + assert rec.last_event == ("Bash" if hook_event == "PostToolUse" else hook_event) + + @pytest.mark.parametrize( + ("event", "status"), + [ + ("drift-guard", HeartbeatStatus.READY), + ("subagent-stop", HeartbeatStatus.SHUTDOWN), + ("precompact", HeartbeatStatus.COMPACTING), + ("session-end", HeartbeatStatus.SHUTDOWN), + ], + ) + def test_handlers_stamp_without_hook_event_name( + self, hb_root: Path, monkeypatch, capsys, event: str, status, + ): + """Older payload shapes lack ``hook_event_name`` — the routed handler + knows its own event and still maps the status correctly.""" + _run(event, {"session_id": "s-old"}, monkeypatch, capsys) + assert _hb_load(hb_root, "lead-s-old").status is status + + def test_stamping_does_not_change_drift_guard_output( + self, git_repo: Path, tmp_path: Path, monkeypatch, capsys, + ): + """The Stop hook still blocks on claim+stub AND a heartbeat lands.""" + mem = tmp_path / "mem" + mem.mkdir() + (git_repo / "mod.py").write_text("def f():\n # TODO: later\n return 1\n") + monkeypatch.setenv("ZO_REPO_ROOT", str(git_repo)) + monkeypatch.setenv("ZO_MEMORY_ROOT", str(mem)) + out = _run( + "drift-guard", + {"session_id": "s-dg", "last_assistant_message": "All tests pass, task complete."}, + monkeypatch, capsys, + ) + assert out is not None and out["decision"] == "block" + assert _hb_load(mem, "lead-s-dg").status is HeartbeatStatus.READY + + def test_env_correlation_fields(self, hb_root: Path, monkeypatch, capsys): + monkeypatch.setenv("ZO_SESSION_ID", "zo-abc") + monkeypatch.setenv("ZO_LEAD_PID", "4242") + monkeypatch.setenv("ZO_LEAD_PID_IDENTITY", "darwin:1700000000:0") + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + rec = _hb_load(hb_root, "agent-42") + assert rec.zo_session_id == "zo-abc" + assert rec.pid == 4242 + assert rec.process_start_identity == "darwin:1700000000:0" + + def test_env_correlation_fields_absent_are_none(self, hb_root: Path, monkeypatch, capsys): + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + rec = _hb_load(hb_root, "agent-42") + assert rec.zo_session_id is None and rec.pid is None + assert rec.process_start_identity is None + + def test_agent_key_is_sanitised_for_filesystem(self, hb_root: Path, monkeypatch, capsys): + payload = {**_SUBAGENT_PAYLOAD, "agent_id": "../evil/agent"} + _run("heartbeat", payload, monkeypatch, capsys) + files = sorted(p.name for p in (hb_root / "heartbeats").glob("*.json")) + assert files == [".._evil_agent.json"] + assert not (hb_root / "evil").exists() + + def test_atomic_write_leaves_no_tmp_file(self, hb_root: Path, monkeypatch, capsys): + _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) + names = [p.name for p in (hb_root / "heartbeats").iterdir()] + assert names == ["agent-42.json"] + + def test_fail_open_without_memory_root(self, tmp_path: Path, monkeypatch, capsys): + repo = tmp_path / "repo" + repo.mkdir() + monkeypatch.setenv("ZO_REPO_ROOT", str(repo)) + monkeypatch.delenv("ZO_MEMORY_ROOT", raising=False) + assert _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) is None + assert not list(tmp_path.rglob("heartbeats")) + + def test_fail_open_when_memory_root_missing_dir(self, tmp_path: Path, monkeypatch, capsys): + monkeypatch.setenv("ZO_REPO_ROOT", str(tmp_path)) + monkeypatch.setenv("ZO_MEMORY_ROOT", str(tmp_path / "nowhere")) + assert _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) is None + assert not (tmp_path / "nowhere").exists() + + def test_fail_open_when_heartbeats_dir_unwritable( + self, hb_root: Path, monkeypatch, capsys, + ): + (hb_root / "heartbeats").write_text("i am a file, not a directory") + assert _run("heartbeat", _SUBAGENT_PAYLOAD, monkeypatch, capsys) is None + + def test_heartbeat_path_imports_no_pydantic(self, hb_root: Path): + """Perf contract: the PostToolUse path must not pay the pydantic + import — the whole handler is stdlib-only (contracts import lazy).""" + code = ( + "import io, json, sys\n" + f"sys.stdin = io.StringIO(json.dumps({_SUBAGENT_PAYLOAD!r}))\n" + "import zo.hookkit as h\n" + "assert h.main(['heartbeat']) == 0\n" + "assert 'pydantic' not in sys.modules, 'pydantic imported'\n" + "assert 'zo.watchdog' not in sys.modules, 'watchdog imported'\n" + ) + src = Path(hookkit.__file__).resolve().parents[1] + env = {**os.environ, "PYTHONPATH": str(src)} + result = subprocess.run( + [sys.executable, "-c", code], capture_output=True, text=True, env=env, + timeout=60, check=False, + ) + assert result.returncode == 0, result.stderr + assert (hb_root / "heartbeats" / "agent-42.json").exists() + + def test_agent_identity_helper(self): + assert hookkit._agent_identity({}) == (None, None) + assert hookkit._agent_identity( + {"agent_type": " builder ", "agent_id": "a-1"} + ) == ("builder", "a-1") + assert hookkit._agent_identity({"subagent_type": "x", "agent_id": ""}) == ("x", None) + # _agent_name is unchanged (contract lookup key) + assert hookkit._agent_name({"agent_name": "data-engineer"}) == "data-engineer" diff --git a/tests/unit/test_project_config.py b/tests/unit/test_project_config.py index 5e34200..002c37e 100644 --- a/tests/unit/test_project_config.py +++ b/tests/unit/test_project_config.py @@ -1,11 +1,17 @@ -"""Unit tests for zo.project_config — .zo/ config reader/writer.""" +"""Unit tests for zo.project_config — .zo/ config reader/writer. + +Includes the WS-C watchdog config threading (plan oracle checks 11-12): +the ``watchdog:`` block round-trips through save/load, legacy configs +without it get defaults, and unknown top-level keys are ignored. +""" from __future__ import annotations -from pathlib import Path +from typing import TYPE_CHECKING import pytest import yaml +from pydantic import ValidationError from zo.project_config import ( LocalConfig, @@ -18,6 +24,10 @@ to_target_config, ) +if TYPE_CHECKING: + from pathlib import Path +from zo.watchdog import WatchdogConfig + # --------------------------------------------------------------------------- # Model defaults # --------------------------------------------------------------------------- @@ -39,6 +49,10 @@ def test_defaults(self) -> None: assert cfg.git_author_name == "ZO Agent" assert cfg.git_author_email == "zo-agent@zero-operators.dev" assert cfg.enforce_isolation is True + # WS-C: watchdog block defaults to ON with the spec thresholds. + assert isinstance(cfg.watchdog, WatchdogConfig) + assert cfg.watchdog.enabled is True + assert cfg.watchdog.stall_threshold_sec == 1200 class TestLocalConfigDefaults: @@ -253,3 +267,118 @@ def test_yaml_is_readable(self, tmp_path: Path) -> None: data = yaml.safe_load(raw) assert data["project_name"] == "readable" assert isinstance(data["agent_working_dirs"], dict) + + +# ---- watchdog config threading (oracle checks 11-12, WS-C) ---- + + +def _write_raw_config(repo: Path, text: str) -> None: + zo_dir = repo / ".zo" + zo_dir.mkdir(parents=True, exist_ok=True) + (zo_dir / "config.yaml").write_text(text, encoding="utf-8") + + +class TestWatchdogConfigThreading: + """The ``watchdog:`` block is a first-class, round-tripping part of ProjectConfig. + + Seeded half: a config carrying a non-default watchdog block (and one with + a typo) is written to disk and must come back intact / be rejected. + Wiring half: legacy files without the block, and files with unknown + top-level keys, still load (documented ``extra="ignore"``). + """ + + def test_watchdog_block_round_trip(self, tmp_path: Path) -> None: + """A non-default watchdog block survives save → load unchanged.""" + original = ProjectConfig( + project_name="wd-round-trip", + watchdog=WatchdogConfig( + enabled=False, + stall_threshold_sec=600, + nudge_budget=1, + nudge_message="carry on", + progress_paths=["logs/train.log", "runs/"], + ), + ) + save_project_config(tmp_path, original) + loaded = load_project_config(tmp_path) + + assert loaded.watchdog == original.watchdog + assert loaded.watchdog.enabled is False + assert loaded.watchdog.stall_threshold_sec == 600 + assert loaded.watchdog.progress_paths == ["logs/train.log", "runs/"] + + def test_saved_yaml_has_nested_watchdog_block(self, tmp_path: Path) -> None: + """save_project_config writes the block as nested YAML, not a repr.""" + save_project_config( + tmp_path, + ProjectConfig( + project_name="wd-yaml", + watchdog=WatchdogConfig(stall_threshold_sec=900), + ), + ) + raw = (tmp_path / ".zo" / "config.yaml").read_text(encoding="utf-8") + data = yaml.safe_load(raw) + + assert "watchdog:" in raw + assert isinstance(data["watchdog"], dict) + assert data["watchdog"]["stall_threshold_sec"] == 900 + assert data["watchdog"]["enabled"] is True + + def test_seeded_watchdog_disabled_in_file_loads_disabled(self, tmp_path: Path) -> None: + """Seeded: a hand-written ``watchdog: {enabled: false}`` is honoured on load.""" + _write_raw_config( + tmp_path, + "project_name: seeded-off\nwatchdog:\n enabled: false\n nudge_budget: 0\n", + ) + loaded = load_project_config(tmp_path) + + assert loaded.watchdog.enabled is False + assert loaded.watchdog.nudge_budget == 0 + # Unspecified nested keys keep their defaults. + assert loaded.watchdog.stall_threshold_sec == 1200 + + def test_legacy_config_without_watchdog_block_gets_defaults(self, tmp_path: Path) -> None: + """A pre-WS-C config.yaml (no watchdog key) loads with the watchdog ON.""" + _write_raw_config( + tmp_path, + "project_name: legacy\nalias: prod-001\nbranch: main\n", + ) + loaded = load_project_config(tmp_path) + + assert loaded.alias == "prod-001" + assert loaded.watchdog == WatchdogConfig() + assert loaded.watchdog.enabled is True + + def test_unknown_top_level_key_is_ignored(self, tmp_path: Path) -> None: + """Documented choice: unknown top-level keys are ignored, not an error.""" + _write_raw_config( + tmp_path, + "project_name: forward-compat\nfuture_feature: {x: 1}\nwatchdog:\n enabled: true\n", + ) + loaded = load_project_config(tmp_path) + + assert loaded.project_name == "forward-compat" + assert not hasattr(loaded, "future_feature") + assert loaded.watchdog.enabled is True + + def test_seeded_watchdog_typo_is_rejected(self, tmp_path: Path) -> None: + """Seeded: a misspelt key INSIDE the watchdog block is a hard error. + + ``WatchdogConfig`` forbids extras so a policy typo cannot silently + turn into "defaults, watchdog on" — the opposite of what the operator + intended. + """ + _write_raw_config( + tmp_path, + "project_name: typo\nwatchdog:\n stall_treshold_sec: 5\n", + ) + with pytest.raises(ValidationError): + load_project_config(tmp_path) + + def test_watchdog_model_dump_is_plain_data(self) -> None: + """model_dump() nests the block as a plain dict (what yaml.dump needs).""" + dumped = ProjectConfig(project_name="dump").model_dump() + + assert isinstance(dumped["watchdog"], dict) + assert dumped["watchdog"]["enabled"] is True + assert dumped["watchdog"]["progress_paths"] == [] diff --git a/tests/unit/test_scaffold.py b/tests/unit/test_scaffold.py index c0c841f..dcad787 100644 --- a/tests/unit/test_scaffold.py +++ b/tests/unit/test_scaffold.py @@ -31,6 +31,11 @@ def test_zo_gitignore_written(self, tmp_path: Path) -> None: assert "local.yaml" in content assert "memory/index.db" in content assert "memory/draft_index.db" in content + # Control-plane runtime files (watchdog heartbeats/state/trace, ledger, + # contracts) are regenerated per run and must never enter delivery history. + assert "memory/heartbeats/" in content.split() + assert "memory/plan-ledger.json" in content.split() + assert "memory/contracts.json" in content.split() def test_zo_directories_created_in_adaptive_mode( self, tmp_path: Path, diff --git a/tests/unit/test_watchdog.py b/tests/unit/test_watchdog.py new file mode 100644 index 0000000..ea36290 --- /dev/null +++ b/tests/unit/test_watchdog.py @@ -0,0 +1,406 @@ +"""Tests for zo.watchdog — the WS-C execution substrate (plan oracle checks 11-12). + +Covers the never-block taxonomy, rate-limit reset parsing, heartbeat +freshness, evidence observers, process identity (positive proof only), +persistence and config. The ``evaluate()`` scenario table (seeded 10-min +stall, rate-limited-never-nudged, check-12 resume, …) lives in +``test_watchdog_policy.py``. +""" + +from __future__ import annotations + +import errno +import json +import os +from datetime import UTC, datetime, timedelta, timezone +from types import SimpleNamespace +from typing import TYPE_CHECKING +from unittest import mock + +import pytest +from pydantic import ValidationError + +from zo import _proc, watchdog +from zo.watchdog import ( + HEARTBEATS_DIRNAME, + WATCHDOG_STATE_FILENAME, + Freshness, + HeartbeatRecord, + HeartbeatStatus, + NeverBlockReason, + WatchdogConfig, + classify_freshness, + classify_never_block, + identities_may_match, + is_process_dead, + is_valid_process_start_identity, + load_all_heartbeats, + load_heartbeat, + load_state, + new_state, + normalize_terminal_text, + observe_files, + observe_heartbeats, + observe_text, + pane_ready_for_nudge, + parse_rate_limit_reset, + pid_alive, + process_start_identity, + process_tree_cpu_seconds, + progress_digest, + rate_limit_banner_key, + rate_limit_match, + resolve_watchdog_config, + save_state, + sweep_stale_heartbeats, + write_heartbeat, +) + +if TYPE_CHECKING: + from pathlib import Path + +T0 = datetime(2026, 8, 17, 14, 0, tzinfo=UTC) +BANNER = "You've hit your usage limit · resets at 3pm" +DIALOG = "Do you want to proceed?\n❯ 1. Yes\n 2. Yes, and don't ask again\n 3. No (esc)" + + +def _hb(key: str = "lead-s1", ticks: int = 1, at: datetime = T0, + status: HeartbeatStatus = HeartbeatStatus.EXECUTING) -> HeartbeatRecord: + return HeartbeatRecord(agent_key=key, session_id="s1", last_tick_at=at, + status=status, tick_count=ticks) + + +# ---- never-block taxonomy (oracle check 11) ---- + +@pytest.mark.parametrize(("text", "reason"), [ + (BANNER, NeverBlockReason.RATE_LIMIT), + ("Rate limit reached. Try again later.", NeverBlockReason.RATE_LIMIT), + ("API Error: 429 too_many_requests", NeverBlockReason.RATE_LIMIT), + ("weekly usage limit exhausted", NeverBlockReason.RATE_LIMIT), + ("prompt is too long", NeverBlockReason.CONTEXT_LIMIT), + ("Context window is full", NeverBlockReason.CONTEXT_LIMIT), + ("stop_reason: context_exceeded", NeverBlockReason.CONTEXT_LIMIT), + ("please run /login", NeverBlockReason.AUTH_ERROR), + ("authentication_error: invalid api key", NeverBlockReason.AUTH_ERROR), + ("HTTP 401 Unauthorized", NeverBlockReason.AUTH_ERROR), + ("Not logged in", NeverBlockReason.AUTH_ERROR), + (DIALOG, NeverBlockReason.AWAITING_INPUT), + ("Waiting for your approval", NeverBlockReason.AWAITING_INPUT), + ("Do you trust this folder?", NeverBlockReason.AWAITING_INPUT), + ("Press Enter to continue", NeverBlockReason.AWAITING_INPUT), + ("Interrupted by user", NeverBlockReason.USER_ABORT), + # The real TUI renders the interrupt behind a ⎿ connector, old and new wording. + ("⎿ Interrupted by user\n› ", NeverBlockReason.USER_ABORT), + ("⎿ Interrupted · What should Claude do instead?\n› ", NeverBlockReason.USER_ABORT), + ("stop_reason=user_cancel", NeverBlockReason.USER_ABORT), + ("Request aborted", NeverBlockReason.USER_ABORT), + ("manual_stop", NeverBlockReason.USER_ABORT), + # Loose rate-limit phrases DO count with rate/usage/quota vocabulary on the line. + ("Too many requests, try again later", NeverBlockReason.RATE_LIMIT), + ("API usage limit reached for this 5-hour window", NeverBlockReason.RATE_LIMIT), + ("Your usage resets in 2 hours at 5pm", NeverBlockReason.RATE_LIMIT), + ("hit the rate limit; backing off", NeverBlockReason.RATE_LIMIT), +]) +def test_taxonomy_positive(text: str, reason: NeverBlockReason) -> None: + assert classify_never_block(text) is reason + + +@pytest.mark.parametrize("text", [ + "epoch 3 val_loss 0.4291", "step 4290 loss 0.12", "GPU overloaded, retrying batch", + "commit 8466d29 fix weekly report", "$ cat transcript.txt ... rate limit", + "turn interrupted", "x = arr[0]\n1. Read the file\n2. Edit it", " › ", + "\x1b[32mtests passed\x1b[0m 401 items collected", + # ML / prose lines that carry a loose token WITHOUT rate-limit vocabulary. + "early stopping: patience limit reached at epoch 30", + "Service temporarily unavailable, try again later", + "Analysis: the 5-hour window in the plan means two full epochs", + "The nightly counter resets every day at midnight", + "we hit the recursion limit in the parser", +]) +def test_taxonomy_negative(text: str) -> None: + assert classify_never_block(text) is None + + +def test_rate_limit_match_tiers_and_banner_key() -> None: + assert rate_limit_match(BANNER) == "banner" + assert rate_limit_match("Error 429 Too Many Requests") == "banner" + assert rate_limit_match("the API hit a rate limit earlier and retried") == "prose" + assert rate_limit_match("Too many requests, try again later") == "banner" + assert rate_limit_match("quota: try again later") == "loose" + assert rate_limit_match("patience limit reached at epoch 30") is None + key = rate_limit_banner_key("work\n" + BANNER + "\n› ") + assert key == BANNER + assert rate_limit_banner_key("work\n› ") == "" + two = rate_limit_banner_key(BANNER + "\n› continue\n" + BANNER.replace("3pm", "5pm")) + assert two.count("resets at") == 2 and two != key + + +def test_taxonomy_precedence_and_interrupt() -> None: + assert classify_never_block(BANNER + "\n" + DIALOG) is NeverBlockReason.RATE_LIMIT + assert classify_never_block("prompt is too long\n" + BANNER) is NeverBlockReason.CONTEXT_LIMIT + assert classify_never_block("aborted\n" + BANNER) is NeverBlockReason.USER_ABORT + assert classify_never_block("healthy", is_interrupt=True) is NeverBlockReason.USER_ABORT + old = "\n".join([BANNER] + ["line"] * 70) # banner scrolled past the 60-line tail + assert classify_never_block(old) is None + + +def test_taxonomy_compacting_heartbeat_window() -> None: + inside = _hb(at=T0 - timedelta(seconds=60), status=HeartbeatStatus.COMPACTING) + outside = _hb(at=T0 - timedelta(seconds=600), status=HeartbeatStatus.COMPACTING) + assert classify_never_block("", heartbeats=[inside], now=T0) is NeverBlockReason.COMPACTING + assert classify_never_block("", heartbeats=[outside], now=T0) is None + assert classify_never_block(BANNER, heartbeats=[inside], now=T0) is NeverBlockReason.RATE_LIMIT + + +def test_normalize_and_digest_ignore_volatile_churn() -> None: + assert "commit" not in normalize_terminal_text("commit abcdef1234 msg\nreal\r\n") + a = progress_digest("✻ Thinking… (12s · ↓ 1.2k tokens · esc to interrupt)\nwork\n› ") + b = progress_digest("✽ Thinking… (14s · ↓ 1.3k tokens · esc to interrupt)\nwork\n\n> ") + assert a == b + assert a != progress_digest("work\nmore work\n› ") + + +def test_pane_ready_for_nudge_guards() -> None: + assert pane_ready_for_nudge("done.\n\n› ") + assert pane_ready_for_nudge("done.\n│ > ") + assert not pane_ready_for_nudge("· Thinking…\n› ") + assert not pane_ready_for_nudge("working (esc to interrupt)\n> ") + assert not pane_ready_for_nudge(DIALOG) + assert not pane_ready_for_nudge("no prompt at all") + assert not pane_ready_for_nudge("") + + +# ---- rate-limit reset parsing (oracle check 12) ---- + +@pytest.mark.parametrize(("text", "expected"), [ + ("resets at 3pm", datetime(2026, 8, 17, 15, 0, tzinfo=UTC)), + ("resets at 1pm", datetime(2026, 8, 18, 13, 0, tzinfo=UTC)), # already past → tomorrow + ("resets at 14:30", datetime(2026, 8, 17, 14, 30, tzinfo=UTC)), + ("try again in 5 minutes", T0 + timedelta(minutes=5)), + ("resets in 2 hours", T0 + timedelta(hours=2)), + ("retry-after: 90", T0 + timedelta(seconds=90)), + ("limit resets 2026-08-17T16:00:00Z", datetime(2026, 8, 17, 16, 0, tzinfo=UTC)), + ("no reset info here", None), + ("resets 3", None), +]) +def test_parse_rate_limit_reset(text: str, expected: datetime | None) -> None: + assert parse_rate_limit_reset(text, now=T0) == expected + + +def test_parse_rate_limit_reset_uses_tz() -> None: + tz = datetime.now().astimezone().tzinfo + got = parse_rate_limit_reset("resets at 3pm", now=T0, tz=tz) + assert got is not None and got.astimezone(tz).hour == 15 + pacific = timezone(timedelta(hours=-7)) + got = parse_rate_limit_reset("resets at 3pm", now=datetime(2026, 8, 17, 20, 0, tzinfo=UTC), + tz=pacific) + assert got == datetime(2026, 8, 17, 22, 0, tzinfo=UTC) # 3pm PDT today, not tomorrow UTC + + +def test_parse_rate_limit_reset_newest_banner_wins() -> None: + """Transcripts grow downward: the LAST reset in a tier is the current one.""" + text = BANNER + "\n› continue\nYou've hit your usage limit · resets at 5pm" + assert parse_rate_limit_reset(text, now=T0) == datetime(2026, 8, 17, 17, 0, tzinfo=UTC) + text = "try again in 5 minutes\n...\ntry again in 20 minutes" + assert parse_rate_limit_reset(text, now=T0) == T0 + timedelta(minutes=20) + + +# ---- heartbeat freshness (oracle check 11) ---- + +def test_freshness_three_state() -> None: + assert classify_freshness(None, now=T0, stale_after_sec=60) is Freshness.UNKNOWN + fresh = _hb(at=T0 - timedelta(seconds=30)) + assert classify_freshness(fresh, now=T0, stale_after_sec=60) is Freshness.FRESH + naive = _hb(at=(T0 - timedelta(seconds=90)).replace(tzinfo=None)) + assert classify_freshness(naive, now=T0, stale_after_sec=60) is Freshness.STALE + + +# ---- evidence observers (oracle check 11) ---- + +def test_observe_heartbeats_ignores_baseline_files() -> None: + state = new_state(now=T0, heartbeats=[_hb("old", ticks=5)]) + assert observe_heartbeats(state, [_hb("old", ticks=5)]) is False + assert observe_heartbeats(state, [_hb("old", ticks=6)]) is True + assert observe_heartbeats(state, [_hb("old", ticks=6)]) is False + assert observe_heartbeats(state, [_hb("new", ticks=1)]) is True + assert state.seen_ticks == {"old": 6, "new": 1} + + +def test_observe_text_digest_change_ignoring_churn() -> None: + state = new_state(now=T0) + assert observe_text(state, "⠋ Working (12s · 100 tokens)\n› ") is False # first observation + assert observe_text(state, "⠙ Working (14s · 200 tokens)\n› ") is False # spinner churn + assert observe_text(state, "⠙ Working\nwrote file\n› ") is True + state.paused_at = T0 + assert observe_text(state, "banner gone\n› ") is False # paused: text is not verified resume + + +def test_observe_files_mtime_advance_and_new_file(tmp_path: Path) -> None: + f, d = tmp_path / "ledger.json", tmp_path / "logs" + f.write_text("{}"), d.mkdir() + state = new_state(now=T0) + assert observe_files(state, [f, d, tmp_path / "missing"]) is False + os.utime(f, (2_000_000_000, 2_000_000_000)) + assert observe_files(state, [f, d]) is True + assert observe_files(state, [f, d]) is False + child = d / "x.jsonl" + child.write_text("1"), os.utime(child, (2_100_000_000, 2_100_000_000)) + assert observe_files(state, [d]) is True + (tmp_path / "missing").write_text("now here") + assert observe_files(state, [tmp_path / "missing"]) is True + + +# ---- process identity: positive proof only (oracle check 11) ---- + +LINUX_STAT = ("4242 (weird proc) name) S 1 4242 4242 0 -1 4194560 100 0 0 0 5 3 0 0 20 0 1 0 " + "987654 1 2 3") + + +def test_identity_linux_proc_parse_with_paren_in_comm() -> None: + assert _proc.parse_linux_stat_starttime(LINUX_STAT) == "987654" + with mock.patch.object(_proc, "_read_proc_stat", return_value=LINUX_STAT): + assert process_start_identity(4242, platform="linux") == "linux:987654" + with mock.patch.object(_proc, "_read_proc_stat", return_value="garbage"): + assert process_start_identity(4242, platform="linux") is None + with mock.patch.object(_proc, "_read_proc_stat", side_effect=FileNotFoundError): + assert process_start_identity(4242, platform="linux") is None + + +def test_identity_darwin_ps_mocked() -> None: + def run(cmd: list[str], **kw: object) -> SimpleNamespace: + assert cmd[:3] == ["ps", "-o", "lstart="] and kw["env"]["LC_ALL"] == "C" + return SimpleNamespace(stdout="Sun Aug 17 10:11:12 2026\n", returncode=0) + + got = process_start_identity(77, platform="darwin", run=run) + assert got is not None and got.startswith("darwin:") and got.endswith(":0") + assert is_valid_process_start_identity(got, platform="darwin") + junk = lambda *a, **k: SimpleNamespace(stdout="?") # noqa: E731 + assert process_start_identity(77, platform="darwin", run=junk) is None + other = process_start_identity(77, platform="freebsd", run=run) + assert other == "freebsd:Sun Aug 17 10:11:12 2026" + + +def test_identity_validator_and_matching() -> None: + assert is_valid_process_start_identity("linux:12345", platform="linux") + assert not is_valid_process_start_identity("linux:0", platform="linux") + assert not is_valid_process_start_identity("darwin:1:0", platform="linux") + assert not is_valid_process_start_identity("x" * 1025, platform="linux") + assert not is_valid_process_start_identity(None, platform="darwin") + assert identities_may_match("darwin:1700000000:0", "darwin:1700000000:4321") + assert not identities_may_match("darwin:1700000000:1", "darwin:1700000000:2") + assert not identities_may_match("linux:1", "linux:2") + + +def test_is_process_dead_positive_proof_only() -> None: + assert is_process_dead(None, "linux:1") is False + with mock.patch("os.kill", side_effect=ProcessLookupError): + assert is_process_dead(4242, None) is True # ESRCH is positive proof + assert pid_alive(4242) is False + with mock.patch("os.kill", side_effect=PermissionError(errno.EPERM, "eperm")): + assert pid_alive(4242) is True + assert is_process_dead(4242, "linux:1", platform="linux") is False # unknown ≠ dead + with mock.patch("os.kill", return_value=None), \ + mock.patch.object(_proc, "_read_proc_stat", return_value=LINUX_STAT): + assert is_process_dead(4242, "linux:987654", platform="linux") is False + assert is_process_dead(4242, "linux:111", platform="linux") is True # recycled pid + assert is_process_dead(4242, "malformed", platform="linux") is False + with mock.patch("os.kill", return_value=None), \ + mock.patch.object(_proc, "_read_proc_stat", return_value="garbage"): + assert is_process_dead(4242, "linux:111", platform="linux") is False # observed unknown + + +@pytest.mark.parametrize(("value", "expected"), [ + ("0:00.05", 0.05), ("12:34.56", 754.56), ("1:02:03.04", 3723.04), # darwin + ("00:00:07", 7.0), ("01:02:03", 3723.0), ("1-02:03:04", 93784.0), # linux (+days) + ("", None), ("junk", None), ("1:2:3:4", None), +]) +def test_parse_ps_time(value: str, expected: float | None) -> None: + assert _proc.parse_ps_time(value) == expected + + +def test_process_tree_cpu_seconds_sums_descendants_and_fails_open() -> None: + table = (" 1 0 0:10.00\n" + " 4242 1 0:01.00\n" # lead + " 4300 4242 0:02.00\n" # bash tool + " 4301 4300 40:00.00\n" # python train.py (busy) + " 5000 1 9:99.99\n" # unrelated + "junk line\n") + run = lambda *a, **k: SimpleNamespace(stdout=table) # noqa: E731 + assert process_tree_cpu_seconds(4242, run=run) == 1.0 + 2.0 + 2400.0 + assert process_tree_cpu_seconds(4301, run=run) == 2400.0 + assert process_tree_cpu_seconds(7777, run=run) is None # not in the table → unknown + assert process_tree_cpu_seconds(0, run=run) is None + boom = mock.MagicMock(side_effect=OSError("no ps")) + assert process_tree_cpu_seconds(4242, run=boom) is None # advisory: never raises + + +# ---- persistence + config (oracle checks 11-12) ---- + +def test_heartbeat_round_trip_and_state_file_excluded(tmp_path: Path) -> None: + write_heartbeat(tmp_path, _hb("agent-a", ticks=3)) + write_heartbeat(tmp_path, _hb("lead-s1", ticks=1)) + hb_dir = tmp_path / HEARTBEATS_DIRNAME + (hb_dir / WATCHDOG_STATE_FILENAME).write_text('{"not": "a heartbeat"}') + (hb_dir / "broken.json").write_text("{oops") + records = load_all_heartbeats(tmp_path) + assert [r.agent_key for r in records] == ["agent-a", "lead-s1"] + assert records[0].tick_count == 3 and records[0].last_tick_at == T0 + assert load_heartbeat(hb_dir / "broken.json") is None + assert not list(hb_dir.glob("*.tmp")) # atomic: no partial files left behind + + +def test_write_heartbeat_atomic_no_partial_file(tmp_path: Path) -> None: + with mock.patch("os.replace", side_effect=OSError("disk full")), pytest.raises(OSError): + write_heartbeat(tmp_path, _hb("agent-a")) + assert not (tmp_path / HEARTBEATS_DIRNAME / "agent-a.json").exists() + assert not list((tmp_path / HEARTBEATS_DIRNAME).glob("*.tmp")) + + +def test_sweep_stale_heartbeats(tmp_path: Path) -> None: + write_heartbeat(tmp_path, _hb("fresh", at=T0 - timedelta(hours=1))) + write_heartbeat(tmp_path, _hb("stale", at=T0 - timedelta(hours=30))) + assert sweep_stale_heartbeats(tmp_path, now=T0) == 1 + assert [r.agent_key for r in load_all_heartbeats(tmp_path)] == ["fresh"] + + +def test_state_round_trip_and_fail_open(tmp_path: Path) -> None: + assert load_state(tmp_path) is None + state = new_state(now=T0, zo_session_id="zo-1", heartbeats=[_hb("a", ticks=2)]) + state.paused_at, state.nudges_used = T0, 2 + path = save_state(tmp_path, state) + assert path == tmp_path / HEARTBEATS_DIRNAME / WATCHDOG_STATE_FILENAME + assert load_state(tmp_path) == state + assert json.loads(path.read_text())["baseline_ticks"] == {"a": 2} + path.write_text("{corrupt") + assert load_state(tmp_path) is None + assert load_all_heartbeats(tmp_path) == [] # state file never counts as a heartbeat + + +def test_watchdog_config_defaults_and_extra_forbid() -> None: + cfg = WatchdogConfig() + assert (cfg.enabled, cfg.stall_threshold_sec, cfg.nudge_budget, cfg.hard_max_restarts) == ( + True, 1200, 3, 3) + with pytest.raises(ValidationError): + WatchdogConfig(tick_cron="* * * * *") + + +def test_resolve_watchdog_config_env_kill_switch() -> None: + assert resolve_watchdog_config(env={}).enabled is True + assert resolve_watchdog_config(env={"ZO_WATCHDOG": "0"}).enabled is False + project = WatchdogConfig(nudge_budget=5) + got = resolve_watchdog_config(project, env={"ZO_WATCHDOG_STALL_SEC": "600"}) + assert (got.stall_threshold_sec, got.nudge_budget) == (600, 5) + assert project.stall_threshold_sec == 1200 # input not mutated + junk = resolve_watchdog_config(env={"ZO_WATCHDOG_STALL_SEC": "junk"}) + assert junk.stall_threshold_sec == 1200 + + +def test_module_carries_mit_attribution() -> None: + assert "MIT" in (watchdog.__doc__ or "") and "Yeachan Heo" in (watchdog.__doc__ or "") + + +def test_public_api_surface_resolves() -> None: + for name in watchdog.__all__: + assert getattr(watchdog, name) is not None, name + assert {"evaluate", "classify_never_block", "is_process_dead", "WatchdogConfig"} <= set( + watchdog.__all__) diff --git a/tests/unit/test_watchdog_policy.py b/tests/unit/test_watchdog_policy.py new file mode 100644 index 0000000..796e5aa --- /dev/null +++ b/tests/unit/test_watchdog_policy.py @@ -0,0 +1,383 @@ +"""``evaluate()`` scenario table for zo.watchdog (WS-C, plan oracle checks 11-12). + +Every scenario drives the pure policy with an injected clock; ``seeded`` +tests plant the failure condition (a 10-minute stall, a rate-limit banner, +a dead process) and assert the watchdog reacts exactly as the contract says. +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta, timezone + +from zo.watchdog import ( + HeartbeatRecord, + HeartbeatStatus, + NeverBlockReason, + StallAction, + StallVerdict, + WatchdogConfig, + WatchdogState, + compute_pause_until, + evaluate, + new_state, +) + +T0 = datetime(2026, 8, 17, 14, 0, tzinfo=UTC) +IDLE = "work done.\n\n› " +BANNER_RESET = "You've hit your usage limit · resets at 3pm\n› " +BANNER_PLAIN = "Rate limit reached. Try again later.\n› " +DIALOG = "Do you want to proceed?\n❯ 1. Yes\n 2. No (esc)" +CFG = WatchdogConfig(stall_threshold_sec=600, startup_grace_sec=120, nudge_delay_sec=30, + nudge_budget=3, escalate_grace_sec=120) + + +class Clock: + """Injected clock: ``at(sec)`` returns T0 + sec.""" + + @staticmethod + def at(sec: float) -> datetime: + return T0 + timedelta(seconds=sec) + + +def _tick(state: WatchdogState, sec: float, *, text: str = IDLE, progress: bool = False, + can_nudge: bool = True, process_dead: bool | None = None, + heartbeats: list[HeartbeatRecord] | None = None, cfg: WatchdogConfig = CFG, + is_interrupt: bool | None = None) -> StallVerdict: + return evaluate(state, cfg, now=Clock.at(sec), text=text, heartbeats=heartbeats or [], + progress=progress, process_dead=process_dead, is_interrupt=is_interrupt, + can_nudge=can_nudge) + + +def _deliver_nudge(state: WatchdogState, sec: float, *, resume: bool = False) -> None: + """What the wrapper does after a nudge was actually delivered.""" + if resume: + state.resume_nudges_used += 1 + else: + state.nudges_used += 1 + state.last_nudge_at = Clock.at(sec) + + +# ---- (a) seeded 10-min stall → nudge ×3 → escalate once (oracle check 11) ---- + +def test_seeded_10min_stall_nudges_then_escalates_once() -> None: + state = new_state(now=T0) + minute_actions = [_tick(state, m * 60).action for m in range(12)] + # t=0..1 grace, t=2..9 healthy, t=10 stalled but inside dwell, t=11 first nudge + assert minute_actions[:11] == [StallAction.NONE] * 11 + assert minute_actions[11] == StallAction.NUDGE + assert state.stall_since == Clock.at(600) and state.stall_events == 1 + _deliver_nudge(state, 660) + assert _tick(state, 670).action == StallAction.NONE # dwell between nudges + assert _tick(state, 690).action == StallAction.NUDGE + _deliver_nudge(state, 690) + assert _tick(state, 720).action == StallAction.NUDGE + _deliver_nudge(state, 720) + assert _tick(state, 740).action == StallAction.NONE # budget spent, dwell before escalating + verdict = _tick(state, 750) + assert verdict.action == StallAction.ESCALATE and verdict.stalled is True + assert "budget" in verdict.reason and state.escalated_at == Clock.at(750) + later = [_tick(state, 750 + 60 * i) for i in range(1, 4)] + assert all(v.action == StallAction.NONE and v.stalled for v in later) # escalate fires ONCE + + +def test_stall_clears_on_progress_and_can_recur() -> None: + state = new_state(now=T0) + _tick(state, 660) + assert state.stall_since is not None + verdict = _tick(state, 700, progress=True) + assert verdict.action == StallAction.NONE and not verdict.stalled and state.stall_since is None + assert _tick(state, 700 + 601).stalled is True + assert state.stall_events == 2 + + +# ---- (b) seeded rate-limited session is never nudged (oracle check 11) ---- + +def test_seeded_rate_limited_never_nudged() -> None: + """A rate-limited session is never stall-NUDGED. Before ``paused_until`` the + verdict is PAUSE; a static plain banner past its backoff is stale → the + check-12 RESUME_NUDGE probe (not a stall nudge); a *fresh* banner (new + text) extends the pause by backoff.""" + state = new_state(now=T0) + actions = [_tick(state, sec, text=BANNER_PLAIN).action for sec in (0, 30, 59)] + assert actions == [StallAction.PAUSE] * 3 + assert state.paused_at == T0 and state.paused_until == T0 + timedelta(seconds=60) + # Static banner past its backoff: not a stall nudge, a resume probe. + assert _tick(state, 60, text=BANNER_PLAIN).action == StallAction.RESUME_NUDGE + # Fresh banner (Claude re-printed a limit line): extend by backoff, still PAUSE. + fresh = BANNER_PLAIN + "\nRate limit reached again. Try again later.\n› " + verdict = _tick(state, 61, text=fresh) + assert verdict.action == StallAction.PAUSE and "extended" in verdict.reason + assert state.pause_attempts == 2 and state.paused_until == T0 + timedelta(seconds=61 + 120) + assert _tick(state, 100, text=fresh).action == StallAction.PAUSE + assert state.nudges_used == 0 and state.last_never_block == "rate_limit" + assert StallAction.NUDGE not in actions + assert compute_pause_until(T0, None, attempt=0, config=CFG) == T0 + timedelta(seconds=60) + assert compute_pause_until(T0, None, attempt=10, config=CFG) == T0 + timedelta(seconds=1800) + + +def test_rate_limit_pause_exceeding_max_escalates_once() -> None: + cfg = CFG.model_copy(update={"rate_limit_max_pause_sec": 300}) + state = new_state(now=T0) + _tick(state, 0, text=BANNER_PLAIN, cfg=cfg) + verdict = _tick(state, 301, text=BANNER_PLAIN, cfg=cfg) + assert verdict.action == StallAction.ESCALATE and "max" in verdict.reason + assert _tick(state, 400, text=BANNER_PLAIN, cfg=cfg).action == StallAction.NONE + + +# ---- (c) check-12 resume: parsed reset → resume nudge → verified resume ---- + +def test_seeded_check12_resume_after_reset() -> None: + state = new_state(now=T0) + verdict = _tick(state, 0, text=BANNER_RESET) + assert verdict.action == StallAction.PAUSE + assert verdict.never_block is NeverBlockReason.RATE_LIMIT + assert state.paused_until == datetime(2026, 8, 17, 15, 0, 15, tzinfo=UTC) + assert _tick(state, 1800, text=BANNER_RESET).action == StallAction.PAUSE + # banner gone but pause not yet elapsed → wait + assert _tick(state, 1900).action == StallAction.NONE + # banner gone, pause elapsed, no progress → resume nudge (bounded) + verdict = _tick(state, 3700) + assert verdict.action == StallAction.RESUME_NUDGE + _deliver_nudge(state, 3700, resume=True) + assert _tick(state, 3710).action == StallAction.RESUME_NUDGE + _deliver_nudge(state, 3710, resume=True) + assert _tick(state, 3720).action == StallAction.NONE # resume budget (2) exhausted; dwell + verdict = _tick(state, 3730, progress=True) + assert verdict.action == StallAction.RESUME and not verdict.stalled + assert state.paused_at is None and state.paused_until is None + assert state.total_paused_sec == 3730.0 and state.resume_nudges_used == 0 + assert _tick(state, 3740).action == StallAction.NONE + + +def test_seeded_check12_banner_clock_time_is_local_tz() -> None: + """Regression: 'resets at 3pm' is the operator's LOCAL 3pm. At 20:00Z in + US/Pacific (13:00 local) the reset is 22:00Z today — not 15:00Z tomorrow.""" + pacific = timezone(timedelta(hours=-7)) + now = datetime(2026, 8, 17, 20, 0, tzinfo=UTC) + state = new_state(now=now) + verdict = evaluate(state, CFG, now=now, text=BANNER_RESET, heartbeats=[], progress=False, + process_dead=None, can_nudge=True, tz=pacific) + assert verdict.action == StallAction.PAUSE + assert state.paused_until == datetime(2026, 8, 17, 22, 0, 15, tzinfo=UTC) + assert state.pause_evidence == "reset" + # tz=None keeps the pure default (now.tzinfo=UTC): "3pm" is already past → + # tomorrow, which is beyond rate_limit_max_pause_sec and therefore NOT + # honoured — exponential backoff (60 s) applies and the evidence tier is + # the banner itself, not a parsed reset. The runner supplies the local tz. + state = new_state(now=now) + evaluate(state, CFG, now=now, text=BANNER_RESET, heartbeats=[], progress=False, + process_dead=None, can_nudge=True) + assert state.paused_until == now + timedelta(seconds=60) + assert state.pause_evidence == "banner" + + +def test_seeded_static_banner_is_stale_at_reset_not_rolled_to_tomorrow() -> None: + """Regression: the TUI never clears the usage-limit line. Past the reset an + UNCHANGED banner → RESUME_NUDGE (never re-parsed into a +24 h extension); + heartbeat progress with the banner still visible → verified RESUME; the + spent banner is then ignored until it scrolls away.""" + state = new_state(now=T0) + assert _tick(state, 0, text=BANNER_RESET).action == StallAction.PAUSE + until = datetime(2026, 8, 17, 15, 0, 15, tzinfo=UTC) + assert state.paused_until == until + assert _tick(state, 3600, text=BANNER_RESET).action == StallAction.PAUSE # 15:00:00 < until + verdict = _tick(state, 3616, text=BANNER_RESET) # 15:00:16 ≥ until, same banner + assert verdict.action == StallAction.RESUME_NUDGE and "stale" in verdict.reason + assert state.paused_until == until and state.pause_attempts == 1 # NOT tomorrow + _deliver_nudge(state, 3616, resume=True) + # Progress (heartbeat) while the banner is still in the tail → RESUME. + verdict = _tick(state, 3640, text=BANNER_RESET, progress=True) + assert verdict.action == StallAction.RESUME and state.paused_at is None + assert state.total_paused_sec == 3640.0 and state.spent_banner_key + # Same banner still on screen next tick: not a new pause; healthy tick. + verdict = _tick(state, 3650, text=BANNER_RESET) + assert verdict.action == StallAction.NONE and verdict.never_block is None + assert state.paused_at is None + # Banner scrolls away → the 'spent' memory is dropped; a NEW banner pauses again. + _tick(state, 3660, text=IDLE) + assert state.spent_banner_key is None + assert _tick(state, 3670, text=BANNER_PLAIN).action == StallAction.PAUSE + + +def test_fresh_banner_past_reset_extends_by_backoff_not_day_rollover() -> None: + """A re-printed 'resets at 3pm' read AFTER 3pm would parse as tomorrow; the + extension rejects a reset beyond ``rate_limit_max_pause_sec`` and falls + back to exponential backoff (a real later reset is honoured).""" + state = new_state(now=T0) + _tick(state, 0, text=BANNER_RESET) # until 15:00:15 + fresh = BANNER_RESET + "\n› continue\nYou've hit your usage limit · resets at 3pm\n› " + verdict = _tick(state, 3616, text=fresh) # 15:00:16, banner text changed + assert verdict.action == StallAction.PAUSE and "extended" in verdict.reason + assert state.paused_until == Clock.at(3616 + 120) # backoff attempt 1, not 15:00 tomorrow + assert state.pause_evidence == "banner" + later = fresh + "\nYou've hit your usage limit · resets at 5pm\n› " + verdict = _tick(state, 3616 + 120, text=later) # newest banner wins: 17:00 today + assert state.paused_until == datetime(2026, 8, 17, 17, 0, 15, tzinfo=UTC) + assert state.pause_evidence == "reset" + + +def test_progress_while_banner_fresh_before_reset_stays_paused() -> None: + """A teammate heartbeat while the banner is fresh and the reset is ahead + is not a verified resume (no PAUSE/RESUME flapping).""" + state = new_state(now=T0) + _tick(state, 0, text=BANNER_RESET) + verdict = _tick(state, 30, text=BANNER_RESET, progress=True) + assert verdict.action == StallAction.PAUSE and state.paused_at == T0 + verdict = _tick(state, 60, progress=True) # banner gone → progress resumes at once + assert verdict.action == StallAction.RESUME + + +def test_resume_nudges_exhausted_without_progress_escalates_once() -> None: + state = new_state(now=T0) + _tick(state, 0, text=BANNER_PLAIN) # paused_until = +60 s + for sec in (61, 71): + assert _tick(state, sec).action == StallAction.RESUME_NUDGE + _deliver_nudge(state, sec, resume=True) + assert _tick(state, 90).action == StallAction.NONE # dwell after the last resume nudge + verdict = _tick(state, 125) + assert verdict.action == StallAction.ESCALATE and "unverified" in verdict.reason + assert _tick(state, 200).action == StallAction.NONE + assert _tick(state, 300, progress=True).action == StallAction.RESUME # late resume still wins + + +# ---- (d) headless resume requires progress ---- + +def test_headless_resume_requires_progress() -> None: + state = new_state(now=T0) + _tick(state, 0, text=BANNER_PLAIN, can_nudge=False) # paused_until = +60 s + assert _tick(state, 61, can_nudge=False).action == StallAction.NONE + assert _tick(state, 100, can_nudge=False).action == StallAction.NONE + verdict = _tick(state, 121, can_nudge=False) # paused_until + backoff base → escalate + assert verdict.action == StallAction.ESCALATE and "unverified" in verdict.reason + assert _tick(state, 130, can_nudge=False).action == StallAction.NONE + verdict = _tick(state, 140, can_nudge=False, progress=True) + # Paused time stops accruing at the escalation (t=121): a late verified + # resume does not retroactively suspend the run timeout past the hand-off. + assert verdict.action == StallAction.RESUME and state.total_paused_sec == 121.0 + + +def test_headless_stall_escalates_after_grace_without_nudging() -> None: + state = new_state(now=T0) + assert _tick(state, 660, can_nudge=False).action == StallAction.NONE + verdict = _tick(state, 660 + 120, can_nudge=False) + assert verdict.action == StallAction.ESCALATE and "headless" in verdict.reason + assert _tick(state, 900, can_nudge=False).action == StallAction.NONE + assert state.nudges_used == 0 + + +# ---- (e) dead process → escalate right after grace (oracle check 11) ---- + +def test_seeded_dead_process_escalates_after_grace() -> None: + state = new_state(now=T0) + assert _tick(state, 60, process_dead=True).action == StallAction.NONE # grace + verdict = _tick(state, 121, process_dead=True) + assert verdict.action == StallAction.ESCALATE and verdict.process_dead is True + assert "dead" in verdict.reason and verdict.stalled + assert _tick(state, 200, process_dead=True).action == StallAction.NONE + assert _tick(state, 200, process_dead=None).action == StallAction.NONE # unknown ≠ dead + + +def test_progress_contradicts_dead_verdict_and_dead_escalates_at_most_once() -> None: + """Regression: a recycled/wrong pid with heartbeats still ticking must not + escalate on EVERY tick — progress wins, and positive-proof death is + escalated once per run even across stall resets.""" + state = new_state(now=T0) + verdicts = [_tick(state, 200 + i * 10, process_dead=True, progress=True) for i in range(5)] + assert {v.action for v in verdicts} == {StallAction.NONE} + assert all(not v.stalled for v in verdicts) and state.stall_events == 0 + # No progress this tick → dead is escalated once … + verdict = _tick(state, 300, process_dead=True) + assert verdict.action == StallAction.ESCALATE and "dead" in verdict.reason + # … progress clears the stall, dead again → NOT a second escalation. + _tick(state, 310, process_dead=True, progress=True) + actions = [_tick(state, 320 + i * 10, process_dead=True).action for i in range(5)] + assert actions == [StallAction.NONE] * 5 and state.stall_events == 2 + assert state.dead_escalated_at == Clock.at(300) + + +def test_busy_tmux_pane_cannot_be_nudged_and_escalates_after_grace() -> None: + """Regression: the wrapper passes ``can_nudge=False`` for a busy pane + (spinner / 'esc to interrupt'); a stall there escalates after + ``escalate_grace_sec`` instead of returning NUDGE forever.""" + state = new_state(now=T0) + busy = "· Running… (25m 12s · esc to interrupt)\n" + actions = [_tick(state, sec, text=busy, can_nudge=False).action for sec in (600, 660, 700)] + assert actions == [StallAction.NONE, StallAction.NONE, StallAction.NONE] # grace 120 s + verdict = _tick(state, 720, text=busy, can_nudge=False) + assert verdict.action == StallAction.ESCALATE and "busy" in verdict.reason + assert _tick(state, 800, text=busy, can_nudge=False).action == StallAction.NONE + assert state.nudges_used == 0 + + +# ---- (f) startup grace ---- + +def test_startup_grace_suppresses() -> None: + cfg = CFG.model_copy(update={"stall_threshold_sec": 10, "startup_grace_sec": 300}) + state = new_state(now=T0) + assert all(_tick(state, s, cfg=cfg).action == StallAction.NONE for s in (0, 100, 299)) + assert _tick(state, 300, cfg=cfg).stalled is True + + +# ---- (g) awaiting_input / other never-block reasons are never nudged ---- + +def test_awaiting_input_never_nudged() -> None: + state = new_state(now=T0) + verdicts = [_tick(state, m * 60, text=DIALOG) for m in range(30)] + assert {v.action for v in verdicts} == {StallAction.NONE} + assert verdicts[-1].never_block is NeverBlockReason.AWAITING_INPUT + assert verdicts[-1].stalled is True # the clock ran, but no nudge and no escalation + + +def test_user_abort_and_interrupt_never_nudged() -> None: + state = new_state(now=T0) + assert _tick(state, 900, is_interrupt=True).action == StallAction.NONE + assert state.last_never_block == "user_abort" + assert _tick(state, 960, text="Interrupted by user\n› ").action == StallAction.NONE + + +def test_auth_error_escalates_after_threshold_but_never_nudges() -> None: + state = new_state(now=T0) + assert _tick(state, 300, text="please run /login\n› ").action == StallAction.NONE + verdict = _tick(state, 601, text="please run /login\n› ") + assert verdict.action == StallAction.ESCALATE and "auth_error" in verdict.reason + assert _tick(state, 700, text="please run /login\n› ").action == StallAction.NONE + assert state.nudges_used == 0 + + +# ---- (h) compacting resets the stall clock ---- + +def test_compacting_resets_clock() -> None: + state = new_state(now=T0) + hb = HeartbeatRecord(agent_key="lead-s1", session_id="s1", tick_count=1, + last_tick_at=Clock.at(590), status=HeartbeatStatus.COMPACTING) + verdict = _tick(state, 600, heartbeats=[hb]) + assert verdict.action == StallAction.NONE and verdict.never_block is NeverBlockReason.COMPACTING + assert state.last_progress_at == Clock.at(600) + assert _tick(state, 1100).stalled is False + assert _tick(state, 1200).stalled is True + + +# ---- (i) progress via any observer feeds evaluate() ---- + +def test_progress_flag_resets_clock_and_disabled_nudges_do_not_fire() -> None: + state = new_state(now=T0) + _tick(state, 500, progress=True) + assert state.last_progress_at == Clock.at(500) + verdict = _tick(state, 1000) + assert verdict.action == StallAction.NONE and verdict.progress is False + off = CFG.model_copy(update={"nudge_enabled": False}) + state = new_state(now=T0) + assert _tick(state, 700, cfg=off).action == StallAction.NONE + verdict = _tick(state, 700 + 120, cfg=off) + assert verdict.action == StallAction.ESCALATE and state.nudges_used == 0 + + +def test_verdict_carries_freshness_and_tick_bookkeeping() -> None: + state = new_state(now=T0) + fresh = HeartbeatRecord(agent_key="a", session_id="s", last_tick_at=Clock.at(50), tick_count=1) + verdict = _tick(state, 60, heartbeats=[fresh]) + assert verdict.freshness == "fresh" and verdict.evaluated_at == Clock.at(60) + assert state.ticks == 1 and state.last_tick_at == Clock.at(60) + stale = HeartbeatRecord(agent_key="a", session_id="s", last_tick_at=T0, tick_count=1) + assert _tick(state, 700, heartbeats=[stale]).freshness == "stale" + assert _tick(state, 700).freshness == "unknown" diff --git a/tests/unit/test_wrapper.py b/tests/unit/test_wrapper.py index 6fc365a..9e6a0a6 100644 --- a/tests/unit/test_wrapper.py +++ b/tests/unit/test_wrapper.py @@ -1,4 +1,10 @@ -"""Unit tests for zo.wrapper and zo._wrapper_models.""" +"""Unit tests for zo.wrapper, zo._wrapper_models and zo._wrapper_watchdog. + +WS-C execution substrate (plan oracle checks 11-12): the watchdog is an +external checker ticked from BOTH poll loops. Every mechanism has a +seeded-failure test (plants the condition, asserts the mechanism catches it) +and a wiring test (proves the runtime path invokes it). +""" from __future__ import annotations @@ -6,7 +12,9 @@ import os import signal import subprocess +from datetime import UTC, datetime, timedelta, timezone from pathlib import Path +from typing import TYPE_CHECKING from unittest import mock import pytest @@ -17,9 +25,21 @@ TeamMember, TeamStatus, ) +from zo._wrapper_watchdog import TICK_TRACE_FILENAME, WatchdogRunner from zo.comms import CommsLogger +from zo.watchdog import ( + HeartbeatRecord, + StallAction, + StallVerdict, + WatchdogConfig, + load_state, + write_heartbeat, +) from zo.wrapper import LifecycleWrapper +if TYPE_CHECKING: + from collections.abc import Callable + # ------------------------------------------------------------------ # # Fixtures # ------------------------------------------------------------------ # @@ -46,6 +66,186 @@ def wrapper(comms: CommsLogger, tmp_log_dir: Path) -> LifecycleWrapper: return LifecycleWrapper(comms, log_dir=tmp_log_dir) +# ------------------------------------------------------------------ # +# Watchdog test helpers (WS-C, oracle checks 11-12) +# ------------------------------------------------------------------ # + +T0 = datetime(2026, 8, 17, 10, 0, tzinfo=UTC) +POLL = 60.0 # virtual seconds per poll (sleep is patched to advance the clock) + +# Neutral pane: some output + the idle prompt (nudge-ready, no never-block). +IDLE_PANE = "Reading files\nRunning tests\n\n\u276f \n" +# The real Claude Code usage-limit banner (rate_limit never-block). +BANNER_PANE = "You've hit your usage limit \u00b7 resets at 10:05\n\n\u276f \n" +# Permission dialog (awaiting_input never-block). +DIALOG_PANE = "Do you want to proceed?\n\u276f 1. Yes\n 2. No\n" +# Busy pane: active task, no idle prompt (nudge guard must skip). +BUSY_PANE = "\u273b Thinking\u2026 (esc to interrupt)\n" + + +class FakeClock: + """Injectable wall + monotonic clock; ``sleep`` advances both.""" + + def __init__(self, start: datetime = T0) -> None: + self.now = start + self.mono = 0.0 + + def __call__(self) -> datetime: + return self.now + + def advance(self, secs: float) -> None: + self.now += timedelta(seconds=secs) + self.mono += secs + + def sleep(self, secs: float) -> None: + self.advance(secs) + + def monotonic(self) -> float: + return self.mono + + +class _StatusSpy: + """Records ``process.status`` after every ``_watchdog_tick`` call.""" + + def __init__(self, wrapper: LifecycleWrapper) -> None: + self._wrapper = wrapper + self.statuses: list[AgentStatus] = [] + self.calls = 0 + self._orig = wrapper._watchdog_tick + + def _spy(self, process: LeadProcess, **kw: object) -> object: + verdict = self._orig(process, **kw) + self.calls += 1 + self.statuses.append(process.status) + return verdict + + def __enter__(self) -> _StatusSpy: + self._patch = mock.patch.object(self._wrapper, "_watchdog_tick", side_effect=self._spy) + self._patch.start() + return self + + def __exit__(self, *exc: object) -> None: + self._patch.stop() + + +class _TmuxScenario: + """Scripted pane: text per poll index; alive for the first N polls.""" + + def __init__(self, *, alive_polls: int, text_for_poll: Callable[[int], str]) -> None: + self.alive_polls = alive_polls + self.text_for_poll = text_for_poll + self.polls = 0 + + def capture(self, pane_id: str, lines: int = 50) -> str: + text = self.text_for_poll(self.polls) + self.polls += 1 + return text + + def alive(self, pane_id: str) -> bool: + # capture() runs first in each poll, so polls == index + 1 here. + return self.polls <= self.alive_polls + + +def _wd_config(**overrides: object) -> WatchdogConfig: + base = dict(stall_threshold_sec=600, startup_grace_sec=0, nudge_delay_sec=0, + escalate_grace_sec=0) + base.update(overrides) + return WatchdogConfig(**base) # type: ignore[arg-type] + + +def _plant_heartbeat(memory_root: Path, *, tick_count: int, at: datetime) -> None: + write_heartbeat(memory_root, HeartbeatRecord( + agent_key="lead-s1", session_id="s1", last_tick_at=at, tick_count=tick_count, + )) + + +def _comms_events(tmp_path: Path) -> list[dict]: + events: list[dict] = [] + for path in sorted((tmp_path / "comms").glob("*.jsonl")): + events.extend(json.loads(ln) for ln in path.read_text().splitlines() if ln.strip()) + return events + + +def _tick_trace(memory_root: Path) -> list[dict]: + path = memory_root / "heartbeats" / TICK_TRACE_FILENAME + if not path.exists(): + return [] + return [json.loads(ln) for ln in path.read_text().splitlines() if ln.strip()] + + +@pytest.fixture() +def clock() -> FakeClock: + return FakeClock() + + +@pytest.fixture() +def memory_root(tmp_path: Path) -> Path: + root = tmp_path / "memory" + root.mkdir() + return root + + +@pytest.fixture() +def wd_wrapper(comms: CommsLogger, tmp_log_dir: Path, clock: FakeClock) -> LifecycleWrapper: + # tz=UTC pins banner clock times ("resets at 10:05") to the fake clock's zone. + return LifecycleWrapper(comms, log_dir=tmp_log_dir, clock=clock, tz=UTC) + + +def _run_tmux( + wrapper: LifecycleWrapper, clock: FakeClock, scenario: _TmuxScenario, *, + watchdog: WatchdogConfig | None, memory_root: Path | None, timeout: float | None = None, + on_status: Callable | None = None, patch_monotonic: bool = False, +) -> tuple[LeadProcess, mock.MagicMock, mock.MagicMock]: + """Drive ``_wait_tmux`` with class-patched tmux helpers; returns (result, paste, capture).""" + lead = LeadProcess(tmux_pane_id="%5", team_name="alpha", status=AgentStatus.SPAWNING) + patches = [ + mock.patch.object(LifecycleWrapper, "_tmux_pane_alive", side_effect=scenario.alive), + mock.patch.object(LifecycleWrapper, "_tmux_claude_running", return_value=True), + mock.patch.object(LifecycleWrapper, "_kill_tmux_window"), + mock.patch.object(wrapper, "monitor_team", return_value=TeamStatus(team_name="alpha")), + mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep), + ] + if patch_monotonic: + patches.append(mock.patch("zo.wrapper.time.monotonic", side_effect=clock.monotonic)) + capture = mock.patch.object(LifecycleWrapper, "_capture_tmux_pane", + side_effect=scenario.capture) + paste = mock.patch.object(LifecycleWrapper, "_paste_and_submit") + for p in patches: + p.start() + try: + with capture as capture_mock, paste as paste_mock: + result = wrapper.wait_for_completion( + lead, poll_interval=POLL, timeout=timeout, + on_status=on_status or (lambda *_: None), + watchdog=watchdog, memory_root=memory_root, zo_session_id="zo-s1", + ) + finally: + for p in reversed(patches): + p.stop() + return result, paste_mock, capture_mock + + +def _headless_setup( + wrapper: LifecycleWrapper, tmp_log_dir: Path, *, poll: object, stdout_text: str, +) -> LeadProcess: + """Wire a fake Popen + stdout log for ``_wait_headless``; returns the LeadProcess.""" + mock_proc = mock.MagicMock() + if callable(poll) or isinstance(poll, list): + mock_proc.poll.side_effect = poll + else: + mock_proc.poll.return_value = poll + mock_proc.wait.return_value = None + wrapper._proc = mock_proc + wrapper._stdout_fh = mock.MagicMock() + wrapper._stderr_fh = mock.MagicMock() + stdout_file = tmp_log_dir / "alpha-stdout.log" + stdout_file.write_text(stdout_text) + stderr_file = tmp_log_dir / "alpha-stderr.log" + stderr_file.write_text("") + return LeadProcess(pid=99, team_name="alpha", stdout_log=stdout_file, stderr_log=stderr_file) + + + # ------------------------------------------------------------------ # # Model tests # ------------------------------------------------------------------ # @@ -63,6 +263,19 @@ def test_lead_process_defaults(self) -> None: assert lp.exit_code is None assert lp.team_name == "" + def test_agent_status_watchdog_values(self) -> None: + assert AgentStatus.PAUSED_RATE_LIMIT == "paused_rate_limit" + assert AgentStatus.STALLED == "stalled" + + def test_lead_process_watchdog_defaults(self) -> None: + lp = LeadProcess() + assert lp.pid_start_identity is None + assert lp.nudges_used == 0 + assert lp.stalled is False + assert lp.paused_until is None + assert lp.resume_at is None + assert lp.pause_total_sec == 0.0 + def test_lead_process_with_values(self) -> None: lp = LeadProcess( pid=1234, @@ -109,7 +322,9 @@ def test_headless_builds_correct_command( bypass_permissions=True, ) - args = mock_popen.call_args + # call_args_list[0] is the claude launch (the identity probe may + # spawn ``ps`` through the same patched Popen afterwards). + args = mock_popen.call_args_list[0] cmd = args[0][0] assert cmd[0] == "claude" assert "--print" in cmd @@ -150,7 +365,7 @@ def test_headless_omits_skip_flag_when_bypass_false( bypass_permissions=False, ) - cmd = mock_popen.call_args[0][0] + cmd = mock_popen.call_args_list[0][0][0] assert "--dangerously-skip-permissions" not in cmd # Sanity: rest of the command is still well-formed assert "--print" in cmd @@ -171,7 +386,7 @@ def test_headless_default_bypass_is_false( use_tmux=False, ) - cmd = mock_popen.call_args[0][0] + cmd = mock_popen.call_args_list[0][0][0] assert "--dangerously-skip-permissions" not in cmd @mock.patch("zo.wrapper.subprocess.Popen") @@ -183,7 +398,7 @@ def test_add_dir_flag_present( "prompt", cwd="/my/delivery", team_name="t", use_tmux=False ) - cmd = mock_popen.call_args[0][0] + cmd = mock_popen.call_args_list[0][0][0] assert "--add-dir" in cmd assert "/my/delivery" in cmd @@ -501,44 +716,59 @@ def test_detects_error_exit( assert result.exit_code == 1 @mock.patch("zo.wrapper.time.sleep") - def test_detects_rate_limit_and_backs_off( - self, mock_sleep: mock.MagicMock, wrapper: LifecycleWrapper, tmp_log_dir: Path + def test_running_process_with_rate_limit_text_pauses_without_backoff( + self, mock_sleep: mock.MagicMock, comms: CommsLogger, tmp_log_dir: Path, + tmp_path: Path, ) -> None: + """Rewritten (WS-C): a RUNNING process whose output carries a + rate-limit banner is PAUSED by the watchdog — no blocking backoff + sleep, no in-wrapper retries. Sleep is only ever the poll interval.""" + wrapper = LifecycleWrapper(comms, log_dir=tmp_log_dir, clock=FakeClock(), tz=UTC) mock_proc = mock.MagicMock() - # First poll: still running; second poll: done. - mock_proc.poll.side_effect = [None, 0] + mock_proc.poll.side_effect = [None, None, 0] wrapper._proc = mock_proc wrapper._stdout_fh = mock.MagicMock() wrapper._stderr_fh = mock.MagicMock() stdout_file = tmp_log_dir / "alpha-stdout.log" - stdout_file.write_text("Error 429 Too Many Requests") - + stdout_file.write_text("Error 429 Too Many Requests\n") lead = LeadProcess(pid=99, team_name="alpha", stdout_log=stdout_file) + memory_root = tmp_path / "memory" - result = wrapper.wait_for_completion(lead, poll_interval=0.01) - assert result.status == AgentStatus.COMPLETED - # Should have called sleep for the backoff. - assert mock_sleep.called + seen = _StatusSpy(wrapper) + with seen: + result = wrapper.wait_for_completion( + lead, poll_interval=0.01, watchdog=_wd_config(), + memory_root=memory_root, + ) + assert AgentStatus.PAUSED_RATE_LIMIT in seen.statuses + # No exponential backoff: every sleep is exactly the poll interval. + assert {c.args[0] for c in mock_sleep.call_args_list} == {0.01} + # Exited while paused → RATE_LIMITED for the driver to relaunch. + assert result.status == AgentStatus.RATE_LIMITED @mock.patch("zo.wrapper.time.sleep") - def test_rate_limit_exhausts_retries( + def test_exited_process_with_rate_limit_text_is_rate_limited_with_resume_at( self, mock_sleep: mock.MagicMock, wrapper: LifecycleWrapper, tmp_log_dir: Path ) -> None: - wrapper._max_retries = 2 + """Rewritten (WS-C): an EXITED process whose final output carries a + rate-limit banner is classified RATE_LIMITED with a parsed + ``resume_at`` — no retries in the wrapper (works without a runner).""" mock_proc = mock.MagicMock() - mock_proc.poll.return_value = None # Never completes. + mock_proc.poll.return_value = 1 wrapper._proc = mock_proc wrapper._stdout_fh = mock.MagicMock() wrapper._stderr_fh = mock.MagicMock() stdout_file = tmp_log_dir / "alpha-stdout.log" - stdout_file.write_text("rate limit exceeded") - + stdout_file.write_text("rate limit exceeded; try again in 5 minutes\n") lead = LeadProcess(pid=99, team_name="alpha", stdout_log=stdout_file) result = wrapper.wait_for_completion(lead, poll_interval=0.01) assert result.status == AgentStatus.RATE_LIMITED + assert result.resume_at is not None + assert result.exit_code == 1 + assert not mock_sleep.called @mock.patch("zo.wrapper.time.sleep") def test_tmux_wait_completes_when_pane_closes( @@ -727,39 +957,32 @@ def test_handles_no_log_path(self, wrapper: LifecycleWrapper) -> None: class TestDetectRateLimit: + """Exit-classification matcher, tightened to the watchdog patterns.""" + @pytest.mark.parametrize( "text", [ - "HTTP 429 response", + "HTTP 429 rate limit response", "rate limit exceeded", - "Rate-Limited by server", - "API overloaded", + "rate limited by server", "too many requests", + "You've hit your usage limit · resets at 3pm", ], ) def test_catches_known_patterns(self, text: str) -> None: assert LifecycleWrapper._detect_rate_limit(text) is True - def test_returns_false_for_normal_output(self) -> None: - assert LifecycleWrapper._detect_rate_limit("All tasks completed successfully.") is False - - -# ------------------------------------------------------------------ # -# _backoff_wait -# ------------------------------------------------------------------ # - - -class TestBackoffWait: - def test_returns_increasing_durations( - self, wrapper: LifecycleWrapper - ) -> None: - d0 = wrapper._backoff_wait(0) - d1 = wrapper._backoff_wait(1) - d2 = wrapper._backoff_wait(2) - # base=30. Without jitter: 30, 60, 120. With jitter (+0..5): - assert 30 <= d0 <= 35 - assert 60 <= d1 <= 65 - assert 120 <= d2 <= 125 + @pytest.mark.parametrize( + "text", + [ + "All tasks completed successfully.", + "val_loss 0.4291", + "step 4290 done", + "GPU overloaded, reducing batch size", + ], + ) + def test_no_bare_429_or_overloaded_false_positives(self, text: str) -> None: + assert LifecycleWrapper._detect_rate_limit(text) is False # ------------------------------------------------------------------ # @@ -917,3 +1140,690 @@ def test_does_not_check_legacy_logs_training_path( wrapper._maybe_open_training_pane() # Legacy file should NOT trigger the dashboard. mock_run.assert_not_called() + + +# ------------------------------------------------------------------ # +# WS-C watchdog integration (plan oracle checks 11-12) +# ------------------------------------------------------------------ # + + +class TestWatchdogWiring: + """Wiring half: the runtime path invokes the watchdog from both loops.""" + + def test_wrapper_proc_defaults_none(self, comms: CommsLogger, tmp_log_dir: Path) -> None: + w = LifecycleWrapper(comms, log_dir=tmp_log_dir) + assert w._proc is None + assert w._wd is None + assert w._stdout_fh is None and w._stderr_fh is None + + def test_watchdog_disabled_when_config_off_or_no_memory_root( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + ) -> None: + """No runner → the loops behave exactly as before (4 polls to confirm).""" + for cfg, root in ((_wd_config(enabled=False), memory_root), (_wd_config(), None)): + scenario = _TmuxScenario(alive_polls=0, text_for_poll=lambda k: IDLE_PANE) + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, watchdog=cfg, + memory_root=root) + assert wd_wrapper._wd is None + assert result.status == AgentStatus.COMPLETED + assert scenario.polls == 4 # 2 grace + 2 confirm, unchanged + assert not paste.called + assert not (memory_root / "heartbeats").exists() + + def test_watchdog_tick_runs_on_suspected_dead_path( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + ) -> None: + """Regression: the tick must fire on the post-grace single-negative + ``continue`` path (which skips the rest of the iteration).""" + # polls: grace, grace, dead(1)->continue, alive(reset), dead, dead->done. + alive_seq = iter([True, True, False, True, False, False]) + with mock.patch.object(LifecycleWrapper, "_tmux_pane_alive", + side_effect=lambda pid: next(alive_seq)) as alive, \ + mock.patch.object(LifecycleWrapper, "_tmux_claude_running", return_value=True), \ + mock.patch.object(LifecycleWrapper, "_kill_tmux_window"), \ + mock.patch.object(LifecycleWrapper, "_capture_tmux_pane", return_value=IDLE_PANE), \ + mock.patch.object(wd_wrapper, "monitor_team", + return_value=TeamStatus(team_name="alpha")), \ + mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep): + spy = _StatusSpy(wd_wrapper) + with spy: + result = wd_wrapper.wait_for_completion( + LeadProcess(tmux_pane_id="%5", team_name="alpha"), poll_interval=POLL, + on_status=lambda *_: None, + watchdog=_wd_config(), memory_root=memory_root, + ) + assert result.status == AgentStatus.COMPLETED + assert alive.call_count == 6 + assert spy.calls == 6 # one tick per poll, including the dead-path poll (#3) + assert len(_tick_trace(memory_root)) == 6 + + def test_single_pane_capture_per_poll( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + ) -> None: + """ONE capture per poll, shared by the watchdog and on_status (last 5 lines).""" + snapshots: list[str] = [] + scenario = _TmuxScenario( + alive_polls=3, text_for_poll=lambda k: f"l1\nl2\nl3\nl4\nl5\nl6-{k}\n\u276f \n") + result, _, capture = _run_tmux( + wd_wrapper, clock, scenario, watchdog=_wd_config(), memory_root=memory_root, + on_status=lambda _ts, snap: snapshots.append(snap), + ) + assert result.status == AgentStatus.COMPLETED + assert capture.call_count == scenario.polls == 5 # 3 alive + 2 confirm + # 3 alive-path snapshots = last 5 lines of the shared capture; the + # suspected-dead poll keeps its pre-existing "" snapshot. + assert [len(snap.splitlines()) for snap in snapshots] == [5, 5, 5, 0] + assert snapshots[0].splitlines()[-1] == "\u276f " + assert snapshots[2].splitlines()[-2] == "l6-2" + + @mock.patch("zo.wrapper.time.sleep") + @mock.patch("zo.wrapper.subprocess.run") + def test_paste_and_submit_uses_named_buffer( + self, mock_run: mock.MagicMock, mock_sleep: mock.MagicMock, + ) -> None: + LifecycleWrapper._paste_and_submit("%5", "hello there") + cmds = [c.args[0] for c in mock_run.call_args_list] + assert cmds[0][:5] == ["tmux", "load-buffer", "-b", "zo-nudge", "-"] + assert mock_run.call_args_list[0].kwargs["input"] == "hello there" + assert cmds[1] == ["tmux", "paste-buffer", "-b", "zo-nudge", "-d", "-t", "%5"] + assert cmds[2] == ["tmux", "send-keys", "-t", "%5", "Enter"] + # No temp file / default buffer involved. + assert not any(c[1] == "load-buffer" and len(c) == 3 for c in cmds) + + @mock.patch("zo.wrapper.time.sleep") + @mock.patch("zo.wrapper.subprocess.run") + def test_tmux_launch_pastes_prompt_via_named_buffer( + self, mock_run: mock.MagicMock, mock_sleep: mock.MagicMock, wrapper: LifecycleWrapper, + ) -> None: + """The launch path is refactored onto ``_paste_and_submit`` (behaviour-preserving).""" + mock_run.return_value = mock.MagicMock(stdout="%5\n", returncode=0) + with mock.patch.dict(os.environ, {"TMUX": "/tmp/tmux,1,0"}), \ + mock.patch.object(LifecycleWrapper, "_paste_and_submit") as paste: + wrapper.launch_lead_session("the prompt", cwd="/target", team_name="a", use_tmux=True) + assert paste.call_args_list[0] == mock.call("%5", "the prompt") + + @mock.patch("zo.wrapper.time.sleep") + @mock.patch("zo.wrapper.subprocess.run") + def test_tmux_launch_records_pid_and_identity_best_effort( + self, mock_run: mock.MagicMock, mock_sleep: mock.MagicMock, wrapper: LifecycleWrapper, + ) -> None: + """pane_pid → pgrep (newest child whose cmdline mentions claude) → pid + identity.""" + def fake_run(cmd, **kw): # noqa: ANN001, ANN202 + out = "%5\n" + if cmd[0] == "tmux" and cmd[1] == "display-message": + out = "4242\n" + elif cmd[0] == "pgrep": + out = "4300\n4301\n" + elif cmd[0] == "which": + out = "/usr/local/bin/claude\n" + return mock.MagicMock(stdout=out, returncode=0) + mock_run.side_effect = fake_run + with mock.patch.dict(os.environ, {"TMUX": "/tmp/tmux,1,0"}), \ + mock.patch("zo.wrapper.process_start_identity", return_value="darwin:1700000000:0"): + result = wrapper.launch_lead_session("p", cwd="/target", team_name="a", use_tmux=True) + assert result.pid == 4300 + assert result.pid_start_identity == "darwin:1700000000:0" + # Filtered by command line so a shell's prompt helper / gitstatusd child + # (which would later "die" and fake a dead lead) is never picked. + assert ["pgrep", "-n", "-P", "4242", "-f", "claude"] in [ + c.args[0] for c in mock_run.call_args_list] + + @mock.patch("zo.wrapper.subprocess.run") + def test_tmux_lead_identity_unresolvable_without_claude_child( + self, mock_run: mock.MagicMock, + ) -> None: + """No claude child (only a prompt helper) → (None, None): unknown, never a wrong pid.""" + mock_run.return_value = mock.MagicMock(stdout="\n", returncode=1) + assert LifecycleWrapper._resolve_tmux_lead_identity(4242) == (None, None) + assert LifecycleWrapper._resolve_tmux_lead_identity(None) == (None, None) + + @mock.patch("zo.wrapper.subprocess.Popen") + def test_headless_launch_records_identity( + self, mock_popen: mock.MagicMock, wrapper: LifecycleWrapper, + ) -> None: + mock_popen.return_value.pid = 4242 + with mock.patch("zo.wrapper.process_start_identity", return_value="linux:12345"): + result = wrapper.launch_lead_session("p", cwd="/t", team_name="a", use_tmux=False) + assert result.pid == 4242 + assert result.pid_start_identity == "linux:12345" + + def test_start_watchdog_gitignores_heartbeats_in_zo_dir_layout( + self, wd_wrapper: LifecycleWrapper, tmp_path: Path, + ) -> None: + """Runtime files under ``<delivery>/.zo/memory/heartbeats/`` must never be + committed with ``git add .zo/``: an existing ``.zo/.gitignore`` gains the + entry once (idempotent); legacy layouts (no such file) are untouched.""" + zo_dir = tmp_path / "repo" / ".zo" + (zo_dir / "memory").mkdir(parents=True) + (zo_dir / ".gitignore").write_text("local.yaml\n") + for _ in range(2): + wd_wrapper._start_watchdog(_wd_config(), memory_root=zo_dir / "memory", + zo_session_id="zo-s1", delivery_repo=None) + text = (zo_dir / ".gitignore").read_text() + assert text.split().count("memory/heartbeats/") == 1 + assert text.startswith("local.yaml\n") + legacy = tmp_path / "zo" / "memory" / "proj" + legacy.mkdir(parents=True) + wd_wrapper._start_watchdog(_wd_config(), memory_root=legacy, + zo_session_id="zo-s1", delivery_repo=None) + assert not (legacy.parent / ".gitignore").exists() + + def test_wait_for_completion_builds_runner_and_persists_state( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + ) -> None: + scenario = _TmuxScenario(alive_polls=1, text_for_poll=lambda k: IDLE_PANE) + _run_tmux(wd_wrapper, clock, scenario, watchdog=_wd_config(), memory_root=memory_root) + state = load_state(memory_root) + assert state is not None + assert state.zo_session_id == "zo-s1" + assert state.ticks == scenario.polls == 4 # 1 alive + grace-absorbed + 2 confirm + + +# ---- seeded stall → nudge → escalate (oracle check 11) ---- + + +class TestSeededStallTmux: + """Seeded half (check 11): a 10-min stall is caught, nudged within budget, + escalated exactly once, and surfaces as STALLED.""" + + def test_seeded_10min_stall_detected_and_escalated_within_one_poll( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, tmp_path: Path, + ) -> None: + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + scenario = _TmuxScenario(alive_polls=15, text_for_poll=lambda k: IDLE_PANE) + + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(), memory_root=memory_root) + + cfg = _wd_config() + assert 1 <= paste.call_count <= cfg.nudge_budget + assert all(c.args == ("%5", cfg.nudge_message) for c in paste.call_args_list) + errors = [e for e in _comms_events(tmp_path) + if e["event_type"] == "error" and e["error_type"] == "stall"] + assert any(e["severity"] == "warning" for e in errors) # first detection + blocking = [e for e in errors if e["severity"] == "blocking"] + assert len(blocking) == 1 and blocking[0]["escalated_to"] == "human" + actions = [t["action"] for t in _tick_trace(memory_root)] + # Stall lands on the first tick past the 600 s threshold (tick 10); + # nudges 10..12; escalate on the very next tick after budget exhaustion. + assert actions[:10] == ["none"] * 10 + assert actions[10:13] == ["nudge"] * 3 + assert actions[13] == "escalate" + assert actions.count("escalate") == 1 + assert result.status == AgentStatus.STALLED + assert result.stalled is True + assert result.nudges_used == 3 + + def test_seeded_rate_limited_session_is_never_nudged( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, tmp_path: Path, + ) -> None: + """Banner with a reset still ahead (10:30) for the whole run: no keys, no + stall nudge, no escalation; died while paused → RATE_LIMITED + resume_at.""" + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + banner = BANNER_PANE.replace("10:05", "10:30") + scenario = _TmuxScenario(alive_polls=14, text_for_poll=lambda k: banner) + spy = _StatusSpy(wd_wrapper) + with spy: + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(), memory_root=memory_root) + assert not paste.called + assert AgentStatus.PAUSED_RATE_LIMIT in spy.statuses + checkpoints = [e for e in _comms_events(tmp_path) + if e["event_type"] == "checkpoint" and e["agent"] == "watchdog"] + assert [c["subtask"] for c in checkpoints].count("rate-limit-pause") == 1 + actions = {t["action"] for t in _tick_trace(memory_root)} + assert actions == {"pause"} + assert all(t["never_block"] == "rate_limit" for t in _tick_trace(memory_root)) + # Died while paused → RATE_LIMITED (not COMPLETED) with the parsed reset time. + assert result.status == AgentStatus.RATE_LIMITED + assert result.resume_at == datetime(2026, 8, 17, 10, 30, 15, tzinfo=UTC) + + def test_seeded_static_banner_resumes_at_reset_without_rollover( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, tmp_path: Path, + ) -> None: + """Regression: the real TUI never clears the usage-limit line. Past the + parsed reset an UNCHANGED banner is stale → one resume nudge (not a + day-rollover extension); heartbeat progress with the banner still on + screen is a verified RESUME; the spent banner is then ignored.""" + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + + def text_for_poll(k: int) -> str: + if k == 7: # Claude worked after the resume nudge; banner still visible + _plant_heartbeat(memory_root, tick_count=6, at=clock.now) + return BANNER_PANE # never changes + + scenario = _TmuxScenario(alive_polls=9, text_for_poll=text_for_poll) + spy = _StatusSpy(wd_wrapper) + with spy: + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(), memory_root=memory_root) + cfg = _wd_config() + assert paste.call_count == 1 + assert paste.call_args == mock.call("%5", cfg.nudge_message) + trace = _tick_trace(memory_root) + actions = [t["action"] for t in trace] + assert actions[:6] == ["pause"] * 6 # 10:00..10:05 < 10:05:15 + assert actions[6] == "resume_nudge" # 10:06: banner unchanged → stale, not extended + assert actions[7] == "resume" # verified by heartbeat delta, banner still on screen + assert trace[8]["never_block"] is None and actions[8] == "none" # spent banner ignored + assert "escalate" not in actions + assert spy.statuses[7] == AgentStatus.RUNNING + state = load_state(memory_root) + assert state is not None and state.paused_at is None and state.paused_until is None + assert state.total_paused_sec == 420.0 and state.pause_attempts == 1 + assert result.status == AgentStatus.COMPLETED # pause resolved → not RATE_LIMITED + assert result.pause_total_sec == 420.0 + + def test_permission_dialog_is_never_nudged( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + ) -> None: + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + scenario = _TmuxScenario(alive_polls=14, text_for_poll=lambda k: DIALOG_PANE) + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(), memory_root=memory_root) + assert not paste.called + trace = _tick_trace(memory_root) + assert len(trace) >= 14 # the watchdog actually ticked (not vacuous) + assert all(t["never_block"] == "awaiting_input" for t in trace) + assert "nudge" not in {t["action"] for t in trace} + assert result.status == AgentStatus.COMPLETED + + def test_prose_rate_limit_mention_does_not_misclassify_finished_session( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + ) -> None: + """Regression: a lead that finishes with a summary *mentioning* a rate + limit (prose, no reset time, no banner) and sits idle at the prompt is + COMPLETED when the pane closes — not RATE_LIMITED for the driver to + relaunch. An ML 'patience limit reached' line is not even a pause.""" + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + prose = ("Done. Note: the API hit a rate limit earlier and retried; " + "all tests pass.\n\n❯ \n") + scenario = _TmuxScenario(alive_polls=2, text_for_poll=lambda k: prose) + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(rate_limit_backoff_base_sec=600), + memory_root=memory_root) + trace = _tick_trace(memory_root) + assert trace[0]["never_block"] == "rate_limit" and trace[0]["action"] == "pause" + assert not paste.called # closed within the backoff: no probe was due + assert result.status == AgentStatus.COMPLETED and result.resume_at is None + ml = "Training finished. early stopping: patience limit reached at epoch 30\n\n❯ \n" + scenario = _TmuxScenario(alive_polls=3, text_for_poll=lambda k: ml) + result, _, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(), memory_root=memory_root) + assert result.status == AgentStatus.COMPLETED + assert all(t["never_block"] is None for t in _tick_trace(memory_root)[len(trace):]) + + def test_busy_pane_never_sends_keys_and_escalates_after_grace( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, tmp_path: Path, + ) -> None: + """Regression: a persistently busy pane (spinner / 'esc to interrupt') + cannot be nudged, so a stall there must ESCALATE once after + ``escalate_grace_sec`` — not return NUDGE forever with nothing sent.""" + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + scenario = _TmuxScenario(alive_polls=14, text_for_poll=lambda k: BUSY_PANE) + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, + watchdog=_wd_config(escalate_grace_sec=60), + memory_root=memory_root) + assert not paste.called + actions = [t["action"] for t in _tick_trace(memory_root)] + assert "nudge" not in actions + assert actions[:10] == ["none"] * 10 # 600 s threshold → stall at tick 10 + assert actions[11] == "escalate" and actions.count("escalate") == 1 # +60 s grace + blocking = [e for e in _comms_events(tmp_path) + if e["event_type"] == "error" and e["error_type"] == "stall" + and e["severity"] == "blocking"] + assert len(blocking) == 1 and "busy" in blocking[0]["description"] + assert result.status == AgentStatus.STALLED # died with no progress since escalation + + def test_nudge_guard_skips_when_pane_turns_busy_before_paste( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, tmp_path: Path, + ) -> None: + """The paste-time pane-ready guard still holds (race: verdict on an idle + capture, pane busy by paste time) → nothing sent, one 'nudge-skipped'.""" + wd_wrapper._start_watchdog(_wd_config(), memory_root=memory_root, + zo_session_id="zo-s1", delivery_repo=None) + assert wd_wrapper._wd is not None + lead = LeadProcess(tmux_pane_id="%5", team_name="alpha") + verdict = StallVerdict(action=StallAction.NUDGE, stalled=True, reason="r", + evaluated_at=clock.now) + with mock.patch.object(LifecycleWrapper, "_paste_and_submit") as paste: + wd_wrapper._wd_nudge(lead, verdict, text=BUSY_PANE) + wd_wrapper._wd_nudge(lead, verdict, text=BUSY_PANE) # same episode: logged once + assert not paste.called + assert wd_wrapper._wd.state.nudges_used == 0 + subtasks = [e["subtask"] for e in _comms_events(tmp_path) + if e["event_type"] == "checkpoint" and e["agent"] == "watchdog"] + assert subtasks == ["nudge-skipped"] + + +# ---- rate-limit pause auto-resumes on reset (oracle check 12) ---- + + +class TestSeededRateLimitResumeTmux: + """Seeded half (check 12): banner with 'resets at' → pause; past reset with + banner gone → one resume nudge; heartbeat delta → verified resume.""" + + def test_seeded_rate_limit_pause_auto_resumes_on_reset( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, tmp_path: Path, + ) -> None: + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + + def text_for_poll(k: int) -> str: + if k == 7: # heartbeat delta lands right after the resume nudge + _plant_heartbeat(memory_root, tick_count=6, at=clock.now) + return BANNER_PANE if k < 3 else IDLE_PANE + + scenario = _TmuxScenario(alive_polls=8, text_for_poll=text_for_poll) + spy = _StatusSpy(wd_wrapper) + with spy: + # timeout=400 < 420 s of wall time spent paused: must NOT time out. + result, paste, _ = _run_tmux(wd_wrapper, clock, scenario, watchdog=_wd_config(), + memory_root=memory_root, timeout=400, + patch_monotonic=True) + + cfg = _wd_config() + assert paste.call_count == 1 + assert paste.call_args == mock.call("%5", cfg.nudge_message) + actions = [t["action"] for t in _tick_trace(memory_root)] + assert actions[0] == "pause" + assert actions[6] == "resume_nudge" + assert actions[7] == "resume" + assert spy.statuses[0] == AgentStatus.PAUSED_RATE_LIMIT + assert spy.statuses[7] == AgentStatus.RUNNING + subtasks = [e["subtask"] for e in _comms_events(tmp_path) + if e["event_type"] == "checkpoint" and e["agent"] == "watchdog"] + assert subtasks.count("rate-limit-pause") == 1 + assert subtasks.count("rate-limit-resume") == 1 + # paused_until parsed from "resets at 10:05" (+15 s slack). + state = load_state(memory_root) + assert state is not None and state.total_paused_sec == 420.0 + assert result.pause_total_sec == 420.0 + assert result.status == AgentStatus.COMPLETED # not TIMED_OUT, not STALLED + assert result.paused_until is None + + +# ---- headless harness: same three mechanisms (checks 11-12) ---- + + +class TestSeededHeadless: + """Headless: no input channel (never a paste); stall → kill → STALLED; + running + banner → PAUSED; exited + banner → RATE_LIMITED with resume_at.""" + + NEUTRAL = "Working on task 1\nWorking on task 2\n" + BANNER = "You've hit your usage limit \u00b7 resets at 10:05\n" + + @mock.patch("zo.wrapper.os.kill") + def test_seeded_10min_stall_kills_headless_session_and_returns_stalled( + self, mock_kill: mock.MagicMock, wd_wrapper: LifecycleWrapper, clock: FakeClock, + memory_root: Path, tmp_log_dir: Path, tmp_path: Path, + ) -> None: + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + # Hard bound: the process "finishes" on its own after 20 polls, so a + # regression that drops the escalation/kill FAILS (COMPLETED) instead + # of hanging the suite forever. + polls = iter([None] * 20 + [0]) + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=lambda: next(polls), + stdout_text=self.NEUTRAL) + with mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep), \ + mock.patch.object(wd_wrapper, "kill_session", wraps=wd_wrapper.kill_session) as kill, \ + mock.patch.object(LifecycleWrapper, "_paste_and_submit") as paste, \ + mock.patch("zo._wrapper_watchdog.process_tree_cpu_seconds", return_value=0.0): + result = wd_wrapper.wait_for_completion( + lead, poll_interval=POLL, watchdog=_wd_config(), memory_root=memory_root) + assert not paste.called + assert kill.call_count == 1 + assert mock_kill.call_args_list[0] == mock.call(99, signal.SIGTERM) + assert result.status == AgentStatus.STALLED + assert result.stalled is True and result.exit_code == -9 + actions = [t["action"] for t in _tick_trace(memory_root)] + assert actions[:10] == ["none"] * 10 and actions[10] == "escalate" + assert actions.count("escalate") == 1 and len(actions) == 11 # killed on that tick + blocking = [e for e in _comms_events(tmp_path) + if e["event_type"] == "error" and e["error_type"] == "stall" + and e["severity"] == "blocking"] + assert len(blocking) == 1 + + @mock.patch("zo.wrapper.os.kill") + def test_seeded_silent_training_with_busy_cpu_is_not_killed( + self, mock_kill: mock.MagicMock, wd_wrapper: LifecycleWrapper, clock: FakeClock, + memory_root: Path, tmp_log_dir: Path, tmp_path: Path, + ) -> None: + """Regression (40-minute training question): a headless lead that prints + nothing and ticks no heartbeat but whose process tree burns CPU is + WORKING, not stalled — no escalation, no SIGTERM, COMPLETED.""" + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + polls = iter([None] * 40 + [0]) # 40 min of silence, then a clean exit + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=lambda: next(polls), + stdout_text=self.NEUTRAL) + # Process-tree CPU time == wall time (one core saturated by train.py). + cpu = mock.MagicMock(side_effect=lambda pid: clock.mono) + with mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep), \ + mock.patch.object(wd_wrapper, "kill_session") as kill, \ + mock.patch("zo._wrapper_watchdog.process_tree_cpu_seconds", cpu): + result = wd_wrapper.wait_for_completion( + lead, poll_interval=POLL, watchdog=_wd_config(), memory_root=memory_root) + assert cpu.call_args_list and cpu.call_args_list[0] == mock.call(99) + assert not kill.called and not mock_kill.called + assert result.status == AgentStatus.COMPLETED and result.stalled is False + trace = _tick_trace(memory_root) + assert len(trace) == 40 and {t["action"] for t in trace} == {"none"} + assert all(t["progress"] for t in trace[1:]) # first sample only baselines + assert not [e for e in _comms_events(tmp_path) + if e["event_type"] == "error" and e["error_type"] == "stall"] + + def test_seeded_rate_limited_headless_session_is_paused_never_nudged( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + tmp_log_dir: Path, tmp_path: Path, + ) -> None: + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=[None] * 5 + [0], + stdout_text=self.BANNER) + spy = _StatusSpy(wd_wrapper) + with mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep), \ + mock.patch.object(LifecycleWrapper, "_paste_and_submit") as paste, spy: + result = wd_wrapper.wait_for_completion( + lead, poll_interval=POLL, watchdog=_wd_config(), memory_root=memory_root) + assert not paste.called + assert spy.statuses[0] == AgentStatus.PAUSED_RATE_LIMIT + assert "nudge" not in {t["action"] for t in _tick_trace(memory_root)} + subtasks = [e["subtask"] for e in _comms_events(tmp_path) + if e["event_type"] == "checkpoint" and e["agent"] == "watchdog"] + assert subtasks.count("rate-limit-pause") == 1 + # Exited while paused → RATE_LIMITED with the parsed reset time. + assert result.status == AgentStatus.RATE_LIMITED + assert result.resume_at == datetime(2026, 8, 17, 10, 5, 15, tzinfo=UTC) + + def test_seeded_headless_pause_resumes_on_heartbeat_progress( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + tmp_log_dir: Path, tmp_path: Path, + ) -> None: + """check 12 headless: resume requires progress (no nudge channel).""" + _plant_heartbeat(memory_root, tick_count=5, at=T0 - timedelta(hours=1)) + polls = {"n": 0} + + def poll() -> int | None: + polls["n"] += 1 + if polls["n"] == 8: # heartbeat delta after the pause window + _plant_heartbeat(memory_root, tick_count=6, at=clock.now) + return 0 if polls["n"] > 9 else None + + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=poll, stdout_text=self.BANNER) + spy = _StatusSpy(wd_wrapper) + with mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep), \ + mock.patch.object(LifecycleWrapper, "_paste_and_submit") as paste, spy: + result = wd_wrapper.wait_for_completion( + lead, poll_interval=POLL, watchdog=_wd_config(), memory_root=memory_root) + assert not paste.called + actions = [t["action"] for t in _tick_trace(memory_root)] + assert actions[0] == "pause" and "resume" in actions + assert "resume_nudge" not in actions and "nudge" not in actions + assert spy.statuses[0] == AgentStatus.PAUSED_RATE_LIMIT + assert spy.statuses[-1] == AgentStatus.RUNNING + assert result.status == AgentStatus.COMPLETED + assert result.pause_total_sec > 0 + subtasks = [e["subtask"] for e in _comms_events(tmp_path) + if e["event_type"] == "checkpoint" and e["agent"] == "watchdog"] + assert "rate-limit-resume" in subtasks + + def test_headless_clean_exit_mentioning_rate_limit_stays_completed( + self, wd_wrapper: LifecycleWrapper, clock: FakeClock, memory_root: Path, + tmp_log_dir: Path, + ) -> None: + """rc == 0 + prose 'rate limit' in the JSON result → COMPLETED (the + pause was prose-only: no reset, no banner, no non-zero rc).""" + text = '{"result": "Done. The API hit a rate limit once and retried."}\n' + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=[None, None, 0], stdout_text=text) + with mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep): + result = wd_wrapper.wait_for_completion( + lead, poll_interval=POLL, watchdog=_wd_config(), memory_root=memory_root) + assert _tick_trace(memory_root)[0]["action"] == "pause" # conservative while running + assert result.status == AgentStatus.COMPLETED and result.resume_at is None + # Same text but a non-zero exit is corroboration → RATE_LIMITED. + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=[None, 1], stdout_text=text) + with mock.patch("zo.wrapper.time.sleep", side_effect=clock.sleep): + result = wd_wrapper.wait_for_completion( + lead, poll_interval=POLL, watchdog=_wd_config(), memory_root=memory_root) + assert result.status == AgentStatus.RATE_LIMITED + + def test_read_new_output_uses_byte_cursor_over_both_logs( + self, wd_wrapper: LifecycleWrapper, tmp_log_dir: Path, + ) -> None: + lead = _headless_setup(wd_wrapper, tmp_log_dir, poll=None, stdout_text="a\n") + assert lead.stderr_log is not None + lead.stderr_log.write_text("e1\n") + assert wd_wrapper._read_new_output(lead) == "a\ne1\n" + assert wd_wrapper._read_new_output(lead) == "" # nothing new + with open(lead.stdout_log, "a") as fh: # type: ignore[arg-type] + fh.write("b\n") + assert wd_wrapper._read_new_output(lead) == "b\n" + assert wd_wrapper._wd_text_window == "a\ne1\nb\n" + + +# ---- WatchdogRunner unit behaviour ---- + + +class TestWatchdogRunner: + def test_start_baselines_existing_heartbeats_and_ticks_write_trace( + self, memory_root: Path, clock: FakeClock, + ) -> None: + _plant_heartbeat(memory_root, tick_count=7, at=T0 - timedelta(minutes=5)) + runner = WatchdogRunner(config=_wd_config(), memory_root=memory_root, + zo_session_id="zo-s1", clock=clock) + runner.start() + assert runner.state.baseline_ticks == {"lead-s1": 7} + proc = LeadProcess(tmux_pane_id="%1") + v = runner.tick(process=proc, text=IDLE_PANE, can_nudge=True) + assert v.progress is False # pre-existing file does not count + _plant_heartbeat(memory_root, tick_count=8, at=clock.now) + v = runner.tick(process=proc, text=IDLE_PANE, can_nudge=True) + assert v.progress is True + trace = _tick_trace(memory_root) + assert len(trace) == 2 and set(trace[0]) == { + "ts", "action", "stalled", "reason", "never_block", "progress"} + assert (memory_root / "heartbeats" / "_watchdog.json").exists() + + def test_nudge_budget_survives_runner_restart_same_session( + self, memory_root: Path, clock: FakeClock, + ) -> None: + r1 = WatchdogRunner(config=_wd_config(), memory_root=memory_root, + zo_session_id="zo-s1", clock=clock) + r1.start() + r1.record_nudge() + r1.record_nudge() + r2 = WatchdogRunner(config=_wd_config(), memory_root=memory_root, + zo_session_id="zo-s1", clock=clock) + r2.start() + assert r2.state.nudges_used == 2 + r3 = WatchdogRunner(config=_wd_config(), memory_root=memory_root, + zo_session_id="zo-other", clock=clock) + r3.start() + assert r3.state.nudges_used == 0 + + def test_paused_seconds_includes_open_pause(self, memory_root: Path, clock: FakeClock) -> None: + runner = WatchdogRunner(config=_wd_config(), memory_root=memory_root, clock=clock) + runner.start() + proc = LeadProcess(tmux_pane_id="%1") + v = runner.tick(process=proc, text=BANNER_PANE, can_nudge=True) + assert v.action.value == "pause" + clock.advance(120) + assert runner.paused_seconds() == 120.0 + assert runner.is_rate_limited is True + + def test_paused_seconds_stops_accruing_at_escalation( + self, memory_root: Path, clock: FakeClock, + ) -> None: + """Regression: banner gone + no progress → 'resume unverified' escalation; + the open pause must NOT keep suspending the wall-clock timeout forever.""" + runner = WatchdogRunner(config=_wd_config(), memory_root=memory_root, clock=clock, tz=UTC) + runner.start() + proc = LeadProcess(tmux_pane_id="%1") + plain = "Rate limit reached. Try again later.\n\n❯ \n" + assert runner.tick(process=proc, text=plain, can_nudge=False).action.value == "pause" + clock.advance(61) # backoff (60 s) elapsed, banner gone, headless-style (no nudge) + assert runner.tick(process=proc, text=IDLE_PANE, can_nudge=False).action.value == "none" + clock.advance(60) + v = runner.tick(process=proc, text=IDLE_PANE, can_nudge=False) + assert v.action.value == "escalate" and "unverified" in v.reason + assert runner.paused_seconds() == 121.0 + clock.advance(3600) + runner.tick(process=proc, text=IDLE_PANE, can_nudge=False) + assert runner.paused_seconds() == 121.0 # capped at the escalation + assert runner.state.paused_at is not None # a late verified resume still wins + + def test_tz_threads_into_banner_parse(self, memory_root: Path, clock: FakeClock) -> None: + """Regression: 'resets at 3pm' is the operator's LOCAL 3pm, not 15:00 UTC.""" + tz = timezone(timedelta(hours=-7)) # e.g. US/Pacific in summer + clock.now = datetime(2026, 8, 17, 20, 0, tzinfo=UTC) # 13:00 local + runner = WatchdogRunner(config=_wd_config(), memory_root=memory_root, clock=clock, tz=tz) + runner.start() + v = runner.tick(process=LeadProcess(tmux_pane_id="%1"), + text="You've hit your usage limit · resets at 3pm\n❯ \n", + can_nudge=True) + assert v.action.value == "pause" + assert runner.state.paused_until == datetime(2026, 8, 17, 22, 0, 15, tzinfo=UTC) + assert runner.parsed_resume_at() == runner.state.paused_until + default = WatchdogRunner(config=_wd_config(), memory_root=memory_root, clock=clock) + assert default.tz == datetime.now().astimezone().tzinfo # local by default + + def test_cpu_busy_process_tree_counts_as_progress( + self, memory_root: Path, clock: FakeClock, + ) -> None: + cpu = {"secs": 0.0} + runner = WatchdogRunner(config=_wd_config(), memory_root=memory_root, clock=clock, + cpu_probe=lambda pid: cpu["secs"]) + runner.start() + proc = LeadProcess(pid=4242) + assert runner.tick(process=proc, text="", can_nudge=False, + process_dead=False).progress is False # baseline sample + clock.advance(60) + cpu["secs"] += 3.0 # 5 % of wall: an idle TUI redrawing, not work + assert runner.tick(process=proc, text="", can_nudge=False, + process_dead=False).progress is False + clock.advance(60) + cpu["secs"] += 30.0 # 50 % of wall: a training job is running + assert runner.tick(process=proc, text="", can_nudge=False, + process_dead=False).progress is True + # No pid → no CPU evidence (never invents progress). + assert runner.tick(process=LeadProcess(tmux_pane_id="%1"), text="", + can_nudge=True).progress is False + + def test_tick_never_probes_death_without_pid(self, memory_root: Path, clock: FakeClock) -> None: + probe = mock.MagicMock(return_value=True) + runner = WatchdogRunner(config=_wd_config(), memory_root=memory_root, clock=clock, + dead_probe=probe) + runner.start() + v = runner.tick(process=LeadProcess(tmux_pane_id="%1"), text=IDLE_PANE, can_nudge=True) + assert v.process_dead is None and not probe.called + v = runner.tick(process=LeadProcess(pid=4242, pid_start_identity="linux:1"), + text=IDLE_PANE, can_nudge=False) + assert v.process_dead is True + probe.assert_called_once_with(4242, "linux:1") + + def test_advisory_paths_fail_open(self, tmp_path: Path, clock: FakeClock) -> None: + """Unwritable memory root: ticks still return verdicts, nothing raises.""" + blocked = tmp_path / "blocked.txt" + blocked.write_text("not a dir") + runner = WatchdogRunner(config=_wd_config(), memory_root=blocked, clock=clock) + runner.start() + v = runner.tick(process=LeadProcess(tmux_pane_id="%1"), text=IDLE_PANE, can_nudge=True) + assert v.action.value == "none" + runner.stop() From 6fe97754b8eb1410a90c074911b5e5c9a2a84cf1 Mon Sep 17 00:00:00 2001 From: SamT <35964759+SamPlvs@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:43:22 +0100 Subject: [PATCH 2/3] =?UTF-8?q?test(wrapper):=20never=20spawn=20the=20real?= =?UTF-8?q?=20CPU=20probe=20from=20unit=20tests=20=E2=80=94=20fixes=20CI-o?= =?UTF-8?q?nly=20red=20on=203.11/3.12?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The watchdog CPU-evidence probe ran ps -A via subprocess.run(timeout=5) inside the headless loop tests; CPython's Popen.wait doubling back-off sleeps leaked into mock_sleep.call_args_list because mock.patch("zo.wrapper.time.sleep") patches the global time module. Race-dependent reap timing made it red on Linux CI and green on macOS. Autouse fixture patches zo._wrapper_watchdog.process_tree_cpu_seconds -> None (CPU unknown), proven with a counting Popen spy (2 spawns before, 0 after). PRIORS PR-049 + DECISION_LOG failure note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- memory/zo-platform/DECISION_LOG.md | 6 ++++++ memory/zo-platform/PRIORS.md | 18 ++++++++++++++++++ tests/unit/test_wrapper.py | 20 +++++++++++++++++++- 3 files changed, 43 insertions(+), 1 deletion(-) diff --git a/memory/zo-platform/DECISION_LOG.md b/memory/zo-platform/DECISION_LOG.md index e43f295..04c338d 100644 --- a/memory/zo-platform/DECISION_LOG.md +++ b/memory/zo-platform/DECISION_LOG.md @@ -1305,3 +1305,9 @@ The `--no-headlines` flag is preserved (not removed) for backwards compatibility **Verification method:** contract-first build (4 concurrent builders on disjoint files) → integrator → 3 adversarial verifier lenses (semantics / wiring+sealing / test quality: 19 findings, 2 high — banner reset times parsed in UTC; static banner could never resume) → fixer (11 applied with regression tests, 4 rejected with reasons). 929 → 1131 passed / 7 skipped, ruff clean, validate-docs 0 failures. Seeded tests for checks 11 and 12 on both loops. **Follow-ups (not done, recorded):** `_watchdog-ticks.jsonl` unbounded growth; `wrapper.py` 1404 lines (split `_wrapper_tmux.py`); verify the CPU-evidence idle threshold on a real tmux session; sealed-prefix symlink resolution in hookkit; `is_interrupt` from the failure feed not yet fed to `evaluate()`. + +## Decision: 2026-08-17T13:05:00Z +**Type:** FAILURE + FIX +**Title:** PR #109 CI red on 3.11/3.12 (green locally on 3.14) — CPU-evidence probe spawned `ps -A` under a global `time.sleep` mock + +**Failure:** `test_running_process_with_rate_limit_text_pauses_without_backoff` saw CPython's `Popen.wait` doubling back-off sleeps in `mock_sleep.call_args_list` because the watchdog's CPU probe ran the real `ps -A` (`subprocess.run(timeout=5)`) and `mock.patch("zo.wrapper.time.sleep")` patches the global `time` module. Race-dependent reap timing → Linux red, macOS green. **Root cause:** `missing_rule` — no rule required neutralising new process-spawning helpers in unit tests. **Fix:** autouse fixture `_no_real_cpu_probe` in `tests/unit/test_wrapper.py` (probe → `None`); proven with a counting `Popen` spy (2 spawns before, 0 after). **Prior:** PR-049 (five rules incl. "raising traps are swallowed by fail-open code — use counting spies" and "default-arg binding defeats late patching"). Suite 1131 / 7 skipped, ruff clean; pushed to #109. diff --git a/memory/zo-platform/PRIORS.md b/memory/zo-platform/PRIORS.md index 036a94b..41e603c 100644 --- a/memory/zo-platform/PRIORS.md +++ b/memory/zo-platform/PRIORS.md @@ -1441,3 +1441,21 @@ Not a code fix in PR-A — the finding shaped Phase 3's design (DECISION_LOG 202 ### Verified Solution `scratchpad/pr-a-build.js` (session 041) rewritten from Core → Build(3) to Build(4 concurrent) → Integrate → Verify(3) → Fix; final: 1131 passed / 7 skipped, 19 verifier findings triaged, no integration defect reached the commit. Same playbook for PR-B. + +## PR-049: Unit tests must never spawn the real process probes — a fail-open probe hides the leak, and `mock.patch("zo.wrapper.time.sleep")` patches the GLOBAL `time.sleep` +**Source:** Session 041 (2026-08-17), PR #109 CI red on Python 3.11 + 3.12 after a green local run on 3.14 +**Root cause category:** missing_rule + +**Failure:** `tests/unit/test_wrapper.py::TestWaitForCompletion::test_running_process_with_rate_limit_text_pauses_without_backoff` asserted `{c.args[0] for c in mock_sleep.call_args_list} == {0.01}` ("no back-off: every sleep is the poll interval"). On CI the recorded sleeps were `{0.001, 0.002, 0.004, 0.008, 0.01, 0.016, 0.032, 0.05}`. Cause chain: the new WatchdogRunner samples process-tree CPU time every tick when the lead has a pid → `zo._proc.process_tree_cpu_seconds` runs `ps -A` via `subprocess.run(timeout=5)` → CPython's `Popen.wait(timeout)` polls with a doubling `time.sleep(0.0005·2ⁿ, cap 0.05)` while the child is reaped → `mock.patch("zo.wrapper.time.sleep")` resolves `zo.wrapper.time` to the `time` MODULE, so it patches `time.sleep` for every caller including `subprocess` → the internal back-off sleeps landed in the assertion. Race-dependent (`ps` reaped before the first `WNOHANG` on macOS, after it on Linux) → green locally, red on both CI Pythons. Two things hid it: the probe is fail-open (`except Exception: return None`), so a raising trap on `subprocess.Popen` was swallowed silently and "proved" nothing; and `process_tree_cpu_seconds(..., run=subprocess.run)` binds `run` at definition time, so patching `subprocess.run` afterwards is inert. + +### Rules + +1. **Any new helper that spawns a process gets an autouse neutralising fixture in every unit-test module whose code path can reach it.** Unit tests must not depend on the host's process table or reap timing. Patch the NAME the caller imported (`zo._wrapper_watchdog.process_tree_cpu_seconds`), not the stdlib function it wraps. +2. **`mock.patch("<module>.time.sleep")` is a global patch** — `<module>.time` is the shared `time` module. Assertions on `mock_sleep.call_args_list` therefore see every sleeper in the process (subprocess, threading, retries). Either isolate the loop from all other sleepers (rule 1) or assert on the loop's own sleeps by value (`poll_interval in calls`), never on the full set. +3. **Verify a "nothing spawns" claim with a counting spy, not a raising trap.** Fail-open code swallows the trap's AssertionError; a spy that records and delegates (`class Spy(subprocess.Popen)`) shows the true count with and without the fix. +4. **Default-argument binding defeats late patching.** `def f(*, run=subprocess.run)` captures the function object; tests that need to intercept must inject `run=` or patch the wrapper's imported name — document which one at the definition. +5. **CI (3.11/3.12 on Linux) is the binding check; local 3.14 on macOS is a preview** (PR-039 family). A CI-only failure after a green local run means an environment-dependent path — find the spawn/race, do not retry. + +### Verified Solution + +`tests/unit/test_wrapper.py`: autouse fixture `_no_real_cpu_probe` patches `zo._wrapper_watchdog.process_tree_cpu_seconds` to return `None` (CPU unknown = no evidence — the default the tests want; the two CPU-evidence tests patch explicitly and win). Proof by counting spy: the failing test spawned `ps -A` ×2 without the fixture and 0 with it. 90 wrapper tests pass; full suite 1131 / 7 skipped; ruff clean. The rule would have caught the original failure: with rule 1 applied at build time the probe could never have run under a global sleep mock. diff --git a/tests/unit/test_wrapper.py b/tests/unit/test_wrapper.py index 9e6a0a6..b7daeff 100644 --- a/tests/unit/test_wrapper.py +++ b/tests/unit/test_wrapper.py @@ -38,13 +38,31 @@ from zo.wrapper import LifecycleWrapper if TYPE_CHECKING: - from collections.abc import Callable + from collections.abc import Callable, Iterator # ------------------------------------------------------------------ # # Fixtures # ------------------------------------------------------------------ # +@pytest.fixture(autouse=True) +def _no_real_cpu_probe() -> Iterator[None]: + """Never spawn the real ``ps -A`` CPU probe from a wrapper unit test. + + The runner samples process-tree CPU time on every tick when the lead has + a pid. Left unpatched, ``subprocess.run(timeout=...)`` inside the probe + reaches CPython's ``Popen.wait`` doubling back-off (``time.sleep(0.001, + 0.002, ...)``) — and ``mock.patch("zo.wrapper.time.sleep")`` patches the + *global* ``time.sleep``, so those internal sleeps leaked into the poll + interval assertions on CI (race-dependent: seen on Linux 3.11/3.12, + not on macOS 3.14). ``None`` = "CPU unknown", which is exactly the + evidence-free default the tests want; the two tests that exercise CPU + evidence patch the probe explicitly and win over this fixture. + """ + with mock.patch("zo._wrapper_watchdog.process_tree_cpu_seconds", return_value=None): + yield + + @pytest.fixture() def tmp_log_dir(tmp_path: Path) -> Path: d = tmp_path / "logs" / "wrapper" From 52ae20cd9a0813218d8dfd6c10a0ec4835b732d9 Mon Sep 17 00:00:00 2001 From: SamT <35964759+SamPlvs@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:44:35 +0100 Subject: [PATCH 3/3] =?UTF-8?q?docs(memory):=20session-041=20STATE=20?= =?UTF-8?q?=E2=80=94=20PR=20#109=20CI=20green=20on=203.11/3.12,=20awaiting?= =?UTF-8?q?=20merge?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- memory/zo-platform/STATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/memory/zo-platform/STATE.md b/memory/zo-platform/STATE.md index 0e48075..5d29eaa 100644 --- a/memory/zo-platform/STATE.md +++ b/memory/zo-platform/STATE.md @@ -8,7 +8,7 @@ status: complete ## Current Position -**Session 041 (current) — pick up here.** v2 **Phase 3 (WS-C execution substrate) — part 1 of 2 SHIPPED: PR-A the watchdog** (plan oracle checks 11–12) on branch `claude/v2-phase3-substrate` (cut off main after **PR #108 / Phase 2 merged** at the start of this session, squash `1faf53a`). Method: read-only recon swarm (7 mappers + synthesis → `memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md`, every claim `file:line`) → build contract (`pr-a-build-contract.md`, contract-first: pinned `zo.watchdog` API, heartbeat schema, tick algorithm, per-builder file ownership) → 4 concurrent builders → integrator → 3 adversarial verifiers (19 findings, 2 high) → fixer (11 applied with regression tests, 4 rejected with reasons). **Structural finding that reframes Phase 3:** `Orchestrator.advance_phase()` / `mark_subtask_complete()` have ZERO runtime callers — the automated gate, `_auto_iterate_if_needed`, the WS-B `mark_phase_passed` flip and gate-nonce minting are unreachable in production; `zo build` = one lead session per phase then `end_session()`; phases only ever advanced via hand-edited STATE.md (the PR-036/037 prod-001 incident). PR-B (the driver) is therefore the first runtime caller, not a Phase-4-only optimisation. **Sam decided (this session):** (1) the driver evaluates gates for ALL phases (Phase 4 alone gets fresh headless spawns per iteration until check 13 says extend); (2) two PRs (A watchdog, B driver + fresh-context loop + deferrals); (3) after the restore cutover the ledger wins over STATE.md with a loud warning + a sanctioned `zo phase set` override logged to DECISION_LOG; (4) tmux nudges default ON behind a pane-ready / no-permission-dialog guard. **PR-A shipped:** `src/zo/watchdog.py` (+`_watchdog_models.py`, `_watchdog_text.py`, `_proc.py`): three-state freshness (unknown never = stall), never-block taxonomy ported from OMC (context-limit #213 / rate-limit #777 / auth #1308 / user-abort incl. `⎿ Interrupted by user`) + ZO-added `awaiting_input` (a permission dialog is never sent Enter) + `compacting`; tiered rate-limit patterns (no bare `429`/`overloaded`); `parse_rate_limit_reset` (local tz); PID + process-start-time identity, positive-proof-only (EPERM = alive, recycled pid = dead); pure `evaluate()` policy (progress → never-block → pause/resume → grace → stall → nudge budget 3 with 30 s dwell → escalate once). Heartbeats: stdlib-only `zo._hook_heartbeat.stamp_heartbeat` on a new `PostToolUse *` settings.json entry (+ Stop→ready, PreCompact→compacting, SubagentStop/SessionEnd→shutdown), keyed `agent_id` or `lead-<session_id>`, written to `<memory_root>/heartbeats/`, **sealed** (`_SEALED_DEFAULTS`) and gitignored (platform root + delivery `.zo/` templates). Wrapper: `WatchdogRunner` (`_wrapper_watchdog.py`) ticked from BOTH loops before the liveness reads (incl. the suspected-dead `continue` path); one pane capture per poll; evidence = heartbeat tick deltas (pre-existing files baselined) + normalized pane/stdout digest (spinners/counters stripped) + progress-path mtimes (ledger, comms dir, `.zo/experiments`) + process-tree CPU time (a silent 40-min training call is NOT a stall); tmux nudge via named paste buffer only when the pane shows the idle prompt; rate-limit = paused state evaluated per poll (no blocking sleep), reset-time parse → probe → verified resume by real progress (static banner gets a bounded resume nudge); timeout excludes paused time; escalation: tmux logs `error_type=stall severity=blocking` + `STALLED` at exit (never kills a human-facing pane), headless kills (`kill_headless_on_escalate`, config). New `AgentStatus.PAUSED_RATE_LIMIT/STALLED`, `LeadProcess.pid_start_identity/resume_at/...`; headless retry loop REMOVED (exit → `RATE_LIMITED` + `resume_at` for the PR-B driver). Config: `ProjectConfig.watchdog: WatchdogConfig` (round-trips), `zo build/continue --no-watchdog`, `ZO_WATCHDOG=0`, `ZO_WATCHDOG_STALL_SEC`; CLI threads watchdog/memory_root/`ZO_SESSION_ID`. Docs: `specs/watchdog.md` rewritten to implemented reality (RFC cron-tick superseded — divergence stated up front), mdx status, COMMANDS.md/build.mdx flag. **Live evidence in this very session:** the heartbeat hook fired for the lead (`lead-<session_id>.json`, `executing/Bash`) and for the workflow subagents (`agent_type=workflow-subagent`, `tick_count` 44 → `shutdown` on SubagentStop) — PostToolUse DOES carry `agent_id`/`agent_type` for subagents (open question resolved). **Correction to session 040:** `PostToolUseFailure` DOES fire on nonzero-exit Bash (`logs/comms/failures-2026-08-17.jsonl` captured this session's own `git checkout` exit 1) — the "infra errors only, queue for WS-D" caveat was wrong; no WS-D work needed there. **929 → 1131 passed / 7 skipped, ruff `src/` clean, validate-docs 0 failures (badge 1053).** Seeded tests: 10-min stall detected + escalated within one poll (both loops), rate-limited session never nudged (both loops), rate-limit pause auto-resumes on reset with verified progress (check 12, both loops), permission dialog never nudged, heartbeats sealed, settings wiring, tick invoked from both loops, CLI threading. **Next:** PR-B on the same branch after PR-A merges (or stacked): `zo.driver.run_phase_loop` — first runtime caller of `advance_phase`; fresh headless spawns per Phase-4 iteration (`_launch_headless`, prompt on stdin, process-group kill); oracle-side subtask completion from ledger criteria; git checkpoint via `surrogate.commit_worktree`; `evaluate_loop_state(..., ledger=)`; restore cutover ledger>STATE.md (+`LedgerEntry.completed`, HOLD write-through, decompose ordering fix, `zo phase set`); wire `parse_next_md/parse_hypothesis_md` (DEAD_END currently dead code); fix absolute `Experiment.artifacts_dir` (check-13 blocker); check 13 needs the Linux box (PR-046). Follow-ups noted, not done: `_watchdog-ticks.jsonl` unbounded, wrapper.py 1404 lines (split `_wrapper_tmux.py`), CPU-evidence idle threshold to verify on a real tmux session. +**Session 041 (current) — pick up here.** v2 **Phase 3 (WS-C execution substrate) — part 1 of 2 SHIPPED: PR-A the watchdog** (plan oracle checks 11–12) on branch `claude/v2-phase3-substrate` (cut off main after **PR #108 / Phase 2 merged** at the start of this session, squash `1faf53a`). Method: read-only recon swarm (7 mappers + synthesis → `memory/zo-platform/research/2026-08-17-phase3-recon/integration-map.md`, every claim `file:line`) → build contract (`pr-a-build-contract.md`, contract-first: pinned `zo.watchdog` API, heartbeat schema, tick algorithm, per-builder file ownership) → 4 concurrent builders → integrator → 3 adversarial verifiers (19 findings, 2 high) → fixer (11 applied with regression tests, 4 rejected with reasons). **Structural finding that reframes Phase 3:** `Orchestrator.advance_phase()` / `mark_subtask_complete()` have ZERO runtime callers — the automated gate, `_auto_iterate_if_needed`, the WS-B `mark_phase_passed` flip and gate-nonce minting are unreachable in production; `zo build` = one lead session per phase then `end_session()`; phases only ever advanced via hand-edited STATE.md (the PR-036/037 prod-001 incident). PR-B (the driver) is therefore the first runtime caller, not a Phase-4-only optimisation. **Sam decided (this session):** (1) the driver evaluates gates for ALL phases (Phase 4 alone gets fresh headless spawns per iteration until check 13 says extend); (2) two PRs (A watchdog, B driver + fresh-context loop + deferrals); (3) after the restore cutover the ledger wins over STATE.md with a loud warning + a sanctioned `zo phase set` override logged to DECISION_LOG; (4) tmux nudges default ON behind a pane-ready / no-permission-dialog guard. **PR-A shipped:** `src/zo/watchdog.py` (+`_watchdog_models.py`, `_watchdog_text.py`, `_proc.py`): three-state freshness (unknown never = stall), never-block taxonomy ported from OMC (context-limit #213 / rate-limit #777 / auth #1308 / user-abort incl. `⎿ Interrupted by user`) + ZO-added `awaiting_input` (a permission dialog is never sent Enter) + `compacting`; tiered rate-limit patterns (no bare `429`/`overloaded`); `parse_rate_limit_reset` (local tz); PID + process-start-time identity, positive-proof-only (EPERM = alive, recycled pid = dead); pure `evaluate()` policy (progress → never-block → pause/resume → grace → stall → nudge budget 3 with 30 s dwell → escalate once). Heartbeats: stdlib-only `zo._hook_heartbeat.stamp_heartbeat` on a new `PostToolUse *` settings.json entry (+ Stop→ready, PreCompact→compacting, SubagentStop/SessionEnd→shutdown), keyed `agent_id` or `lead-<session_id>`, written to `<memory_root>/heartbeats/`, **sealed** (`_SEALED_DEFAULTS`) and gitignored (platform root + delivery `.zo/` templates). Wrapper: `WatchdogRunner` (`_wrapper_watchdog.py`) ticked from BOTH loops before the liveness reads (incl. the suspected-dead `continue` path); one pane capture per poll; evidence = heartbeat tick deltas (pre-existing files baselined) + normalized pane/stdout digest (spinners/counters stripped) + progress-path mtimes (ledger, comms dir, `.zo/experiments`) + process-tree CPU time (a silent 40-min training call is NOT a stall); tmux nudge via named paste buffer only when the pane shows the idle prompt; rate-limit = paused state evaluated per poll (no blocking sleep), reset-time parse → probe → verified resume by real progress (static banner gets a bounded resume nudge); timeout excludes paused time; escalation: tmux logs `error_type=stall severity=blocking` + `STALLED` at exit (never kills a human-facing pane), headless kills (`kill_headless_on_escalate`, config). New `AgentStatus.PAUSED_RATE_LIMIT/STALLED`, `LeadProcess.pid_start_identity/resume_at/...`; headless retry loop REMOVED (exit → `RATE_LIMITED` + `resume_at` for the PR-B driver). Config: `ProjectConfig.watchdog: WatchdogConfig` (round-trips), `zo build/continue --no-watchdog`, `ZO_WATCHDOG=0`, `ZO_WATCHDOG_STALL_SEC`; CLI threads watchdog/memory_root/`ZO_SESSION_ID`. Docs: `specs/watchdog.md` rewritten to implemented reality (RFC cron-tick superseded — divergence stated up front), mdx status, COMMANDS.md/build.mdx flag. **Live evidence in this very session:** the heartbeat hook fired for the lead (`lead-<session_id>.json`, `executing/Bash`) and for the workflow subagents (`agent_type=workflow-subagent`, `tick_count` 44 → `shutdown` on SubagentStop) — PostToolUse DOES carry `agent_id`/`agent_type` for subagents (open question resolved). **Correction to session 040:** `PostToolUseFailure` DOES fire on nonzero-exit Bash (`logs/comms/failures-2026-08-17.jsonl` captured this session's own `git checkout` exit 1) — the "infra errors only, queue for WS-D" caveat was wrong; no WS-D work needed there. **929 → 1131 passed / 7 skipped, ruff `src/` clean, validate-docs 0 failures (badge 1053). PR #109 opened; first CI run red on 3.11/3.12 only (CPU probe spawned `ps -A` under the global `time.sleep` mock — PRIORS PR-049), fixed with an autouse fixture, second run GREEN on both + validate-docs, merge state CLEAN — awaiting Sam's merge.** Seeded tests: 10-min stall detected + escalated within one poll (both loops), rate-limited session never nudged (both loops), rate-limit pause auto-resumes on reset with verified progress (check 12, both loops), permission dialog never nudged, heartbeats sealed, settings wiring, tick invoked from both loops, CLI threading. **Next:** PR-B on the same branch after PR-A merges (or stacked): `zo.driver.run_phase_loop` — first runtime caller of `advance_phase`; fresh headless spawns per Phase-4 iteration (`_launch_headless`, prompt on stdin, process-group kill); oracle-side subtask completion from ledger criteria; git checkpoint via `surrogate.commit_worktree`; `evaluate_loop_state(..., ledger=)`; restore cutover ledger>STATE.md (+`LedgerEntry.completed`, HOLD write-through, decompose ordering fix, `zo phase set`); wire `parse_next_md/parse_hypothesis_md` (DEAD_END currently dead code); fix absolute `Experiment.artifacts_dir` (check-13 blocker); check 13 needs the Linux box (PR-046). Follow-ups noted, not done: `_watchdog-ticks.jsonl` unbounded, wrapper.py 1404 lines (split `_wrapper_tmux.py`), CPU-evidence idle threshold to verify on a real tmux session. **Session 040 (prior).** Research + decision session: deep-dive review of three agent-orchestration repos (oh-my-claudecode, ruflo, ralph — cloned to `~/Documents/code/`) to inform the ZO v2 rearchitecture. 9-agent workflow (7 source-reading lenses + ZO baseline + adversarial synthesis, ~1.06M tokens) catalogued **63 features**, distilled to **12 ranked adoptions** + 6 rearchitecture themes + 11 anti-patterns; all findings persisted to `memory/zo-platform/research/2026-08-12-repo-reviews/` (per-repo markdown + `raw-findings.json`). **Sam decided: adopt all 12.** Work organized into **five layer-based workstreams** (A enforcement plane, B control plane, C execution substrate, D self-learning/platform oracle, E operator experience) — NOT source-repo categories, because features from different repos interlock into single mechanisms. Shipped this session: `plans/zo-v2-rearchitecture.md` (full plan: 6 gated phases, 20-check oracle, anti-scope; + `.gitignore` exception), `docs/reference/v2-rearchitecture.mdx` (all 12 features w/ provenance + repo links; added to mint.json Reference nav), `docs/roadmap.mdx` v2 section (4 pillars + repo credits), website §11 "What's next" (new section w/ 3 repo credit cards; quick start renumbered §12; drawer nav updated). **Verification caveat: no Node.js on this machine** — Astro build NOT run; website change verified via HTML-parser balance check + static-server DOM inspection (section text, all 4 links, drawer entry, renumbering all confirmed rendered). CI/deploy build must confirm. **Same session, part 2 — v2 Phase 1 (WS-A enforcement plane) SHIPPED** on branch `claude/v2-phase1-enforcement` (stacked on the plan branch, PR #106): (A1) `src/zo/contracts.py` — contracts.json emitted at decompose into memory_root (gate_mode precedent), `contract_produced` upgraded from prose placeholders to concrete paths (ownership ∩ required_artifacts, ownership-dir fallback), SubagentStop hook validates deliverables (missing/undersized/pattern/empty-dir) and blocks with a violation list; (A2) drift-guard Stop hook — completion-claim regex over the last assistant transcript message + added TODO/FIXME/NotImplementedError lines in `git diff HEAD` → block (env kill-switch `ZO_DRIFT_GUARD=0`); (A3) PreCompact (STATE flush + checkpoint decision), SessionEnd (summary backfill), PostToolUseFailure (`logs/comms/failures-{date}.jsonl` feed) — specs/memory.md recovery section updated to match (replaces the never-built periodic postToolUse checkpoint design); (A4) sealed-paths PreToolUse guard — memory-root control files (gate_mode/gate_nonce/gate_decision/contracts.json/sealed_paths) + user `sealed_paths` prefixes denied, off-limits write-scope enforced per contracts.json when agent identity present in hook input (plan check 6 AMENDED: no disallowedTools frontmatter exists for subagents and verifiers need scoped writes — path-scoped enforcement instead, fail-open without identity); (A5) nonce gates — minted at GATED (`secrets.token_hex(8)` → `gate_nonce` file), surfaced in `prepare_gate_review`, `apply_human_decision` raises PermissionError without it (single-use, cleared on terminal decisions), new `zo gates approve/reject --nonce` CLI writes DECISION_LOG + comms + `gate_decision` file consumed on next decompose, `/approve`+`/reject` slash commands rewritten to route through the CLI (forgeable hand-edit path CLOSED). All via one shim (`.claude/hooks/zo-hookkit.sh` → `python3 -m zo.hookkit`, venv-preferring, fail-open) + 6 new settings.json wirings. **First-ever hook-script tests** (subprocess + stdin JSON pattern). **854 → 904 passed / 7 skipped, ruff `src/` clean, validate-docs green.** 4 pre-existing integration tests updated to pass the nonce (designed behaviour change). Plan oracle checks 1-5 + 7 have passing seeded-failure tests; check 6 as amended. **Part 3 — live pre-PR verification (Sam-directed):** added always-on hook-trace observability (`logs/hook-trace-{date}.jsonl`, `ZO_HOOK_TRACE=0` off-switch) and verified in the live session itself: sealed-paths DENIED a real Write to gate_mode; drift-guard fired correctly-silent on a real Stop; subagent-stop fired with `agent_type`+`agent_id` in the live payload — **agent-identity open question RESOLVED** (per-agent enforcement keys correctly). Drift-guard now prefers the live payload's `last_assistant_message` (transcript parse = fallback). Caveats logged: PostToolUseFailure doesn't fire on nonzero-exit Bash (infrastructure errors only); PreCompact/SessionEnd not yet observed live; full `zo build` demo needs a machine with the claude CLI (this Mac has none — PR-046). **908 passed / 7 skipped, ruff clean.** **Part 4 — Phases merged + Phase 2 (WS-B control plane) SHIPPED:** #106 + #107 merged to main (stack conflict resolved by merging main into the branch, branch side kept — main had nothing unique). Then WS-B on `claude/v2-phase2-control-plane`: `src/zo/ledger.py` (plan-ledger.json: per-subtask entries w/ synthesized criteria, merge-preserving regeneration, atomic writes, phase_status map); oracle-owned flips wired at the two verified-completion sites (automated gate + nonce-verified human PROCEED), resets on ITERATE/loop-CONTINUE, attempts on mark_subtask_complete, ledger sealed via `_SEALED_DEFAULTS`; `## Stories` plan section + sizing lint in validate_plan (fires only when stories declared — legacy plans untouched); `zo status` renders the control-plane table from the ledger (STATE.md = fallback/projection). Phase-1 hardenings from recon: `contracts.set_active_phase` now atomic; `zo build` exports ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_CONTRACTS_PATH so per-project sealing works in delivery sessions. Deferred to Phase 3 (documented): evaluate_loop_state ledger input, session-restore cutover (PR-036 precedence). **929 passed / 7 skipped, ruff clean, validate-docs green. Oracle checks 8-10 seeded tests pass.** **Next:** Phase 2 PR review/merge → Phase 3 (WS-C: watchdog, then fresh-context loop — demo validation needs the Linux box w/ claude CLI, PR-046).