Refactor orchestration with a Rust lifecycle supervisor

README.md

@@ -67,12 +67,16 @@ Tmux owns PTYs and interactive terminal lifecycle. Python is restricted to
 evaluation and provenance; it does not implement a second production workflow
 or acceptance gate.
 
-On production Linux, separate Unix identities isolate the orchestrator, the
-single active writer, read-only agents, and the authority supervisor. The
+On production Linux, separate Unix identities isolate the orchestrator,
+path-scoped writers, read-only agents, and the authority supervisor. The
 orchestrator can read worker and reviewer state but cannot write the target
 repository or protected lifecycle state. Completion is a request to the
 supervisor, which checks every gate under the lifecycle lock before changing
-the phase to `complete`.
+the phase to `complete`. A Rust lifecycle reconciler also performs mechanical
+process polling, terminal cleanup, assignment settlement, stale validation
+lease handling, and only pre-budgeted infrastructure retries. It owns no
+semantic policy and sends protected mutations to the authority supervisor;
+prompts retain semantic routing but no longer implement those fixed operations.
 
 ## Common Commands
 

docs/architecture.md

@@ -86,7 +86,7 @@ Production Linux runs four process identities:
 | Identity | May read target | May write target | Protected-state authority |
 | --- | --- | --- | --- |
 | Orchestrator | yes | no | typed requests only |
-| Writer | yes | assigned paths while leased | no |
+| Writers | yes | assigned paths while leased | no |
 | Scout/reviewer | yes | no | sealed evidence only |
 | Supervisor | metadata needed for gates | grants/revokes writer paths | yes |
 
@@ -106,6 +106,15 @@ workspace.
 The supervisor isolates connection failures. A disconnected client or broken
 pipe terminates that request, not the authority service.
 
+A dedicated Rust lifecycle reconciler runs in the tmux session under the
+non-writing orchestrator identity, because that identity owns the tmux socket.
+It observes panes, persists terminal output, and closes completed processes.
+Assignment and validation-lease settlement crosses the typed socket to the
+authority supervisor; the reconciler cannot write protected state directly. It
+may restore a recoverable infrastructure exit only when the agent was launched
+with remaining `infra_retry_budget`. It never chooses the worker count, task
+split, owned paths, or semantic repair.
+
 ## Lifecycle
 
 The normal state sequence is:
@@ -127,10 +136,22 @@ pre-implementation -> implementation -> post-implementation -> complete
 
 ### Implementation
 
-The supervisor authorizes one writer for predeclared paths. The writer receives
-the complete approved context and registered contract. Assignment checks reject
-changed files outside the declared scope. Read-only exploration can still run
-in parallel.
+The orchestrator builds an adaptive task graph and chooses worker
+responsibilities. The supervisor does not choose a worker count or task split;
+it authorizes each writer for predeclared paths only after the implementation
+gate is ready. The DAG reports dependency-ready nodes for orchestrator routing.
+Assignment checks reject changed files outside the declared scope.
+
+Disjoint writers can run concurrently when Linux Landlock provides a distinct
+per-process write boundary. Overlapping ownership is rejected. On systems
+without that boundary, the same workflow remains valid but writer execution is
+serialized as a security fallback. Read-only agents are not subject to the
+writer lease.
+
+This separates decisions from mechanics: the orchestrator reacts to `blocked`
+or exhausted recovery states, while the supervisor continuously reconciles
+observable process state without needing prompt instructions for polling,
+finalization, ownership release, or lease cleanup.
 
 ### Post-implementation
 

docs/decisions.md

@@ -77,19 +77,31 @@ state, create work requests, and ask for completion, but it cannot directly edit
 the target or protected state. A bypassed high-level CLI still runs under the
 same non-writing Unix identity.
 
-## Permit One Path-Scoped Writer
-
-**Decision:** Only one writer may be active. Its launch authorization binds the
-workflow, assignment, backend, prompt, and predeclared owned paths. Ownership is
-granted for the role lifetime and revoked after exit or cancellation.
-
-**Why:** Multiple writers complicate overlap detection and permit detached or
-late writes after cancellation. Sequential publication gives one consistent
-snapshot and acceptance boundary while agents may still explore in parallel as
-read-only roles.
-
-**Consequence:** Multiagent prioritizes deterministic authority over maximum
-write concurrency.
+The tmux-owning identity also hosts a fixed Rust lifecycle reconciler. That
+process may observe and close panes, but protected assignment and lease changes
+still cross the supervisor's typed authority socket. This avoids granting the
+shared worker/reviewer group access to the tmux control socket.
+
+## Keep Topology Adaptive and Writing Path-Scoped
+
+**Decision:** The orchestrator chooses the task graph, worker count, and each
+worker's responsibility. The supervisor does not encode a preferred topology.
+It admits write-capable workers only when their durable assignments own
+non-overlapping paths and the lifecycle gate is open.
+
+On Linux kernels with Landlock, disjoint writers may run concurrently under
+per-process write allowlists. If that isolation is unavailable, the supervisor
+falls back to a single active writer because shared Unix ownership cannot
+safely distinguish two processes using the writer UID.
+
+**Why:** Task decomposition is semantic and belongs to the orchestrator;
+non-overlap, lifecycle readiness, and isolation capability are mechanical and
+belong to the supervisor. A hard-coded worker count makes simple work expensive
+and parallelizable work unnecessarily slow.
+
+**Consequence:** Multiagent imposes no fixed worker count or responsibility
+catalog. Available isolation and actual path conflicts determine concurrency,
+while completion still evaluates one canonical diff.
 
 ## Bind Semantics and Reviews to Immutable Evidence
 

docs/getting-started.md

@@ -50,7 +50,8 @@ multiagent launch \
   --root /absolute/path/to/target-repo
 ```
 
-The default launch creates one tmux orchestrator window. It is a clean launch:
+The default launch creates an orchestrator window and a Rust lifecycle-supervisor
+window. It is a clean launch:
 persisted subagents are not automatically restored.
 
 Resume after an interrupted run:
@@ -190,8 +191,6 @@ multiagent subagent spawn contract-scout-01 \
   --role scout \
   --instruction "Extract structured must and must-not contract rules. Do not edit."
 
-multiagent subagent wait contract-scout-01 --timeout 900
-multiagent subagent finalize contract-scout-01
 multiagent workflow contract-register "$MULTIAGENT_WORKFLOW_ID" \
   --scout contract-scout-01
 ```
@@ -239,15 +238,18 @@ multiagent subagent spawn worker-01 \
   --workflow-id "$MULTIAGENT_WORKFLOW_ID" \
   --decision-id DEC-001 \
   --plan-id PLAN-A \
+  --infra-retries 1 \
   --instruction-file /absolute/path/to/worker-instruction.md
 
-multiagent subagent wait worker-01 --timeout 1800
 multiagent subagent assignment-check worker-01
 ```
 
 Only the supervisor-authorized writer receives temporary access to its existing
-owned paths. The global writer lease prevents a second writer from becoming
-active at the same time.
+owned paths. On Linux, disjoint path-scoped writers may run concurrently;
+overlapping writers are rejected. The lifecycle supervisor observes completion,
+settles ownership and validation state, and applies only the explicitly
+budgeted infrastructure retry. Use `multiagent status` to decide semantic next
+steps such as repairing blocked work.
 
 Update a durable checkpoint during long work:
 
@@ -269,7 +271,8 @@ multiagent snapshot --root "$MULTIAGENT_ROOT" --base HEAD --format json
 Transition to post-implementation with the reported hash, then run read-only
 scope, technical, decision-drift, and reflection reviews. Review instructions
 must include the original task, registered contract, approved context, and
-canonical diff. Finalize each reviewer so the supervisor can seal its output.
+canonical diff. The lifecycle supervisor finalizes terminal reviewer processes
+so their output can be sealed.
 
 Record review findings and todos through `multiagent workflow` and
 `multiagent subagent` commands. A changed diff invalidates previous acceptance.

evaluation/evalscope_multiagent_native_runner.py

@@ -82,7 +82,10 @@
 
 
 def solver_internal_timeout(agent_timeout: float) -> int:
-    reserve = int(os.environ.get("EVAL_NATIVE_SOLVER_TIMEOUT_RESERVE", "600"))
+    # Trace export and official verification happen after the solver process
+    # returns, so this reserve only needs to cover orderly tmux shutdown and
+    # workspace ownership restoration before AgentEnvironment's hard timeout.
+    reserve = int(os.environ.get("EVAL_NATIVE_SOLVER_TIMEOUT_RESERVE", "180"))
     reserve = max(90, min(reserve, int(agent_timeout) - 300))
     return max(300, int(agent_timeout) - reserve)
 

evaluation/native_solver/swe_prod_lifecycle.py

@@ -313,14 +313,20 @@ def run_prod_solver(prompt_path: str | None, workdir: Path, repo_root: Path, tim
 
     deadline = time.monotonic() + timeout
     resume_count = 0
+    final_phase: str | None = None
     try:
         while time.monotonic() < deadline:
             while time.monotonic() < deadline and tmux_has_orchestrator(session):
                 time.sleep(5)
 
             phase = active_workflow_phase()
-            if phase in {None, "complete"} or time.monotonic() >= deadline:
+            final_phase = phase
+            if phase == "complete" or time.monotonic() >= deadline:
                 break
+            if phase is None:
+                raise RuntimeError(
+                    "production multiagent orchestrator exited without a persisted workflow lifecycle"
+                )
 
             resume_count += 1
             log(
@@ -343,8 +349,15 @@ def run_prod_solver(prompt_path: str | None, workdir: Path, repo_root: Path, tim
     finally:
         if tmux_has_session(session):
             run(["tmux", "-S", str(TMUX_SOCKET), "kill-session", "-t", session], timeout=30)
+        restore_workspace_owner(workdir)
+
+    if final_phase != "complete":
+        rendered_phase = final_phase or "missing"
+        raise RuntimeError(
+            "production multiagent workflow did not reach supervisor-owned completion before "
+            f"the solver deadline (phase={rendered_phase}); refusing workspace handoff"
+        )
 
-    restore_workspace_owner(workdir)
     materialize_committed_changes(workdir, start_head)
     mark_untracked_intent_to_add(workdir, baseline_untracked=baseline_untracked)
     log("workspace prepared for EvalScope submission")

evaluation/native_solver/swe_prod_repository.py

@@ -59,12 +59,23 @@ def list_untracked_files(cwd: Path) -> list[str]:
     return [line.strip() for line in others.stdout.splitlines() if line.strip()]
 
 
+def is_framework_internal_path(path: str) -> bool:
+    """Return whether an untracked path belongs to multiagent's control plane."""
+
+    normalized = path[2:] if path.startswith("./") else path
+    return normalized == ".multiagent" or normalized.startswith(".multiagent/")
+
+
 def mark_untracked_intent_to_add(cwd: Path, *, baseline_untracked: set[str] | None = None) -> list[str]:
     """Expose newly created solver files without submitting image residue."""
 
     baseline = baseline_untracked or set()
     untracked = list_untracked_files(cwd)
-    intent_to_add = [path for path in untracked if (cwd / path).is_file()]
+    intent_to_add = [
+        path
+        for path in untracked
+        if (cwd / path).is_file() and not is_framework_internal_path(path)
+    ]
     intent_to_add = [path for path in intent_to_add if path not in baseline]
     if intent_to_add:
         result = run(["git", "add", "-N", "--", *intent_to_add], cwd=cwd, timeout=120)

orchestrator_prompt.md

@@ -5,8 +5,9 @@ You are the orchestrator, a commander running on Codex CLI.
 You run inside a dedicated tmux window. Your job is to coordinate worker agents
 and long-running subagents running in other tmux windows. You do not implement
 code yourself. You plan, spawn agents, monitor them, coordinate handoffs,
-finalize results, kill finished or stuck agents, spawn more agents when needed,
-and report status.
+make semantic decisions for blocked or exhausted work, spawn more agents when
+needed, and report status. The Rust lifecycle reconciler performs polling,
+terminal cleanup, assignment settlement, and budgeted infrastructure retry.
 
 ## Role
 
@@ -142,20 +143,22 @@ Use clear names:
 - Verifiers: `verifier-01-short-task`
 - Long-running subagents: `subagent-build-watch`
 
-Use one verifier window per worker assignment at a time. A verifier is a
-read-only reviewer, not a second implementer.
+Choose verifier topology from the evidence boundary. A verifier may review one
+assignment or a consolidated diff; the framework does not require one verifier
+per worker. A verifier is a read-only reviewer, not a second implementer.
 
-Before spawning a replacement worker for the same owned files, poll the existing
-worker and either finalize/kill it or explicitly wait. Prefer the bounded
-`multiagent subagent wait NAME --timeout SECONDS` command when a result is
-required before continuing; one immediate poll is not evidence that an agent is
-stalled. If validation ownership
+Before spawning a replacement worker for the same owned files, inspect the
+supervisor-observed state of the existing worker. The lifecycle supervisor owns
+polling, terminal cleanup, assignment settlement, abandoned validation leases,
+and explicitly budgeted infrastructure retry. The orchestrator decides whether
+a blocked or exhausted task needs replacement; one immediate observation is not
+evidence that an agent is stalled. If validation ownership
 is unclear, use the validation coordinator role before adding more workers.
 
 ## Role Routing
 
 Load `$PROMPT_DIR/prompts/playbooks/orchestration-routing.md` before spawning,
-verifying, replacing, or finalizing agents. It owns the detailed role-routing
+verifying, replacing, or accepting agent results. It owns the detailed role-routing
 workflow, progress/status procedure, safety rules, and optional playbook
 selection.
 

prompts/playbooks/agent-spawning.md

@@ -1,7 +1,7 @@
 # Agent Spawning Playbook
 
-Use this playbook whenever the orchestrator is about to create, monitor,
-replace, verify, or finalize worker windows or named subagents.
+Use this playbook whenever the orchestrator is about to create, inspect,
+replace, or verify worker windows or named subagents.
 
 ## Worker First Instruction
 
@@ -43,14 +43,18 @@ SUBAGENT_CLI="$WORKER_CLI" multiagent subagent spawn worker-01-task \
   --decision-id DECISION_ID \
   --plan-id PLAN_ID \
   --branch BRANCH \
+  --infra-retries 1 \
   --instruction-file WORKER_INSTRUCTION
-multiagent subagent wait worker-01-task --timeout 1800
 ```
 
 The supervisor creates the assignment under its lock, completes authority
-registration, and only then launches the trusted workspace-write worker. The
-orchestrator remains unable to edit the target workspace. Inspect a terminal
-`blocked` or `failed` result instead of treating it as completion. Separate git
+registration, and only then launches the trusted workspace-write worker. Its
+lifecycle reconciler observes the process, closes terminal windows, updates the
+assignment, and marks any abandoned validation lease stale. `--infra-retries`
+is an optional per-task policy; the supervisor may restore only that many
+recoverable infrastructure exits and never invents repair work. The orchestrator
+remains unable to edit the target workspace. Inspect a terminal `blocked` or
+`failed` result instead of treating it as completion. Separate git
 worktrees remain available for intentionally parallel, disjoint assignments,
 but require an explicit integration step before completion; do not use them for
 the normal SWE single-writer path.
@@ -62,11 +66,13 @@ persists context:
 
 ```bash
 multiagent subagent spawn subagent-build-watch --instruction "FIRST_INSTRUCTION_TEXT"
-multiagent subagent wait subagent-build-watch --timeout 900
 multiagent subagent inspect subagent-build-watch --lines 160
-multiagent subagent finalize subagent-build-watch
 ```
 
+The lifecycle supervisor monitors and finalizes the process automatically.
+Use `multiagent status` when a semantic decision depends on its result; do not
+reimplement polling or terminal cleanup in the prompt.
+
 For a bounded worker in the current worktree, `spawn` can create the durable
 assignment and worker in one command:
 
@@ -85,17 +91,18 @@ progress, before stopping, and whenever a blocker appears.
 ## Scout To Worker Handoff
 
 Read-only scouts are temporary evidence gatherers. Before spawning the first
-edit-capable worker, poll or inspect any active scout once, persist the useful
-ledger/findings, then finalize or kill the scout if it is still running. Do not
-let an active generic scout block `multiagent subagent spawn` for the implementation
-worker. Use `MULTIAGENT_ALLOW_PARALLEL_WORKERS=1` only when you intentionally
-want parallel disjoint workers and have recorded non-overlapping ownership.
+edit-capable worker, inspect the supervisor-observed status and persist the
+useful ledger/findings. Do not
+let an active generic scout block `multiagent subagent spawn` for implementation.
+The orchestrator may select any number of workers and task-specific
+responsibilities. Give every concurrent writer a durable path-scoped
+assignment; the supervisor admits disjoint leases and rejects overlap
+mechanically.
 
-For a contract scout, finalize it and register its sealed output before any
-worker or reviewer starts:
+For a contract scout, wait for supervisor-finalized output and register its
+sealed output before any worker or reviewer starts:
 
 ```bash
-multiagent subagent finalize CONTRACT_SCOUT_NAME
 multiagent workflow contract-register "$MULTIAGENT_WORKFLOW_ID" \
   --scout CONTRACT_SCOUT_NAME
 ```
@@ -107,8 +114,8 @@ injects the supervisor-owned original task and registered contract into every
 later worker and reviewer instruction.
 
 Give a live contract scout one bounded wait of at least 300 seconds before
-classifying it as stalled. Do not kill or finalize a running scout merely
-because one short poll has no final message. If it exits with an empty sealed
+classifying it as stalled. Do not cancel a running scout merely because one
+short status observation has no final message. If it exits with an empty sealed
 artifact, allow at most one replacement with a narrower source list and an
 explicit "return the structured artifact before any ninth tool call" reminder.
 If that replacement also has no artifact, stop with a recorded infrastructure
@@ -185,11 +192,10 @@ continue indefinitely: the next state must be a source diff,
 `required-path-outside-owned: RELATIVE_PATH`, `validation-repair-needed:`, or
 blocked status with a source-visible reason.
 
-After `multiagent subagent kill NAME` or `multiagent subagent finalize NAME`, ensure the
-assignment no longer owns paths before reusing them. If needed, run
-`multiagent subagent assignment-status NAME failed` for killed workers or
-`multiagent subagent assignment-status NAME done` for finalized workers before
-creating the replacement assignment.
+After a worker reaches a terminal state, wait for the lifecycle supervisor to
+settle its assignment before reusing those paths. Do not manually reproduce the
+assignment or validation-lease cleanup sequence. A blocked worker is not
+terminal; replacement remains an explicit orchestrator decision.
 
 Before final acceptance, run: