Epic: Resume Interrupted/Incomplete Flows (Continue after timeout, stop, or failure)

[maddes8cht]

Description

A flow interrupted in any form – agent timeout, token budget exhaustion, manual stop, max_loops_reached, agent error, missing state update – is markable as incomplete and therefore resumable. The user inspects the stored termination reason, can disable specific reasons, and continues the same flow – optionally with a new timeout (and, if needed, a new max_loops).

Design principle: any state with termination_reason != "completed" is potentially resumable; a configurable filter decides which reasons are allowed. There is deliberately no flow-level timeout – the per-agent subprocess timeout stays the only time bound.

Checkpoint (file & lifecycle)

• Naming rule identical to the run log: <log_dir>/openloop-run-{name}-{ts}.json (same stem as the .log, same directory). --resume <log-or-json> derives the counterpart by swapping the extension.
• Contains the full state (WorkflowState.to_json(), incl. payload/meta), the agent position (phase, iteration, agent_index), and a full copy of the flow definition – resume always runs with the unmodified definition, independent of later edits to workflows/*.json or agents/*.md.
• Written atomically (tmp+rename) after every successful agent merge and at every interruption point (with the termination_reason).
• Deleted automatically on successful completion – the presence of a checkpoint file is a reliable "resumable run exists" probe.

Resume semantics

• Resume continues at the last completed agent boundary: the interrupted agent re-runs from scratch in a fresh opencode session.
• Mid-iteration resume does not re-increment the iteration (the interrupted iteration continues at agent_index); a max_loops_reached state resumes at the next iteration with a raised bound.
• Resume is itself re-resumable: the checkpoint is overwritten at every boundary, so repeated interruptions form an idempotent continuation chain.

Resumable reasons (config resume_reasons)

Default: all except "completed" – stopped, max_loops_reached, agent_error:<name>, timeout:<name>, missing_state:<name>. Users disable reasons they consider non-resumable.

Distinct timeout handling

RunResult gains timed_out: bool; the engine records timeout:<agent_name>:<seconds> instead of the generic agent_error:<name>, so the used timeout survives into the checkpoint for review and adjustment.

Resume invocation

• CLI: --resume <log|json>; mutually exclusive with --workflow (the definition comes from the checkpoint). Combined with --timeout <new> (required knob), --max-loops <new> (required for max_loops_reached resumes), and existing --opencode-defaults / --workdir / --init-script.
• Log: resume appends to the same log file (with a <resume> marker) to keep the 1:1 log↔checkpoint pairing.
• GUI: Continue button lists resumable runs (checkpoint files in log_dir), shows stored reason + timeout + iteration/phase, allows adjusting timeout/max-loops, relaunches with restored state.

Sub-Issues (ToDo)

• Configuration System (core/config.py) #1 Checkpoint persistence + position tracking – atomic write after merge + at interruption points, meta.position, auto-delete on completion
• Core State Management (core/state.py, core/parser.py) #2 RunResult.timed_out + timeout:<name>:<sec> reason
• Agent Definition Loader (core/agent.py) #3 CLI --resume + engine bootstrap – skip state reset, load embedded flow definition, mid-iteration vs boundary resume, validate position
• Subprocess Runner (core/runner.py) #4 Config resume_reasons filter (default: all except completed)
• Execution Engine (core/engine.py) #5 GUI Continue button + run picker
• Main Entry Point (openloop.py) #6 Resume log continuation (append same file, <resume> marker)
• Example Agents & Workflow #7 Integration tests – timeout→resume, stop→resume, max_loops→resume with raised bound, repeated resume, completion auto-deletes checkpoint

Scope

• In: resume mechanism, timeout + max-loops override, reason filter, GUI continue.
• Out: flow-level timeout (stays per-agent only), token-budget enforcement (see Future), opencode session reuse (fresh session on resume).
• Known limitation (documented): re-running an interrupted agent may re-apply side effects from before the crash – agents should treat payload as an accumulated journal.

Future (separate issue): Token budget

opencode run exposes no per-run budget flag; opencode stats is aggregate. A future token_budget config key could sum per-run usage from opencode session data and abort the loop on overflow – plugging directly into this resume mechanism as another interruption reason.

Prerequisites

Reuses existing building blocks: WorkflowState.to_json()/from_json() (core/state.py:20-39), cooperative stop_event, meta writes (core/engine.py:418-436), log-dir resolution (core/engine.py:270-278), per-agent timeout (core/runner.py:148).