fix: bind HITL resumes to the reviewed tool invocation

docs/HUMAN_IN_THE_LOOP.md

@@ -205,26 +205,54 @@ To resume, `Engine.resume(run_id, approval_id, decision)` or the streamed
 > identity used when the approval was created (for the engine API, via
 > `caller_session_id` / `X-Session-ID`). Omitting or changing it returns 403.
 
+> **Upgrade note:** Runs suspended before checkpointed model/tool steps existed
+> cannot be resumed; they fail and their approval is rejected. Decide or cancel
+> pending approvals before upgrading. See
+> [`ADR 0005`](adr/0005-checkpointed-hitl-continuation.md).
+
 1. **Atomically claims** the approval (`PENDING → RESUMING`) — exactly one caller
    wins (see §7).
 2. Moves the run through `RESUMING → RUNNING` and resumes the **same** LangGraph
    thread from its checkpoint via
-   `Command(resume={"decision": ...})`. The graph is not restarted, the agent is
-   not re-selected, and intent/planning are not re-run.
-3. The node re-executes, the coordinator interprets the typed decision, and:
+   `Command(resume={"decision": ..., "approval_id": ..., "tool_call_id": ...})`.
+   The graph is not restarted, the agent is not re-selected, and intent/planning
+   are not re-run.
+3. The node re-executes, restoring its checkpointed model responses and completed
+   tool outcomes instead of regenerating them. The gate checks that the resume's
+   `approval_id` and stable `tool_call_id` match the invocation it is about to
+   run; a missing or different binding aborts the run as `FAILED` and rejects
+   the approval. Otherwise the coordinator interprets the typed decision, and:
    - `ALLOW_ONCE` / `ALLOW_FOR_SESSION` → the original tool call executes and its
      result flows back to the agent (and `ALLOW_FOR_SESSION` records a session
      permission);
    - `DENY` → a structured `[denied]` result is returned and **no** tool/MCP call
      is made.
 
+The claimed decision takes precedence over auto mode and session grants for that
+one pending invocation. A grant added by another run, or an auto-mode change
+after rebuilding the engine, cannot bypass a pending denial. A request-local
+binding validates the first unfinished tool step before policy evaluation and
+is consumed only by the matching approval interrupt. Later calls use the normal
+policy; completed steps are restored without consuming this new decision. If a
+rebuilt graph removes or blocks the pending invocation before it reaches that
+interrupt, the unconsumed binding fails the run and rejects the claimed approval.
+
 An unrecognized or missing resume decision **fails closed to `DENY`**, so an
 ambiguous resume never triggers a side effect.
 
-Because a LangGraph node re-executes from its start on resume, tool *preparation*
-is deterministic and safe to repeat, and all non-idempotent execution happens
-**after** the interrupt returns. Duplicate execution on re-entry is prevented by
-the idempotency ledger (§7).
+A LangGraph node re-executes from its start on resume, but each model response
+and each completed leaf-tool outcome is a checkpointed LangGraph task. Replay
+returns those saved values, so the model cannot propose a different call than
+the one reviewed, and earlier tools, decisions, and session grants are not
+repeated. Restored steps must match the node, its input, and the invocation
+identity; otherwise the run fails with `CheckpointReplayMismatch`. Runs suspended
+before this format existed fail closed on resume. Duplicate execution of a call
+that started but did not finish is still prevented by the idempotency ledger
+(§7).
+
+Because the reviewed call must execute exactly as proposed, the checkpoint holds
+the model's raw tool arguments and the tool results; only the `ApprovalRecord`
+and the approval payload are masked. Treat the checkpoint store as sensitive.
 
 Identifiers are kept strictly separate:
 

docs/RUNTIME_LIFECYCLE.md

@@ -77,7 +77,31 @@ During a model tool loop, the tool-call message and matching tool result remain
 structured and ordered for the provider. A run paused for tool
 approval stores its continuation in the LangGraph checkpoint keyed by `run_id`;
 session-wide approval grants live in a separate approval repository keyed by
-session and tool identity. Normal stream completion does not cancel a suspended
+session and tool identity. Model responses and completed leaf-tool outcomes are
+checkpointed as individual steps, so resume restores the reviewed tool request
+and reuses earlier results instead of regenerating calls or repeating side
+effects. This also applies to nested agents and sequential approvals. Resumed
+decisions carry the approval id and stable invocation id, which covers the run,
+agent path, provider/server, tool name, and raw arguments without the provider's
+transient call id. The tool approval gate rejects missing or mismatched bindings
+before execution; the displayed and persisted approval arguments stay masked.
+The claimed decision also overrides any auto-mode or session-grant change while
+the invocation waited. Its binding is request-local and consumed only at the
+matching interrupt; later calls return to normal policy evaluation.
+Completing a replay with that binding still unconsumed is a mismatch: for
+example, a rebuilt graph cannot silently remove the reviewed tool and report its
+approval as applied.
+Restored steps also validate the node/input or invocation identity, so a changed
+route cannot reuse another agent's positional checkpoint. A mismatch aborts the
+run, rejects its claimed approval, and removes any newly created pending approval.
+The graph state carries an internal continuation version. Resuming a suspended
+run created before these checkpointed substeps existed fails closed before model
+or tool execution, because replaying that older format could repeat side effects.
+Execution limits still account for restored calls before their cached outcomes
+are returned; checkpointing does not reset the run's allowed tool budget.
+Restored model steps emit no new answer deltas or token usage. The existing tool
+execution ledger still protects calls whose step started but did not finish.
+Normal stream completion does not cancel a suspended
 approval. An explicit owner cancellation atomically rejects the pending
 approval, transitions the run to terminal `CANCELLED`, and prevents later
 resume.

docs/adr/0005-checkpointed-hitl-continuation.md

@@ -0,0 +1,80 @@
+# ADR 0005 — Checkpoint model and tool steps so resume runs the reviewed call
+
+- **Status:** Proposed
+- **Date:** 2026-10-14
+
+## Context
+
+A run suspended for tool approval resumed by re-executing its LangGraph node
+from the start. The node invoked the model again, and the resume command carried
+only `{"decision": ...}`. A non-deterministic model could therefore propose a
+different tool, agent, or arguments than the human reviewed, and the positional
+decision would still apply to it. Earlier tool calls in the same node were
+protected from repeated side effects only by the idempotency ledger, and
+restored turns reported their token usage a second time.
+
+## Decision
+
+The engine replays a suspended node from saved steps and binds each decision to
+one invocation.
+
+- `run_tool_loop` runs every model response as a LangGraph `@task`, and
+  `ToolInvoker` runs each leaf tool's gate and execution as a `@task`. On resume
+  LangGraph returns the saved values, so the reviewed tool request is never
+  regenerated and completed outcomes, denials, and session grants are not
+  repeated. Execution limits are registered before a step is restored, so the
+  run's tool budget is unchanged.
+- Restored steps validate their identity. A model step carries the node path,
+  its non-system input, and its turn. A tool step carries the stable
+  `tool_call_id`. A mismatch, for example after route filtering changes the
+  replay order, raises `CheckpointReplayMismatch`.
+- Resume sends `{"decision", "approval_id", "tool_call_id"}`. The stable id
+  covers the run, agent path, provider/server, tool name, and raw arguments, but
+  not the provider's transient call id. The approval gate raises
+  `ApprovalInvocationMismatch` before execution when the binding is missing or
+  different. Masked arguments are never used for binding, because distinct
+  secrets share the same redacted display.
+- Either mismatch fails the run, rejects the claimed approval, and cancels any
+  pending approval created during the failed replay. Orchestrators re-raise both
+  errors from nested children instead of turning them into tool text.
+- Each resume request carries its claimed binding in a context variable. The
+  first unfinished tool step validates that binding before policy evaluation;
+  its existing `ToolInvocation.approval_id` marks the claimed decision as
+  authoritative over auto mode or session grants. The matching interrupt consumes
+  the binding once, so later calls still follow normal policy. A replay that
+  completes without consuming it fails closed, covering rebuilt graphs that
+  remove or block the pending invocation. Separate requests never share this
+  state, and completed steps do not consume a new decision.
+- Approval cleanup is best effort. Repository failures are chained or logged
+  by type and cannot replace the original replay/consent mismatch.
+- Graph state carries `continuation_version`. Resuming a checkpoint without the
+  current version fails closed before any model or tool step runs, because an
+  older continuation has no saved steps and replaying it could repeat side
+  effects.
+- Outside a LangGraph runtime (`in_graph_run()` is false) the loop and invoker
+  run directly, without tasks.
+
+## Contract changes
+
+- The runtime dependency floor is `langgraph>=1.0,<2.0`, for functional tasks
+  inside `StateGraph` nodes and `langgraph.runtime.get_runtime`.
+- `GraphState` adds the internal `continuation_version` key.
+- The internal resume payload adds `approval_id` and `tool_call_id`.
+- New errors: `ApprovalInvocationMismatch` (an `ApprovalError`) and
+  `CheckpointReplayMismatch` (a `RuntimeError`).
+
+There is no YAML or HTTP request-shape change.
+
+## Consequences
+
+- Runs that are suspended when this version is deployed cannot be resumed; they
+  fail and their approval is rejected. Decide or cancel pending approvals before
+  upgrading.
+- The checkpoint stores the model's raw tool arguments and the tool results for
+  every step. It must be treated as a sensitive store. Only the `ApprovalRecord`
+  and the approval payload are masked.
+- Checkpoints grow with each model turn and tool result in a suspended run.
+- Restored model steps emit no new answer deltas or token usage, so a resumed run
+  no longer double-counts tokens.
+- The idempotency ledger still protects a call whose step started but did not
+  finish.

pyproject.toml

@@ -45,7 +45,8 @@ dependencies = [
     "uvicorn>=0.30",
     "httpx>=0.27",
     "jinja2>=3.1",
-    "langgraph>=0.2",
+    # HITL continuation uses functional tasks and the public runtime accessor.
+    "langgraph>=1.0,<2.0",
     "langchain>=0.3",
     "langchain-core>=0.3",
     "mcp>=1.27,<2",

src/agent_engine/approvals/coordinator.py

@@ -4,7 +4,7 @@
 together. It contains no tool-specific logic and performs no UI interaction (the
 injected provider owns that). Its single entry point is :meth:`resolve`:
 
-1. Ask the policy whether approval is required (auto mode → session → ask).
+1. Honor a claimed pending approval; otherwise ask policy (auto mode → session → ask).
 2. If not required, permit execution immediately (the provider is never called).
 3. Otherwise request a typed decision from the provider.
 4. Apply the decision: store a session permission only for ``ALLOW_FOR_SESSION``
@@ -76,7 +76,9 @@ def __init__(
 
     async def resolve(self, invocation: ToolInvocation, *, auto_mode: bool) -> ApprovalOutcome:
         query = await self._build_query(invocation, auto_mode=auto_mode)
-        required = self._policy.requires_approval(query)
+        # A claimed decision governs this pending invocation even if auto mode
+        # or a session grant changed while the human was reviewing it.
+        required = invocation.approval_id is not None or self._policy.requires_approval(query)
         self._log_evaluated(query, required=required)
 
         if not required:

src/agent_engine/approvals/errors.py

@@ -46,6 +46,14 @@ def __init__(self, approval_id: str, status: str) -> None:
         super().__init__(f"approval {approval_id} already processed (status={status})")
 
 
+class ApprovalInvocationMismatch(ApprovalError):
+    """A resumed decision does not bind to the invocation at the tool gate."""
+
+    def __init__(self, approval_id: str) -> None:
+        self.approval_id = approval_id
+        super().__init__(f"approval {approval_id} does not match the pending invocation")
+
+
 class UnauthorizedApprover(ApprovalError):
     def __init__(self, approval_id: str) -> None:
         self.approval_id = approval_id

src/agent_engine/approvals/invocation.py

@@ -85,6 +85,7 @@ class ToolInvocation:
     organization_id: str = ""
     run_id: str | None = None
     approval_id: str | None = None
+    """A claimed approval for this invocation; its decision overrides broad grants."""
     human_description: str = ""
 
     @property

src/agent_engine/engine/langgraph/approval_provider.py

@@ -17,6 +17,9 @@
 from agent_engine.approvals.approval_manager import ApprovalManager
 from agent_engine.approvals.approval_provider import ApprovalProvider, ApprovalRequest
 from agent_engine.approvals.decision import ApprovalDecision, parse_decision
+from agent_engine.approvals.errors import ApprovalInvocationMismatch
+from agent_engine.approvals.models import ApprovalStatus
+from agent_engine.engine.langgraph.execution.approval_resume import current_approval_resume
 from agent_engine.logging_config import log
 from agent_engine.runtime.hooks import current_run_context
 
@@ -96,4 +99,29 @@ async def request_decision(self, request: ApprovalRequest) -> ApprovalDecision:
             tool_call_id=inv.tool_call_id,
         )
         resume = interrupt(payload)
-        return parse_decision(resume, default=ApprovalDecision.DENY)
+        # The stable id covers the run, agent path, provider/server, tool, and
+        # raw arguments. Masked arguments cannot bind consent: distinct secrets
+        # have the same redacted display. Provider-generated call ids are absent.
+        if (
+            not isinstance(resume, dict)
+            or resume.get("approval_id") != record.approval_id
+            or resume.get("tool_call_id") != inv.tool_call_id
+        ):
+            # A changed invocation may have created a new pending row before the
+            # positional resume was consumed. It must not outlive this failure.
+            if record.status == ApprovalStatus.PENDING:
+                try:
+                    await self._approvals.cancel_pending(
+                        run_id=run_id,
+                        approval_id=record.approval_id,
+                        caller_user_id=authorized_user_id,
+                        caller_auth_ref=auth_ref,
+                    )
+                except Exception as exc:
+                    raise ApprovalInvocationMismatch(record.approval_id) from exc
+            raise ApprovalInvocationMismatch(record.approval_id)
+        decision = parse_decision(resume, default=ApprovalDecision.DENY)
+        binding = current_approval_resume.get()
+        if binding is not None:
+            binding.consume(record.approval_id, inv.tool_call_id)
+        return decision

src/agent_engine/engine/langgraph/checkpointing.py

@@ -13,6 +13,7 @@
 
 from langgraph.checkpoint.base import BaseCheckpointSaver
 from langgraph.checkpoint.memory import InMemorySaver
+from langgraph.runtime import get_runtime
 
 from agent_engine.logging_config import log
 
@@ -23,6 +24,21 @@ class CheckpointConfigError(RuntimeError):
     """A checkpoint connection string was supplied but its backend is unusable."""
 
 
+class CheckpointReplayMismatch(RuntimeError):
+    """A positional checkpoint belongs to a different model/tool invocation."""
+
+    def __init__(self, node_path: str) -> None:
+        super().__init__(f"checkpoint replay does not match the invocation at {node_path}")
+
+
+def in_graph_run() -> bool:
+    """Distinguish a LangGraph runtime from an ordinary LangChain Runnable."""
+    try:
+        return get_runtime() is not None
+    except RuntimeError:
+        return False
+
+
 @dataclass(frozen=True)
 class CheckpointerHandle:
     """A checkpointer plus a flag describing whether it is durable/shared.